Oct 30 20:34:57.566281 ip-10-0-128-165 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 30 20:34:57.566292 ip-10-0-128-165 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 30 20:34:57.566299 ip-10-0-128-165 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 30 20:34:57.566509 ip-10-0-128-165 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 30 20:35:08.939219 ip-10-0-128-165 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 30 20:35:08.939244 ip-10-0-128-165 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 84fae1cfa6614325bd4a15278dc62013 --
Oct 30 20:37:24.476155 ip-10-0-128-165 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 20:37:24.827137 ip-10-0-128-165 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 20:37:24.827137 ip-10-0-128-165 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 20:37:24.827137 ip-10-0-128-165 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 20:37:24.827137 ip-10-0-128-165 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 20:37:24.833020 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.832933    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:37:24.833020 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833014    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:37:24.833020 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833018    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:37:24.833020 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833021    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:37:24.833020 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833024    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:37:24.833020 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833028    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:37:24.833020 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833031    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:37:24.833274 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833033    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:37:24.833274 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833036    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:37:24.833274 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833039    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:37:24.833274 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833041    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:37:24.833274 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833044    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:37:24.833274 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833047    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:37:24.833274 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833049    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:37:24.833274 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833052    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:37:24.833274 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833054    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:37:24.833274 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833057    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:37:24.833274 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833060    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:37:24.833274 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833062    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:37:24.833274 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833065    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:37:24.833274 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833068    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:37:24.833274 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833070    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:37:24.833274 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833073    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:37:24.833274 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833075    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:37:24.833274 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833078    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:37:24.833274 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833080    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:37:24.833725 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833083    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:37:24.833725 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833086    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:37:24.833725 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833088    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:37:24.833725 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833094    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:37:24.833725 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833097    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:37:24.833725 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833099    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:37:24.833725 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833102    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:37:24.833725 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833105    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:37:24.833725 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833107    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:37:24.833725 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833109    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:37:24.833725 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833112    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:37:24.833725 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833114    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:37:24.833725 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833117    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:37:24.833725 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833120    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:37:24.833725 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833123    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:37:24.833725 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833125    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:37:24.833725 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833128    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:37:24.833725 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833130    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:37:24.833725 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833133    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:37:24.834199 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833136    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:37:24.834199 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833138    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:37:24.834199 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833141    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:37:24.834199 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833144    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:37:24.834199 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833146    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:37:24.834199 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833149    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:37:24.834199 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833151    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:37:24.834199 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833154    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:37:24.834199 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833156    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:37:24.834199 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833159    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:37:24.834199 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833161    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:37:24.834199 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833164    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:37:24.834199 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833166    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:37:24.834199 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833168    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:37:24.834199 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833171    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:37:24.834199 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833174    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:37:24.834199 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833176    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:37:24.834199 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833178    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:37:24.834199 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833181    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:37:24.834199 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833183    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:37:24.834659 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833185    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:37:24.834659 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833188    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:37:24.834659 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833193    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:37:24.834659 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833197    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:37:24.834659 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833199    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:37:24.834659 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833202    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:37:24.834659 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833205    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:37:24.834659 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833209    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:37:24.834659 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833212    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:37:24.834659 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833215    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:37:24.834659 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833218    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:37:24.834659 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833220    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:37:24.834659 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833223    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:37:24.834659 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833225    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:37:24.834659 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833228    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:37:24.834659 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833230    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:37:24.834659 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833233    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:37:24.834659 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833236    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:37:24.834659 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833240    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:37:24.835159 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833243    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:37:24.835159 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833245    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:37:24.835159 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833248    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:37:24.835159 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833250    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:37:24.835159 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833253    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:37:24.835159 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833255    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:37:24.835159 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833259    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:37:24.835159 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833264    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:37:24.835159 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833267    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:37:24.835159 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833269    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:37:24.835159 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833272    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:37:24.835159 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833275    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:37:24.835159 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833277    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:37:24.835159 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833280    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:37:24.835159 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833283    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:37:24.835159 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833286    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:37:24.835159 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833288    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:37:24.835159 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833291    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:37:24.835159 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833294    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:37:24.835159 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833297    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:37:24.835627 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833299    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:37:24.835627 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.833302    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:37:24.835627 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834117    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:37:24.835627 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834124    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:37:24.835627 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834127    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:37:24.835627 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834130    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:37:24.835627 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834133    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:37:24.835627 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834136    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:37:24.835627 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834138    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:37:24.835627 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834141    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:37:24.835627 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834144    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:37:24.835627 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834146    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:37:24.835627 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834149    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:37:24.835627 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834153    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:37:24.835627 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834156    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:37:24.835627 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834158    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:37:24.835627 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834161    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:37:24.835627 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834164    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:37:24.835627 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834166    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:37:24.835627 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834168    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:37:24.836130 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834171    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:37:24.836130 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834173    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:37:24.836130 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834176    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:37:24.836130 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834178    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:37:24.836130 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834181    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:37:24.836130 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834183    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:37:24.836130 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834186    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:37:24.836130 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834188    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:37:24.836130 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834191    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:37:24.836130 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834193    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:37:24.836130 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834197    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:37:24.836130 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834199    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:37:24.836130 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834202    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:37:24.836130 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834205    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:37:24.836130 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834208    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:37:24.836130 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834211    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:37:24.836130 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834213    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:37:24.836130 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834216    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:37:24.836130 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834218    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:37:24.836130 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834221    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:37:24.836608 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834223    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:37:24.836608 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834225    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:37:24.836608 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834228    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:37:24.836608 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834230    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:37:24.836608 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834233    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:37:24.836608 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834236    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:37:24.836608 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834239    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:37:24.836608 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834241    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:37:24.836608 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834244    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:37:24.836608 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834246    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:37:24.836608 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834249    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:37:24.836608 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834251    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:37:24.836608 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834254    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:37:24.836608 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834256    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:37:24.836608 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834259    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:37:24.836608 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834262    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:37:24.836608 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834264    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:37:24.836608 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834267    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:37:24.836608 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834269    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:37:24.837088 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834272    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:37:24.837088 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834274    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:37:24.837088 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834276    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:37:24.837088 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834280    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:37:24.837088 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834283    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:37:24.837088 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834285    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:37:24.837088 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834288    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:37:24.837088 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834292    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:37:24.837088 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834296    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:37:24.837088 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834299    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:37:24.837088 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834301    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:37:24.837088 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834304    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:37:24.837088 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834306    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:37:24.837088 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834309    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:37:24.837088 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834312    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:37:24.837088 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834314    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:37:24.837088 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834317    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:37:24.837088 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834319    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:37:24.837088 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834322    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:37:24.837528 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834325    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:37:24.837528 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834328    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:37:24.837528 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834330    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:37:24.837528 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834332    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:37:24.837528 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834335    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:37:24.837528 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834337    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:37:24.837528 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834340    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:37:24.837528 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834343    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:37:24.837528 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834346    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:37:24.837528 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834348    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:37:24.837528 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834351    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:37:24.837528 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834354    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:37:24.837528 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834356    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:37:24.837528 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834359    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:37:24.837528 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834361    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:37:24.837528 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834364    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:37:24.837528 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834366    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:37:24.837528 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834369    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:37:24.837528 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834371    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:37:24.837997 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834373    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:37:24.837997 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834378    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:37:24.837997 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834381    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:37:24.837997 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834385    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:37:24.837997 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834387    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:37:24.837997 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834391    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:37:24.837997 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834394    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:37:24.837997 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834396    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:37:24.837997 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834399    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:37:24.837997 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834401    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:37:24.837997 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834403    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:37:24.837997 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834977    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:37:24.837997 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834985    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:37:24.837997 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834989    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:37:24.837997 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834992    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:37:24.837997 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834995    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:37:24.837997 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.834997    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:37:24.837997 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835000    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:37:24.837997 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835003    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:37:24.838447 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835005    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:37:24.838447 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835008    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:37:24.838447 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835017    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:37:24.838447 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835019    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:37:24.838447 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835022    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:37:24.838447 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835025    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:37:24.838447 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835027    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:37:24.838447 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835030    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:37:24.838447 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835032    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:37:24.838447 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835035    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:37:24.838447 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835037    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:37:24.838447 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835040    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:37:24.838447 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835042    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:37:24.838447 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835045    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:37:24.838447 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835047    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:37:24.838447 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835050    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:37:24.838447 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835052    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:37:24.838447 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835054    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:37:24.838447 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835057    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:37:24.838447 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835059    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:37:24.838447 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835062    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:37:24.838953 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835064    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:37:24.838953 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835067    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:37:24.838953 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835069    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:37:24.838953 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835073    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:37:24.838953 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835077    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:37:24.838953 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835080    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:37:24.838953 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835083    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:37:24.838953 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835086    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:37:24.838953 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835088    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:37:24.838953 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835091    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:37:24.838953 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835095    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:37:24.838953 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835098    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:37:24.838953 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835101    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:37:24.838953 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835106    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:37:24.838953 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835123    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:37:24.838953 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835127    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:37:24.838953 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835130    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:37:24.838953 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835134    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:37:24.838953 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835137    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:37:24.839388 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835139    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:37:24.839388 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835142    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:37:24.839388 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835145    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:37:24.839388 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835147    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:37:24.839388 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835150    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:37:24.839388 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835152    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:37:24.839388 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835155    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:37:24.839388 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835158    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:37:24.839388 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835161    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:37:24.839388 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835163    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:37:24.839388 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835166    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:37:24.839388 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835168    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:37:24.839388 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835171    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:37:24.839388 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835173    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:37:24.839388 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835176    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:37:24.839388 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835178    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:37:24.839388 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835181    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:37:24.839388 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835184    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:37:24.839388 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835186    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:37:24.839878 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835189    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:37:24.839878 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835192    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:37:24.839878 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835194    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:37:24.839878 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835197    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:37:24.839878 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835200    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:37:24.839878 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835202    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:37:24.839878 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835205    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:37:24.839878 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835208    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:37:24.839878 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835210    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:37:24.839878 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835213    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:37:24.839878 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835215    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:37:24.839878 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835218    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:37:24.839878 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835220    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:37:24.839878 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835223    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:37:24.839878 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835225    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:37:24.839878 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835227    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:37:24.839878 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835230    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:37:24.839878 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835232    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:37:24.839878 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835234    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:37:24.839878 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835237    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:37:24.840344 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835239    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:37:24.840344 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835242    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:37:24.840344 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835244    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:37:24.840344 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835246    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:37:24.840344 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835249    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:37:24.840344 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835251    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:37:24.840344 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835254    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:37:24.840344 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835256    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:37:24.840344 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835258    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:37:24.840344 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835261    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:37:24.840344 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835263    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:37:24.840344 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835265    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:37:24.840344 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835269    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:37:24.840344 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835272    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:37:24.840344 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835275    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:37:24.840344 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835278    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:37:24.840344 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835280    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:37:24.840344 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835283    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:37:24.840344 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.835287    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:37:24.840801 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836661    2576 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 20:37:24.840801 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836674    2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 20:37:24.840801 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836681    2576 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 20:37:24.840801 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836685    2576 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 20:37:24.840801 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836690    2576 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 20:37:24.840801 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836693    2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 20:37:24.840801 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836697    2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 20:37:24.840801 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836702    2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 20:37:24.840801 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836705    2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 20:37:24.840801 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836708    2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 20:37:24.840801 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836712    2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 20:37:24.840801 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836715    2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 20:37:24.840801 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836718    2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 20:37:24.840801 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836721    2576 flags.go:64] FLAG: --cgroup-root=""
Oct 30 20:37:24.840801 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836724    2576 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 20:37:24.840801 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836727    2576 flags.go:64] FLAG: --client-ca-file=""
Oct 30 20:37:24.840801 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836729    2576 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 20:37:24.840801 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836732    2576 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 20:37:24.840801 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836737    2576 flags.go:64] FLAG: --cluster-domain=""
Oct 30 20:37:24.840801 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836740    2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 20:37:24.840801 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836743    2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 20:37:24.840801 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836746    2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 20:37:24.840801 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836750    2576 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 20:37:24.840801 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836759    2576 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 20:37:24.841483 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836762    2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 20:37:24.841483 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836766    2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 20:37:24.841483 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836769    2576 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 20:37:24.841483 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836772    2576 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 20:37:24.841483 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836776    2576 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 20:37:24.841483 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836779    2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 20:37:24.841483 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836782    2576 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 20:37:24.841483 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836796    2576 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 20:37:24.841483 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836800    2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 20:37:24.841483 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836804    2576 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 20:37:24.841483 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836806    2576 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 20:37:24.841483 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836809    2576 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 20:37:24.841483 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836812    2576 flags.go:64] FLAG: --enable-server="true"
Oct 30 20:37:24.841483 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836816    2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 20:37:24.841483 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836819    2576 flags.go:64] FLAG: --event-burst="100"
Oct 30 20:37:24.841483 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836822    2576 flags.go:64] FLAG: --event-qps="50"
Oct 30 20:37:24.841483 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836825    2576 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 20:37:24.841483 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836828    2576 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 20:37:24.841483 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836831    2576 flags.go:64] FLAG: --eviction-hard=""
Oct 30 20:37:24.841483 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836835    2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 20:37:24.841483 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836838    2576 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 20:37:24.841483 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836841    2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 20:37:24.841483 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836844    2576 flags.go:64] FLAG: --eviction-soft=""
Oct 30 20:37:24.841483 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836847    2576 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 20:37:24.841483 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836850    2576 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 20:37:24.842129 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836853    2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 20:37:24.842129 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836855    2576 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 20:37:24.842129 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836858    2576 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 20:37:24.842129 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836861    2576 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 20:37:24.842129 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836863    2576 flags.go:64] FLAG: --feature-gates=""
Oct 30 20:37:24.842129 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836867    2576 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 20:37:24.842129 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836870    2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 20:37:24.842129 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836873    2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 20:37:24.842129 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836876    2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 20:37:24.842129 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836879    2576 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 20:37:24.842129 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836882    2576 flags.go:64] FLAG: --help="false"
Oct 30 20:37:24.842129 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836885    2576 flags.go:64] FLAG: --hostname-override="ip-10-0-128-165.us-west-1.compute.internal"
Oct 30 20:37:24.842129 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836889    2576 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 20:37:24.842129 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836892    2576 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 20:37:24.842129 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836895    2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 20:37:24.842129 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836898    2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 20:37:24.842129 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836902    2576 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 20:37:24.842129 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836904    2576 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 20:37:24.842129 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836907    2576 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 20:37:24.842129 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836910    2576 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 20:37:24.842129 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836912    2576 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 20:37:24.842129 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836915    2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 20:37:24.842129 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836918    2576 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 20:37:24.842675 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836921    2576 flags.go:64] FLAG: --kube-reserved=""
Oct 30 20:37:24.842675 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836924    2576 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 20:37:24.842675 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836926    2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 20:37:24.842675 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836929    2576 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 20:37:24.842675 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836932    2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 20:37:24.842675 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836934    2576 flags.go:64] FLAG: --lock-file=""
Oct 30 20:37:24.842675 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836937    2576 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 20:37:24.842675 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836940    2576 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 20:37:24.842675 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836943    2576 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 20:37:24.842675 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836949    2576 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 20:37:24.842675 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836951    2576 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 20:37:24.842675 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836954    2576 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 20:37:24.842675 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836957    2576 flags.go:64] FLAG: --logging-format="text"
Oct 30 20:37:24.842675 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836960    2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 20:37:24.842675 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836963    2576 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 20:37:24.842675 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836966    2576 flags.go:64] FLAG: --manifest-url=""
Oct 30 20:37:24.842675 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836969    2576 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 20:37:24.842675 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836973    2576 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 20:37:24.842675 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836977    2576 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 20:37:24.842675 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836980    2576 flags.go:64] FLAG: --max-pods="110"
Oct 30 20:37:24.842675 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836983    2576 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 20:37:24.842675 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836986    2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 20:37:24.842675 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836996    2576 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 20:37:24.842675 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.836999    2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 20:37:24.842675 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.837002    2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 20:37:24.843341 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.837005    2576 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 20:37:24.843341 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.837008    2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 20:37:24.843341 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.837015    2576 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 20:37:24.843341 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.837018    2576 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 20:37:24.843341 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.837021    2576 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 20:37:24.843341 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.837024    2576 flags.go:64] FLAG: --pod-cidr=""
Oct 30 20:37:24.843341 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.837027    2576 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 20:37:24.843341 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.837030    2576 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 20:37:24.843341 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.837033    2576 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 20:37:24.843341 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.837035    2576 flags.go:64] FLAG: --port="10250"
Oct 30 20:37:24.843341 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.837038    2576 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 20:37:24.843341 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.837041    2576 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0153b63859c7ccd86"
Oct 30 20:37:24.843341 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.837044    2576 flags.go:64] FLAG: --qos-reserved=""
Oct 30 20:37:24.843341 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.837047    2576 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 20:37:24.843341 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.837050    2576 flags.go:64] FLAG: --register-node="true"
Oct 30 20:37:24.843341 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.837053    2576 flags.go:64] FLAG: --register-with-taints=""
Oct 30 20:37:24.843341 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.837057    2576 flags.go:64] FLAG: --registry-burst="10"
Oct 30 20:37:24.843341 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.837059    2576 flags.go:64] FLAG: --registry-qps="5"
Oct 30 20:37:24.843341 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.837062    2576 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 20:37:24.843341 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.837065    2576 flags.go:64] FLAG: --reserved-memory=""
Oct 30 20:37:24.843341 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.837068    2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 20:37:24.843341 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.837071    2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 20:37:24.843341 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.837074    2576 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 20:37:24.843341 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.837077    2576 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 20:37:24.843341 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.837080    2576 flags.go:64] FLAG: --runonce="false"
Oct 30 20:37:24.843341 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.837082    2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 20:37:24.843981 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.837086    2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 20:37:24.843981 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.837088    2576 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 20:37:24.843981 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.837091    2576 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 20:37:24.843981 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.837096    2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 20:37:24.843981 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.837100    2576 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 20:37:24.843981 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.837103    2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 20:37:24.843981 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.837107    2576 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 20:37:24.843981 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.837109    2576 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 20:37:24.843981 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.837112    2576 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 20:37:24.843981 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.837115    2576 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 20:37:24.843981 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.837118    2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 20:37:24.843981 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.837120    2576 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 20:37:24.843981 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.837123    2576 flags.go:64] FLAG: --system-cgroups=""
Oct 30 20:37:24.843981 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.837126    2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 20:37:24.843981 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.837132    2576 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 20:37:24.843981 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.837134    2576 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 20:37:24.843981 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.837137    2576 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 20:37:24.843981 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.837142    2576 flags.go:64] FLAG: --tls-min-version=""
Oct 30 20:37:24.843981 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.837144    2576 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 20:37:24.843981 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.837147    2576 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 20:37:24.843981 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.837150    2576 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 20:37:24.843981 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.837153    2576 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 20:37:24.843981 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.837156    2576 flags.go:64] FLAG: --v="2"
Oct 30 20:37:24.843981 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.837160    2576 flags.go:64] FLAG: --version="false"
Oct 30 20:37:24.843981 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.837164    2576 flags.go:64] FLAG: --vmodule=""
Oct 30 20:37:24.844571 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.837168    2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 20:37:24.844571 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.837171    2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 20:37:24.844571 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837313    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:37:24.844571 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837318    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:37:24.844571 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837321    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:37:24.844571 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837324    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:37:24.844571 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837327    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:37:24.844571 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837331    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:37:24.844571 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837334    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:37:24.844571 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837337    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:37:24.844571 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837340    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:37:24.844571 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837343    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:37:24.844571 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837346    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:37:24.844571 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837349    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:37:24.844571 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837352    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:37:24.844571 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837354    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:37:24.844571 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837357    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:37:24.844571 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837359    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:37:24.844571 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837361    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:37:24.844571 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837364    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:37:24.845073 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837366    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:37:24.845073 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837370    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:37:24.845073 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837374    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:37:24.845073 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837377    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:37:24.845073 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837380    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:37:24.845073 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837383    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:37:24.845073 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837385    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:37:24.845073 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837388    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:37:24.845073 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837390    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:37:24.845073 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837393    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:37:24.845073 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837395    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:37:24.845073 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837398    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:37:24.845073 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837400    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:37:24.845073 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837403    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:37:24.845073 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837406    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:37:24.845073 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837410    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:37:24.845073 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837412    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:37:24.845073 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837415    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:37:24.845073 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837417    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:37:24.845523 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837420    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:37:24.845523 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837422    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:37:24.845523 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837425    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:37:24.845523 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837429    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:37:24.845523 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837431    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:37:24.845523 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837433    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:37:24.845523 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837436    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:37:24.845523 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837439    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:37:24.845523 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837441    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:37:24.845523 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837444    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:37:24.845523 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837446    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:37:24.845523 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837449    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:37:24.845523 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837451    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:37:24.845523 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837454    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:37:24.845523 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837457    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:37:24.845523 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837459    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:37:24.845523 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837462    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:37:24.845523 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837464    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:37:24.845523 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837467    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:37:24.846006 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837469    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:37:24.846006 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837471    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:37:24.846006 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837474    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:37:24.846006 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837476    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:37:24.846006 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837479    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:37:24.846006 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837481    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:37:24.846006 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837484    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:37:24.846006 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837486    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:37:24.846006 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837488    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:37:24.846006 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837492    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:37:24.846006 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837495    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:37:24.846006 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837497    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:37:24.846006 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837500    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:37:24.846006 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837502    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:37:24.846006 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837505    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:37:24.846006 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837507    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:37:24.846006 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837511    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:37:24.846006 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837514    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:37:24.846006 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837516    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:37:24.846451 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837519    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:37:24.846451 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837525    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:37:24.846451 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837527    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:37:24.846451 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837530    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:37:24.846451 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837533    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:37:24.846451 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837536    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:37:24.846451 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837538    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:37:24.846451 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837541    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:37:24.846451 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837543    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:37:24.846451 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837546    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:37:24.846451 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837548    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:37:24.846451 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837551    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:37:24.846451 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837553    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:37:24.846451 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837555    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:37:24.846451 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837558    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:37:24.846451 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837560    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:37:24.846451 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837563    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:37:24.846451 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837565    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:37:24.846451 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837567    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:37:24.846451 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837570    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:37:24.846948 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837572    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:37:24.846948 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837574    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:37:24.846948 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837578    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:37:24.846948 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837581    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:37:24.846948 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837583    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:37:24.846948 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837586    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:37:24.846948 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837588    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:37:24.846948 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837591    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:37:24.846948 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837594    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:37:24.846948 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837598    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:37:24.846948 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837603    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:37:24.846948 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.837630    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 20:37:24.846948 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837675    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:37:24.846948 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837680    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:37:24.846948 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837684    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:37:24.847303 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837687    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:37:24.847303 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837690    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:37:24.847303 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837693    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:37:24.847303 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837695    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:37:24.847303 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837698    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:37:24.847303 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837700    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:37:24.847303 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837703    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:37:24.847303 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837705    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:37:24.847303 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837707    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:37:24.847303 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837710    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:37:24.847303 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837713    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:37:24.847303 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837715    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:37:24.847303 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837718    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:37:24.847303 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837721    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:37:24.847303 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837723    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:37:24.847303 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837726    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:37:24.847303 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837728    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:37:24.847303 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837731    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:37:24.847303 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837733    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:37:24.847747 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837735    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:37:24.847747 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837739    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:37:24.847747 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837749    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:37:24.847747 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837752    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:37:24.847747 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837755    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:37:24.847747 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837758    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:37:24.847747 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837760    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:37:24.847747 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837764    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:37:24.847747 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837767    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:37:24.847747 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837773    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:37:24.847747 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837776    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:37:24.847747 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837779    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:37:24.847747 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837781    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:37:24.847747 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837802    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:37:24.847747 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837805    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:37:24.847747 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837808    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:37:24.847747 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837811    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:37:24.847747 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837813    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:37:24.847747 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837822    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:37:24.848212 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837825    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:37:24.848212 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837828    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:37:24.848212 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837830    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:37:24.848212 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837833    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:37:24.848212 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837835    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:37:24.848212 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837838    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:37:24.848212 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837841    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:37:24.848212 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837843    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:37:24.848212 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837846    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:37:24.848212 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837848    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:37:24.848212 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837850    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:37:24.848212 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837853    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:37:24.848212 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837855    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:37:24.848212 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837858    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:37:24.848212 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837861    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:37:24.848212 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837864    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:37:24.848212 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837866    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:37:24.848212 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837869    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:37:24.848212 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837871    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:37:24.848212 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837874    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:37:24.848212 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837878    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:37:24.848739 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837880    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:37:24.848739 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837883    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:37:24.848739 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837885    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:37:24.848739 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837888    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:37:24.848739 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837891    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:37:24.848739 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837894    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:37:24.848739 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837905    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:37:24.848739 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837909    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:37:24.848739 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837912    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:37:24.848739 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837915    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:37:24.848739 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837918    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:37:24.848739 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837920    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:37:24.848739 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837923    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:37:24.848739 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837926    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:37:24.848739 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837928    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:37:24.848739 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837931    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:37:24.848739 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837933    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:37:24.848739 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837936    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:37:24.848739 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837938    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:37:24.849204 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837941    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:37:24.849204 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837943    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:37:24.849204 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837946    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:37:24.849204 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837948    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:37:24.849204 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837950    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:37:24.849204 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837953    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:37:24.849204 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837957    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:37:24.849204 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837960    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:37:24.849204 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837964    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:37:24.849204 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837967    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:37:24.849204 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837969    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:37:24.849204 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837972    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:37:24.849204 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837976    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:37:24.849204 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837978    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:37:24.849204 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837981    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:37:24.849204 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837983    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:37:24.849204 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837986    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:37:24.849204 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837988    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:37:24.849204 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837991    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:37:24.849646 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.837995    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:37:24.849646 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.838003    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:37:24.849646 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.838006    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:37:24.849646 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.838009    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:37:24.849646 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.838011    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:37:24.849646 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.838014    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:37:24.849646 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.845895    2576 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 20:37:24.849646 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.845911    2576 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 20:37:24.849646 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.845928    2576 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 20:37:24.849646 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.845932    2576 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 20:37:24.849646 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.845995    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:37:24.849646 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846001    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:37:24.849646 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846004    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:37:24.849646 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846007    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:37:24.849646 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846010    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:37:24.849646 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846013    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:37:24.849646 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846016    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:37:24.849646 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846019    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:37:24.849646 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846022    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:37:24.849646 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846025    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:37:24.850127 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846027    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:37:24.850127 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846029    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:37:24.850127 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846032    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:37:24.850127 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846034    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:37:24.850127 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846037    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:37:24.850127 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846039    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:37:24.850127 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846042    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:37:24.850127 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846044    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:37:24.850127 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846047    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:37:24.850127 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846049    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:37:24.850127 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846052    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:37:24.850127 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846055    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:37:24.850127 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846057    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:37:24.850127 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846061    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:37:24.850127 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846063    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:37:24.850127 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846066    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:37:24.850127 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846068    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:37:24.850127 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846070    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:37:24.850127 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846073    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:37:24.850566 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846076    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:37:24.850566 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846079    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:37:24.850566 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846081    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:37:24.850566 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846084    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:37:24.850566 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846086    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:37:24.850566 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846088    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:37:24.850566 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846091    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:37:24.850566 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846093    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:37:24.850566 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846096    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:37:24.850566 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846099    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:37:24.850566 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846101    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:37:24.850566 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846104    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:37:24.850566 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846106    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:37:24.850566 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846108    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:37:24.850566 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846111    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:37:24.850566 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846114    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:37:24.850566 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846116    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:37:24.850566 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846118    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:37:24.850566 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846121    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:37:24.851145 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846124    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:37:24.851145 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846126    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:37:24.851145 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846128    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:37:24.851145 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846131    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:37:24.851145 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846133    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:37:24.851145 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846136    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:37:24.851145 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846138    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:37:24.851145 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846141    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:37:24.851145 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846144    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:37:24.851145 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846147    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:37:24.851145 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846150    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:37:24.851145 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846152    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:37:24.851145 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846155    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:37:24.851145 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846158    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:37:24.851145 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846161    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:37:24.851145 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846163    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:37:24.851145 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846166    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:37:24.851145 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846168    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:37:24.851145 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846171    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:37:24.851145 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846173    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:37:24.851612 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846176    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:37:24.851612 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846178    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:37:24.851612 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846181    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:37:24.851612 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846183    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:37:24.851612 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846186    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:37:24.851612 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846188    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:37:24.851612 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846191    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:37:24.851612 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846194    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:37:24.851612 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846196    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:37:24.851612 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846198    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:37:24.851612 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846201    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:37:24.851612 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846204    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:37:24.851612 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846208    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:37:24.851612 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846211    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:37:24.851612 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846214    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:37:24.851612 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846216    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:37:24.851612 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846220    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:37:24.851612 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846224    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:37:24.851612 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846227    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:37:24.852069 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846229    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:37:24.852069 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846232    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:37:24.852069 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846236    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:37:24.852069 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846238    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:37:24.852069 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846241    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:37:24.852069 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846243    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:37:24.852069 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846246    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:37:24.852069 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846249    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:37:24.852069 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846251    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:37:24.852069 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846254    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:37:24.852069 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846256    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:37:24.852069 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846259    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:37:24.852069 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846261    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:37:24.852069 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846264    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:37:24.852069 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846266    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:37:24.852069 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846268    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:37:24.852069 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846271    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:37:24.852069 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846273    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:37:24.852069 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846276    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:37:24.852510 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.846303    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 20:37:24.852510 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846434    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:37:24.852510 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846438    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:37:24.852510 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846440    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:37:24.852510 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846443    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:37:24.852510 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846447    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:37:24.852510 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846451    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:37:24.852510 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846454    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:37:24.852510 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846456    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:37:24.852510 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846459    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:37:24.852510 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846462    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:37:24.852510 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846464    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:37:24.852510 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846467    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:37:24.852510 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846469    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:37:24.852510 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846471    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:37:24.852510 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846474    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:37:24.852919 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846477    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:37:24.852919 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846479    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:37:24.852919 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846482    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:37:24.852919 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846484    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:37:24.852919 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846487    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:37:24.852919 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846489    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:37:24.852919 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846492    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:37:24.852919 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846494    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:37:24.852919 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846496    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:37:24.852919 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846499    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:37:24.852919 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846501    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:37:24.852919 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846504    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:37:24.852919 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846506    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:37:24.852919 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846508    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:37:24.852919 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846511    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:37:24.852919 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846513    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:37:24.852919 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846516    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:37:24.852919 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846518    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:37:24.852919 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846521    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:37:24.852919 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846523    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:37:24.853395 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846525    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:37:24.853395 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846528    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:37:24.853395 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846531    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:37:24.853395 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846533    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:37:24.853395 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846536    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:37:24.853395 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846538    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:37:24.853395 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846540    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:37:24.853395 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846543    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:37:24.853395 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846545    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:37:24.853395 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846548    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:37:24.853395 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846550    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:37:24.853395 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846553    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:37:24.853395 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846555    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:37:24.853395 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846558    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:37:24.853395 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846560    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:37:24.853395 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846563    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:37:24.853395 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846565    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:37:24.853395 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846568    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:37:24.853395 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846570    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:37:24.853863 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846573    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:37:24.853863 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846575    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:37:24.853863 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846578    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:37:24.853863 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846580    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:37:24.853863 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846582    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:37:24.853863 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846585    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:37:24.853863 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846587    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:37:24.853863 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846590    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:37:24.853863 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846592    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:37:24.853863 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846595    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:37:24.853863 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846597    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:37:24.853863 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846599    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:37:24.853863 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846602    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:37:24.853863 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846604    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:37:24.853863 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846607    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:37:24.853863 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846609    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:37:24.853863 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846612    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:37:24.853863 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846614    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:37:24.853863 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846616    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:37:24.854352 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846620    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:37:24.854352 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846623    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:37:24.854352 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846626    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:37:24.854352 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846630    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:37:24.854352 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846633    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:37:24.854352 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846635    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:37:24.854352 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846637    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:37:24.854352 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846640    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:37:24.854352 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846642    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:37:24.854352 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846645    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:37:24.854352 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846648    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:37:24.854352 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846650    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:37:24.854352 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846652    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:37:24.854352 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846655    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:37:24.854352 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846657    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:37:24.854352 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846660    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:37:24.854352 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846662    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:37:24.854352 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846664    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:37:24.854352 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846667    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:37:24.854821 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846669    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:37:24.854821 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846672    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:37:24.854821 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846674    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:37:24.854821 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846676    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:37:24.854821 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846678    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:37:24.854821 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846681    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:37:24.854821 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846685    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:37:24.854821 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846688    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:37:24.854821 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846690    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:37:24.854821 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846693    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:37:24.854821 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846695    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:37:24.854821 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846698    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:37:24.854821 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846700    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:37:24.854821 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846703    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:37:24.854821 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.846725    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 20:37:24.854821 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846759    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:37:24.855205 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846763    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:37:24.855205 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846767    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:37:24.855205 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846769    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:37:24.855205 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846772    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:37:24.855205 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846774    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:37:24.855205 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846777    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:37:24.855205 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846780    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:37:24.855205 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846783    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:37:24.855205 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846804    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:37:24.855205 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846807    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:37:24.855205 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846812    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:37:24.855205 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846816    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:37:24.855205 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846819    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:37:24.855205 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846822    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:37:24.855205 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846825    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:37:24.855205 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846828    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:37:24.855205 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846830    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:37:24.855205 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846833    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:37:24.855205 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846835    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:37:24.855663 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846838    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:37:24.855663 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846841    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:37:24.855663 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846843    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:37:24.855663 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846846    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:37:24.855663 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846852    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:37:24.855663 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846855    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:37:24.855663 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846857    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:37:24.855663 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846860    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:37:24.855663 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846862    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:37:24.855663 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846865    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:37:24.855663 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846867    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:37:24.855663 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846870    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:37:24.855663 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846873    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:37:24.855663 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846875    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:37:24.855663 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846878    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:37:24.855663 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846880    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:37:24.855663 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846883    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:37:24.855663 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846885    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:37:24.855663 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846888    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:37:24.855663 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846890    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:37:24.856144 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846894    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:37:24.856144 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846897    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:37:24.856144 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846899    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:37:24.856144 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846902    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:37:24.856144 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846904    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:37:24.856144 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846907    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:37:24.856144 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846909    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:37:24.856144 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846911    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:37:24.856144 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846914    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:37:24.856144 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846917    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:37:24.856144 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846919    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:37:24.856144 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846921    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:37:24.856144 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846924    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:37:24.856144 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846927    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:37:24.856144 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846929    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:37:24.856144 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846931    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:37:24.856144 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846935    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:37:24.856144 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846937    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:37:24.856144 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846940    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:37:24.856669 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846942    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:37:24.856669 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846945    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:37:24.856669 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846948    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:37:24.856669 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846950    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:37:24.856669 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846953    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:37:24.856669 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846957    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:37:24.856669 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846959    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:37:24.856669 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846962    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:37:24.856669 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846965    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:37:24.856669 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846967    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:37:24.856669 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846970    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:37:24.856669 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846972    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:37:24.856669 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846975    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:37:24.856669 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846977    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:37:24.856669 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846983    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:37:24.856669 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846986    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:37:24.856669 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846988    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:37:24.856669 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846990    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:37:24.856669 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846993    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:37:24.857431 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846995    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:37:24.857431 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.846998    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:37:24.857431 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.847002    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:37:24.857431 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.847005    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:37:24.857431 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.847008    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:37:24.857431 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.847010    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:37:24.857431 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.847013    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:37:24.857431 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.847016    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:37:24.857431 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.847018    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:37:24.857431 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.847021    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:37:24.857431 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.847024    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:37:24.857431 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.847027    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:37:24.857431 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.847029    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:37:24.857431 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.847031    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:37:24.857431 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.847034    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:37:24.857431 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.847038    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:37:24.857431 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.847040    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:37:24.857431 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.847043    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:37:24.857431 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.847045    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:37:24.858175 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.847048    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:37:24.858175 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.847051    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:37:24.858175 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.847053    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:37:24.858175 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.847056    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:37:24.858175 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.847058    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:37:24.858175 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.847061    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:37:24.858175 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.847063    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:37:24.858175 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.847066    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:37:24.858175 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:24.847068    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:37:24.858175 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.847121    2576 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 20:37:24.858175 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.849076    2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 20:37:24.858175 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.849901    2576 server.go:1014] "Starting client certificate rotation"
Oct 30 20:37:24.858175 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.849993    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 20:37:24.858175 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.850681    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 20:37:24.858175 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.852047    2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 20:37:24.858175 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.853162    2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 20:37:24.865162 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.865145    2576 log.go:25] "Validated CRI v1 runtime API"
Oct 30 20:37:24.868078 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.868064    2576 log.go:25] "Validated CRI v1 image API"
Oct 30 20:37:24.868931 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.868904    2576 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 20:37:24.876475 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.876455    2576 fs.go:135] Filesystem UUIDs: map[402345bb-b82b-4c5c-b016-5c2986310405:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d0a89e01-143a-4f81-b78e-46622c485f15:/dev/nvme0n1p3]
Oct 30 20:37:24.876528 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.876475    2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 20:37:24.882119 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.882013    2576 manager.go:217] Machine: {Timestamp:2026-10-30 20:37:24.880888635 +0000 UTC m=+0.298163325 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3072946 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2060fc5e4a2a4e0051100d8a2bad43 SystemUUID:ec2060fc-5e4a-2a4e-0051-100d8a2bad43 BootID:84fae1cf-a661-4325-bd4a-15278dc62013 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:3f:87:41:db:bf Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:3f:87:41:db:bf Speed:0 Mtu:9001} {Name:ovs-system MacAddress:f6:c3:91:ce:73:f5 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 20:37:24.882205 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.882121    2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 20:37:24.882205 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.882193    2576 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 20:37:24.884951 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.884924    2576 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 20:37:24.885075 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.884953    2576 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-165.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 20:37:24.885117 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.885077    2576 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 20:37:24.885117 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.885085    2576 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 20:37:24.885176 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.885125    2576 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 20:37:24.885176 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.885142    2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 20:37:24.885870 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.885860    2576 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 20:37:24.889454 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.889443    2576 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 20:37:24.889585 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.889576    2576 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 20:37:24.890217 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.890207    2576 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 20:37:24.890273 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.890223    2576 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 20:37:24.890273 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.890241    2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 20:37:24.890273 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.890253    2576 kubelet.go:405] "Adding apiserver pod source"
Oct 30 20:37:24.890273 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.890261    2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 20:37:24.890590 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.890578    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 20:37:24.892921 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.892889    2576 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 20:37:24.898013 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.897995    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 20:37:24.898738 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.898722    2576 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 20:37:24.898828 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.898754    2576 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 20:37:24.898936 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.898925    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 20:37:24.898967 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.898942    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 20:37:24.898967 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.898949    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 20:37:24.898967 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.898954    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 20:37:24.898967 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.898959    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 20:37:24.898967 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.898965    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 20:37:24.899498 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.899482    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 20:37:24.899498 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.899492    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 20:37:24.899498 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.899499    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 20:37:24.899589 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.899505    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 20:37:24.899589 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.899514    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 20:37:24.899589 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.899523    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 20:37:24.900083 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.900073    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 20:37:24.900114 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.900086    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 20:37:24.903799 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.903776    2576 server.go:1263] "Started kubelet"
Oct 30 20:37:24.903886 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.903851    2576 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 20:37:24.903985 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.903875    2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 20:37:24.903985 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.903969    2576 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 20:37:24.904487 ip-10-0-128-165 systemd[1]: Started Kubernetes Kubelet.
Oct 30 20:37:24.905140 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.905082    2576 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 20:37:24.907042 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.906938    2576 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 20:37:24.910107 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:24.910089    2576 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 20:37:24.910313 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.910296    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 20:37:24.910746 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.910730    2576 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 20:37:24.911095 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:24.911077    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-165.us-west-1.compute.internal\" not found"
Oct 30 20:37:24.911386 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.911369    2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 20:37:24.911386 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.911373    2576 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 20:37:24.911513 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.911391    2576 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 20:37:24.911513 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.911502    2576 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 20:37:24.911610 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.911523    2576 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 20:37:24.911749 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.911738    2576 factory.go:153] Registering CRI-O factory
Oct 30 20:37:24.911819 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.911755    2576 factory.go:223] Registration of the crio container factory successfully
Oct 30 20:37:24.911867 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.911828    2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 20:37:24.911867 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.911839    2576 factory.go:55] Registering systemd factory
Oct 30 20:37:24.911867 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.911845    2576 factory.go:223] Registration of the systemd container factory successfully
Oct 30 20:37:24.911867 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.911857    2576 factory.go:103] Registering Raw factory
Oct 30 20:37:24.911867 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.911866    2576 manager.go:1199] Started watching for new ooms in manager
Oct 30 20:37:24.912747 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.912736    2576 manager.go:319] Starting recovery of all containers
Oct 30 20:37:24.924311 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.924288    2576 manager.go:324] Recovery completed
Oct 30 20:37:24.928213 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.928199    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:37:24.930404 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.930385    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-165.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:37:24.930474 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.930423    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-165.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:37:24.930474 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.930435    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-165.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:37:24.930962 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.930947    2576 cpu_manager.go:226] "Starting" policy="none"
Oct 30 20:37:24.931010 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.930961    2576 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 20:37:24.931010 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.930977    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:37:24.931010 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.930982    2576 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 20:37:24.931661 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.931645    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-165.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:37:24.931694 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.931673    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-165.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:37:24.931694 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.931683    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-165.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:37:24.933299 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.933287    2576 policy_none.go:50] "Start"
Oct 30 20:37:24.933345 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.933309    2576 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 20:37:24.933345 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.933320    2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 20:37:24.935310 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.935299    2576 policy_none.go:44] "Start"
Oct 30 20:37:24.974829 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.974777    2576 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 20:37:24.977667 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.977655    2576 manager.go:341] "Starting Device Plugin manager"
Oct 30 20:37:24.977732 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:24.977681    2576 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 20:37:24.977732 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.977690    2576 server.go:87] "Starting device plugin registration server"
Oct 30 20:37:24.978007 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.977995    2576 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 20:37:24.978040 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.978010    2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 20:37:24.978120 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.978102    2576 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 20:37:24.978217 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.978197    2576 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 20:37:24.978217 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:24.978214    2576 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 20:37:24.978660 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:24.978644    2576 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 20:37:24.978735 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:24.978695    2576 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-165.us-west-1.compute.internal\" not found"
Oct 30 20:37:25.049178 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:25.049152    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 20:37:25.050848 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:25.050465    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 20:37:25.050848 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:25.050483    2576 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 20:37:25.050848 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:25.050499    2576 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 20:37:25.050848 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:25.050530    2576 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 20:37:25.078158 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:25.078079    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:37:25.078943 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:25.078924    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-165.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:37:25.079046 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:25.078954    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-165.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:37:25.079046 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:25.078963    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-165.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:37:25.079046 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:25.078989    2576 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-165.us-west-1.compute.internal"
Oct 30 20:37:25.127935 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:25.127915    2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 20:37:25.128034 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:25.127957    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fdvzz"
Oct 30 20:37:25.151595 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:25.151572    2576 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-165.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-165.us-west-1.compute.internal"]
Oct 30 20:37:25.151646 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:25.151641    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:37:25.152522 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:25.152500    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-165.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:37:25.152649 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:25.152537    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-165.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:37:25.152649 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:25.152552    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-165.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:37:25.154508 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:25.154492    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:37:25.155153 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:25.155138    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-165.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:37:25.155252 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:25.155160    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-165.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:37:25.155252 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:25.155175    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-165.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:37:25.155392 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:25.155376    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-165.us-west-1.compute.internal"
Oct 30 20:37:25.155452 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:25.155414    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:37:25.156157 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:25.156143    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-165.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:37:25.156231 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:25.156169    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-165.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:37:25.156231 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:25.156180    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-165.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:37:25.156956 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:25.156938    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-165.us-west-1.compute.internal"
Oct 30 20:37:25.157033 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:25.156968    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:37:25.157586 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:25.157572    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-165.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:37:25.157642 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:25.157598    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-165.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:37:25.157642 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:25.157612    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-165.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:37:25.189226 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:25.189207    2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-165.us-west-1.compute.internal\" not found" node="ip-10-0-128-165.us-west-1.compute.internal"
Oct 30 20:37:25.192000 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:25.191984    2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-165.us-west-1.compute.internal\" not found" node="ip-10-0-128-165.us-west-1.compute.internal"
Oct 30 20:37:25.204237 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:25.204214    2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:37:25.206521 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:25.206507    2576 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:37:25.212559 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:25.212539    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ccc9dd125588c9f50e9b9a79ea47623e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-165.us-west-1.compute.internal\" (UID: \"ccc9dd125588c9f50e9b9a79ea47623e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-165.us-west-1.compute.internal"
Oct 30 20:37:25.212623 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:25.212567    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/40ad3a6a5fd37a90a3238925c9fca019-config\") pod \"kube-apiserver-proxy-ip-10-0-128-165.us-west-1.compute.internal\" (UID: \"40ad3a6a5fd37a90a3238925c9fca019\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-165.us-west-1.compute.internal"
Oct 30 20:37:25.212623 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:25.212583    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ccc9dd125588c9f50e9b9a79ea47623e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-165.us-west-1.compute.internal\" (UID: \"ccc9dd125588c9f50e9b9a79ea47623e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-165.us-west-1.compute.internal"
Oct 30 20:37:25.253521 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:25.251748    2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:37:25.302723 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:25.302697    2576 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:37:25.302955 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:25.302929    2576 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-165.us-west-1.compute.internal"
Oct 30 20:37:25.311843 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:25.311819    2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-165.us-west-1.compute.internal"
Oct 30 20:37:25.312989 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:25.312963    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ccc9dd125588c9f50e9b9a79ea47623e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-165.us-west-1.compute.internal\" (UID: \"ccc9dd125588c9f50e9b9a79ea47623e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-165.us-west-1.compute.internal"
Oct 30 20:37:25.313099 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:25.312990    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ccc9dd125588c9f50e9b9a79ea47623e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-165.us-west-1.compute.internal\" (UID: \"ccc9dd125588c9f50e9b9a79ea47623e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-165.us-west-1.compute.internal"
Oct 30 20:37:25.313099 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:25.313008    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/40ad3a6a5fd37a90a3238925c9fca019-config\") pod \"kube-apiserver-proxy-ip-10-0-128-165.us-west-1.compute.internal\" (UID: \"40ad3a6a5fd37a90a3238925c9fca019\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-165.us-west-1.compute.internal"
Oct 30 20:37:25.313099 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:25.313050    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/40ad3a6a5fd37a90a3238925c9fca019-config\") pod \"kube-apiserver-proxy-ip-10-0-128-165.us-west-1.compute.internal\" (UID: \"40ad3a6a5fd37a90a3238925c9fca019\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-165.us-west-1.compute.internal"
Oct 30 20:37:25.313099 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:25.313085    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ccc9dd125588c9f50e9b9a79ea47623e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-165.us-west-1.compute.internal\" (UID: \"ccc9dd125588c9f50e9b9a79ea47623e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-165.us-west-1.compute.internal"
Oct 30 20:37:25.313248 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:25.313087    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ccc9dd125588c9f50e9b9a79ea47623e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-165.us-west-1.compute.internal\" (UID: \"ccc9dd125588c9f50e9b9a79ea47623e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-165.us-west-1.compute.internal"
Oct 30 20:37:25.376272 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:25.376215    2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 20:37:25.377010 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:25.376998    2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-165.us-west-1.compute.internal"
Oct 30 20:37:25.435898 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:25.435881    2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 20:37:25.492543 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:25.492521    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-165.us-west-1.compute.internal"
Oct 30 20:37:25.495343 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:25.495326    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-165.us-west-1.compute.internal"
Oct 30 20:37:25.849593 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:25.849403    2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 20:37:25.850156 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:25.849731    2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 20:37:25.850156 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:25.849767    2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 20:37:25.850156 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:25.849828    2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 20:37:25.850156 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:25.849837    2576 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a3138b275a5484ea19a8d9971322f756-b97bd68a7b677e68.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-128-165.us-west-1.compute.internal.18e3691bd7e060d2\": read tcp 10.0.128.165:45930->3.130.166.39:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-128-165.us-west-1.compute.internal.18e3691bd7e060d2  default   5513 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-165.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-128-165.us-west-1.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-128-165.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 20:37:24 +0000 UTC,LastTimestamp:2026-10-30 20:37:25.078958384 +0000 UTC m=+0.496233054,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-165.us-west-1.compute.internal,}"
Oct 30 20:37:25.890808 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:25.890764    2576 apiserver.go:52] "Watching apiserver"
Oct 30 20:37:25.911363 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:25.911334    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 20:37:26.131839 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.131729    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 20:32:25 +0000 UTC" deadline="2028-07-25 12:44:22.395952142 +0000 UTC"
Oct 30 20:37:26.131839 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.131760    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15208h6m56.264194914s"
Oct 30 20:37:26.152150 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.152126    2576 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 20:37:26.152509 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.152480    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-brmwl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-165.us-west-1.compute.internal","openshift-multus/multus-h4rs7","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvcbb","openshift-image-registry/node-ca-5clg5","openshift-network-operator/iptables-alerter-bpgsn","kube-system/global-pull-secret-syncer-xj584","openshift-multus/multus-additional-cni-plugins-4gl7z","openshift-multus/network-metrics-daemon-hjssq","openshift-network-diagnostics/network-check-target-kbpjj","kube-system/kube-apiserver-proxy-ip-10-0-128-165.us-west-1.compute.internal","openshift-dns/node-resolver-d69bq","openshift-ovn-kubernetes/ovnkube-node-r7lpf","kube-system/konnectivity-agent-lkxfb"]
Oct 30 20:37:26.154913 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.154893    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:37:26.156902 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.156881    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-h4rs7"
Oct 30 20:37:26.158873 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.158854    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvcbb"
Oct 30 20:37:26.160898 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.160882    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5clg5"
Oct 30 20:37:26.162997 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.162980    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bpgsn"
Oct 30 20:37:26.165031 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.165017    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xj584"
Oct 30 20:37:26.165103 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:26.165071    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xj584" podUID="1a74da60-0284-410b-9d69-0d5192ce0928"
Oct 30 20:37:26.167067 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.167049    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4gl7z"
Oct 30 20:37:26.169109 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.169093    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjssq"
Oct 30 20:37:26.169179 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:26.169150    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hjssq" podUID="242a67ca-f9ea-431d-adee-ece061468a02"
Oct 30 20:37:26.171287 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.171057    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbpjj"
Oct 30 20:37:26.171287 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:26.171115    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbpjj" podUID="e54d0918-84e8-40f3-9b18-6e2fafb3d500"
Oct 30 20:37:26.173159 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.173141    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-d69bq"
Oct 30 20:37:26.175420 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.175403    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:37:26.177495 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.177474    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lkxfb"
Oct 30 20:37:26.207717 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.207687    2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 20:37:26.207952 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.207774    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-ktfst"
Oct 30 20:37:26.208753 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.208728    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:37:26.208863 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.208822    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 20:37:26.208863 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.208849    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 20:37:26.209140 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.209122    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-86vxf\""
Oct 30 20:37:26.209182 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.209171    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 20:37:26.209285 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.209273    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 20:37:26.209358 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.209319    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 20:37:26.209457 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.209445    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-rv75s\""
Oct 30 20:37:26.210402 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.209758    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 20:37:26.210402 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.210042    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 20:37:26.210402 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.210098    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 20:37:26.210858 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.210837    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-94s75\""
Oct 30 20:37:26.211867 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.211850    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 20:37:26.212180 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.212166    2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 20:37:26.219129 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.219107    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1a74da60-0284-410b-9d69-0d5192ce0928-dbus\") pod \"global-pull-secret-syncer-xj584\" (UID: \"1a74da60-0284-410b-9d69-0d5192ce0928\") " pod="kube-system/global-pull-secret-syncer-xj584"
Oct 30 20:37:26.219221 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.219142    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-host-cni-bin\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:37:26.219221 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.219162    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b7f60156-3771-486a-a9d2-7acd5ab9f475-host\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:37:26.219221 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.219187    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/08ddd17c-0134-45dc-8a2f-aca6e3668e9f-cni-binary-copy\") pod \"multus-additional-cni-plugins-4gl7z\" (UID: \"08ddd17c-0134-45dc-8a2f-aca6e3668e9f\") " pod="openshift-multus/multus-additional-cni-plugins-4gl7z"
Oct 30 20:37:26.219221 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.219212    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/08ddd17c-0134-45dc-8a2f-aca6e3668e9f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4gl7z\" (UID: \"08ddd17c-0134-45dc-8a2f-aca6e3668e9f\") " pod="openshift-multus/multus-additional-cni-plugins-4gl7z"
Oct 30 20:37:26.219374 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.219234    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-host-kubelet\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:37:26.219374 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.219257    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-systemd-units\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:37:26.219374 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.219280    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-var-lib-openvswitch\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:37:26.219374 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.219329    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c3704770-6fb6-4086-8a22-9b29468d5a70-agent-certs\") pod \"konnectivity-agent-lkxfb\" (UID: \"c3704770-6fb6-4086-8a22-9b29468d5a70\") " pod="kube-system/konnectivity-agent-lkxfb"
Oct 30 20:37:26.219374 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.219352    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b7f60156-3771-486a-a9d2-7acd5ab9f475-run\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:37:26.219374 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.219367    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b7f60156-3771-486a-a9d2-7acd5ab9f475-tmp\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:37:26.219551 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.219382    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-cni-binary-copy\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:37:26.219551 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.219402    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frzdn\" (UniqueName: \"kubernetes.io/projected/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-kube-api-access-frzdn\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:37:26.219551 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.219426    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a7b1237b-e022-4570-9c48-4a8449996c69-registration-dir\") pod \"aws-ebs-csi-driver-node-nvcbb\" (UID: \"a7b1237b-e022-4570-9c48-4a8449996c69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvcbb"
Oct 30 20:37:26.219551 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.219446    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1a74da60-0284-410b-9d69-0d5192ce0928-kubelet-config\") pod \"global-pull-secret-syncer-xj584\" (UID: \"1a74da60-0284-410b-9d69-0d5192ce0928\") " pod="kube-system/global-pull-secret-syncer-xj584"
Oct 30 20:37:26.219551 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.219474    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b7f60156-3771-486a-a9d2-7acd5ab9f475-etc-kubernetes\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:37:26.219551 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.219499    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7f60156-3771-486a-a9d2-7acd5ab9f475-sys\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:37:26.219551 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.219526    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-host-var-lib-kubelet\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:37:26.219774 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.219553    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c7a31d20-2bf3-4c38-8816-2181571a1fda-hosts-file\") pod \"node-resolver-d69bq\" (UID: \"c7a31d20-2bf3-4c38-8816-2181571a1fda\") " pod="openshift-dns/node-resolver-d69bq"
Oct 30 20:37:26.219774 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.219592    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4m4z\" (UniqueName: \"kubernetes.io/projected/242a67ca-f9ea-431d-adee-ece061468a02-kube-api-access-k4m4z\") pod \"network-metrics-daemon-hjssq\" (UID: \"242a67ca-f9ea-431d-adee-ece061468a02\") " pod="openshift-multus/network-metrics-daemon-hjssq"
Oct 30 20:37:26.219774 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.219653    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b7f60156-3771-486a-a9d2-7acd5ab9f475-etc-sysctl-d\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:37:26.219774 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.219672    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b7f60156-3771-486a-a9d2-7acd5ab9f475-etc-sysctl-conf\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:37:26.219774 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.219688    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b7f60156-3771-486a-a9d2-7acd5ab9f475-lib-modules\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:37:26.219774 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.219704    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-multus-socket-dir-parent\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:37:26.219774 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.219719    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a7b1237b-e022-4570-9c48-4a8449996c69-sys-fs\") pod \"aws-ebs-csi-driver-node-nvcbb\" (UID: \"a7b1237b-e022-4570-9c48-4a8449996c69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvcbb"
Oct 30 20:37:26.219774 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.219737    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl5v9\" (UniqueName: \"kubernetes.io/projected/08ddd17c-0134-45dc-8a2f-aca6e3668e9f-kube-api-access-sl5v9\") pod \"multus-additional-cni-plugins-4gl7z\" (UID: \"08ddd17c-0134-45dc-8a2f-aca6e3668e9f\") " pod="openshift-multus/multus-additional-cni-plugins-4gl7z"
Oct 30 20:37:26.219774 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.219753    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-system-cni-dir\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:37:26.219774 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.219766    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-host-var-lib-cni-bin\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:37:26.220131 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.219782    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/08ddd17c-0134-45dc-8a2f-aca6e3668e9f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4gl7z\" (UID: \"08ddd17c-0134-45dc-8a2f-aca6e3668e9f\") " pod="openshift-multus/multus-additional-cni-plugins-4gl7z"
Oct 30 20:37:26.220131 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.219832    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:37:26.220131 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.219859    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-host-run-k8s-cni-cncf-io\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:37:26.220131 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.219876    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a7b1237b-e022-4570-9c48-4a8449996c69-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nvcbb\" (UID: \"a7b1237b-e022-4570-9c48-4a8449996c69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvcbb"
Oct 30 20:37:26.220131 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.219891    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl4qd\" (UniqueName: \"kubernetes.io/projected/764461d5-e0b2-4856-a97e-0c5d3d34a497-kube-api-access-nl4qd\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:37:26.220131 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.219930    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-multus-daemon-config\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:37:26.220131 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.219951    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-host-run-multus-certs\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:37:26.220131 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.219966    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/88a8166b-aa0c-41dc-8bf3-92708c60df57-host\") pod \"node-ca-5clg5\" (UID: \"88a8166b-aa0c-41dc-8bf3-92708c60df57\") " pod="openshift-image-registry/node-ca-5clg5"
Oct 30 20:37:26.220131 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.219999    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/88a8166b-aa0c-41dc-8bf3-92708c60df57-serviceca\") pod \"node-ca-5clg5\" (UID: \"88a8166b-aa0c-41dc-8bf3-92708c60df57\") " pod="openshift-image-registry/node-ca-5clg5"
Oct 30 20:37:26.220131 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.220025    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/08ddd17c-0134-45dc-8a2f-aca6e3668e9f-cnibin\") pod \"multus-additional-cni-plugins-4gl7z\" (UID: \"08ddd17c-0134-45dc-8a2f-aca6e3668e9f\") " pod="openshift-multus/multus-additional-cni-plugins-4gl7z"
Oct 30 20:37:26.220131 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.220054    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-log-socket\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:37:26.220131 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.220081    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b7f60156-3771-486a-a9d2-7acd5ab9f475-etc-systemd\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:37:26.220131 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.220096    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-multus-conf-dir\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:37:26.220131 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.220118    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/08ddd17c-0134-45dc-8a2f-aca6e3668e9f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4gl7z\" (UID: \"08ddd17c-0134-45dc-8a2f-aca6e3668e9f\") " pod="openshift-multus/multus-additional-cni-plugins-4gl7z"
Oct 30 20:37:26.220751 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.220142    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzhsg\" (UniqueName: \"kubernetes.io/projected/e54d0918-84e8-40f3-9b18-6e2fafb3d500-kube-api-access-bzhsg\") pod \"network-check-target-kbpjj\" (UID: \"e54d0918-84e8-40f3-9b18-6e2fafb3d500\") " pod="openshift-network-diagnostics/network-check-target-kbpjj"
Oct 30 20:37:26.220751 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.220159    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-etc-openvswitch\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:37:26.220751 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.220177    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-host-run-ovn-kubernetes\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:37:26.220751 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.220198    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b7f60156-3771-486a-a9d2-7acd5ab9f475-etc-tuned\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:37:26.220751 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.220214    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6qnn\" (UniqueName: \"kubernetes.io/projected/b7f60156-3771-486a-a9d2-7acd5ab9f475-kube-api-access-k6qnn\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:37:26.220751 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.220249    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftcfw\" (UniqueName: \"kubernetes.io/projected/0fb51194-9a91-4430-8ded-4844ffb6727a-kube-api-access-ftcfw\") pod \"iptables-alerter-bpgsn\" (UID: \"0fb51194-9a91-4430-8ded-4844ffb6727a\") " pod="openshift-network-operator/iptables-alerter-bpgsn"
Oct 30 20:37:26.220751 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.220286    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/242a67ca-f9ea-431d-adee-ece061468a02-metrics-certs\") pod \"network-metrics-daemon-hjssq\" (UID: \"242a67ca-f9ea-431d-adee-ece061468a02\") " pod="openshift-multus/network-metrics-daemon-hjssq"
Oct 30 20:37:26.220751 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.220313    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-host-slash\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:37:26.220751 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.220346    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-run-openvswitch\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:37:26.220751 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.220369    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-host-cni-netd\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:37:26.220751 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.220394    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a7b1237b-e022-4570-9c48-4a8449996c69-socket-dir\") pod \"aws-ebs-csi-driver-node-nvcbb\" (UID: \"a7b1237b-e022-4570-9c48-4a8449996c69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvcbb"
Oct 30 20:37:26.220751 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.220415    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a7b1237b-e022-4570-9c48-4a8449996c69-device-dir\") pod \"aws-ebs-csi-driver-node-nvcbb\" (UID: \"a7b1237b-e022-4570-9c48-4a8449996c69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvcbb"
Oct 30 20:37:26.220751 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.220438    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z96wz\" (UniqueName: \"kubernetes.io/projected/88a8166b-aa0c-41dc-8bf3-92708c60df57-kube-api-access-z96wz\") pod \"node-ca-5clg5\" (UID: \"88a8166b-aa0c-41dc-8bf3-92708c60df57\") " pod="openshift-image-registry/node-ca-5clg5"
Oct 30 20:37:26.220751 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.220476    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0fb51194-9a91-4430-8ded-4844ffb6727a-iptables-alerter-script\") pod \"iptables-alerter-bpgsn\" (UID: \"0fb51194-9a91-4430-8ded-4844ffb6727a\") " pod="openshift-network-operator/iptables-alerter-bpgsn"
Oct 30 20:37:26.220751 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.220496    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0fb51194-9a91-4430-8ded-4844ffb6727a-host-slash\") pod \"iptables-alerter-bpgsn\" (UID: \"0fb51194-9a91-4430-8ded-4844ffb6727a\") " pod="openshift-network-operator/iptables-alerter-bpgsn"
Oct 30 20:37:26.220751 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.220510    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-run-systemd\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:37:26.221408 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.220524    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-node-log\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:37:26.221408 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.220545    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/764461d5-e0b2-4856-a97e-0c5d3d34a497-env-overrides\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:37:26.221408 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.220577    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c3704770-6fb6-4086-8a22-9b29468d5a70-konnectivity-ca\") pod \"konnectivity-agent-lkxfb\" (UID: \"c3704770-6fb6-4086-8a22-9b29468d5a70\") " pod="kube-system/konnectivity-agent-lkxfb"
Oct 30 20:37:26.221408 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.220604    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b7f60156-3771-486a-a9d2-7acd5ab9f475-etc-modprobe-d\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:37:26.221408 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.220632    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7f60156-3771-486a-a9d2-7acd5ab9f475-var-lib-kubelet\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:37:26.221408 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.220663    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-host-run-netns\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:37:26.221408 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.220693    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-host-var-lib-cni-multus\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:37:26.221408 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.220714    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a7b1237b-e022-4570-9c48-4a8449996c69-etc-selinux\") pod \"aws-ebs-csi-driver-node-nvcbb\" (UID: \"a7b1237b-e022-4570-9c48-4a8449996c69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvcbb"
Oct 30 20:37:26.221408 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.220738    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqm5k\" (UniqueName: \"kubernetes.io/projected/c7a31d20-2bf3-4c38-8816-2181571a1fda-kube-api-access-jqm5k\") pod \"node-resolver-d69bq\" (UID: \"c7a31d20-2bf3-4c38-8816-2181571a1fda\") " pod="openshift-dns/node-resolver-d69bq"
Oct 30 20:37:26.221408 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.220761    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a74da60-0284-410b-9d69-0d5192ce0928-original-pull-secret\") pod \"global-pull-secret-syncer-xj584\" (UID: \"1a74da60-0284-410b-9d69-0d5192ce0928\") " pod="kube-system/global-pull-secret-syncer-xj584"
Oct 30 20:37:26.221408 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.220803    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/764461d5-e0b2-4856-a97e-0c5d3d34a497-ovnkube-config\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:37:26.221408 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.220828    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/764461d5-e0b2-4856-a97e-0c5d3d34a497-ovn-node-metrics-cert\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:37:26.221408 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.220849    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b7f60156-3771-486a-a9d2-7acd5ab9f475-etc-sysconfig\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:37:26.221408 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.220873    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-host-run-netns\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:37:26.221408 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.220920    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-hostroot\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:37:26.221408 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.220944    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-etc-kubernetes\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:37:26.221898 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.220970    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkjkh\" (UniqueName: \"kubernetes.io/projected/a7b1237b-e022-4570-9c48-4a8449996c69-kube-api-access-xkjkh\") pod \"aws-ebs-csi-driver-node-nvcbb\" (UID: \"a7b1237b-e022-4570-9c48-4a8449996c69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvcbb"
Oct 30 20:37:26.221898 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.220994    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/764461d5-e0b2-4856-a97e-0c5d3d34a497-ovnkube-script-lib\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:37:26.221898 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.221020    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08ddd17c-0134-45dc-8a2f-aca6e3668e9f-system-cni-dir\") pod \"multus-additional-cni-plugins-4gl7z\" (UID: \"08ddd17c-0134-45dc-8a2f-aca6e3668e9f\") " pod="openshift-multus/multus-additional-cni-plugins-4gl7z"
Oct 30 20:37:26.221898 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.221054    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7a31d20-2bf3-4c38-8816-2181571a1fda-tmp-dir\") pod \"node-resolver-d69bq\" (UID: \"c7a31d20-2bf3-4c38-8816-2181571a1fda\") " pod="openshift-dns/node-resolver-d69bq"
Oct 30 20:37:26.221898 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.221108    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-run-ovn\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:37:26.221898 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.221133    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-multus-cni-dir\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:37:26.221898 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.221155    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-cnibin\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:37:26.312736 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.312710    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 20:37:26.312939 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.312755    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:37:26.312939 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.312859    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 20:37:26.312939 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.312899    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 20:37:26.313072 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.312960    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 20:37:26.313072 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.312989    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 20:37:26.313152 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.313078    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 20:37:26.313152 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.313110    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jwfqc\""
Oct 30 20:37:26.313152 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.313116    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-sr85n\""
Oct 30 20:37:26.313246 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.313189    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 20:37:26.313246 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.313200    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 20:37:26.313246 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.313201    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 20:37:26.321061 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.321045    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-g2pwr\""
Oct 30 20:37:26.321143 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.321074    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 20:37:26.322288 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.322258    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-multus-daemon-config\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:37:26.322345 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.322310    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 20:37:26.322345 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.322305    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-host-run-multus-certs\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:37:26.322447 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.322342    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 20:37:26.322447 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.322370    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 20:37:26.322447 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.322368    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-host-run-multus-certs\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:37:26.322447 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.322368    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/88a8166b-aa0c-41dc-8bf3-92708c60df57-host\") pod \"node-ca-5clg5\" (UID: \"88a8166b-aa0c-41dc-8bf3-92708c60df57\") " pod="openshift-image-registry/node-ca-5clg5"
Oct 30 20:37:26.322447 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.322408    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-c5chh\""
Oct 30 20:37:26.322447 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.322416    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 20:37:26.322447 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.322419    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/88a8166b-aa0c-41dc-8bf3-92708c60df57-host\") pod \"node-ca-5clg5\" (UID: \"88a8166b-aa0c-41dc-8bf3-92708c60df57\") " pod="openshift-image-registry/node-ca-5clg5"
Oct 30 20:37:26.322447 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.322432    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/88a8166b-aa0c-41dc-8bf3-92708c60df57-serviceca\") pod \"node-ca-5clg5\" (UID: \"88a8166b-aa0c-41dc-8bf3-92708c60df57\") " pod="openshift-image-registry/node-ca-5clg5"
Oct 30 20:37:26.322806 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.322455    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/08ddd17c-0134-45dc-8a2f-aca6e3668e9f-cnibin\") pod \"multus-additional-cni-plugins-4gl7z\" (UID: \"08ddd17c-0134-45dc-8a2f-aca6e3668e9f\") " pod="openshift-multus/multus-additional-cni-plugins-4gl7z"
Oct 30 20:37:26.322806 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.322473    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-log-socket\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:37:26.322806 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.322493    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b7f60156-3771-486a-a9d2-7acd5ab9f475-etc-systemd\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:37:26.322806 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.322503    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/08ddd17c-0134-45dc-8a2f-aca6e3668e9f-cnibin\") pod \"multus-additional-cni-plugins-4gl7z\" (UID: \"08ddd17c-0134-45dc-8a2f-aca6e3668e9f\") " pod="openshift-multus/multus-additional-cni-plugins-4gl7z"
Oct 30 20:37:26.322806 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.322515    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-multus-conf-dir\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:37:26.322806 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.322540    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/08ddd17c-0134-45dc-8a2f-aca6e3668e9f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4gl7z\" (UID: \"08ddd17c-0134-45dc-8a2f-aca6e3668e9f\") " pod="openshift-multus/multus-additional-cni-plugins-4gl7z"
Oct 30 20:37:26.322806 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.322543    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-log-socket\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:37:26.322806 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.322552    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b7f60156-3771-486a-a9d2-7acd5ab9f475-etc-systemd\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:37:26.322806 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.322565    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzhsg\" (UniqueName: \"kubernetes.io/projected/e54d0918-84e8-40f3-9b18-6e2fafb3d500-kube-api-access-bzhsg\") pod \"network-check-target-kbpjj\" (UID: \"e54d0918-84e8-40f3-9b18-6e2fafb3d500\") " pod="openshift-network-diagnostics/network-check-target-kbpjj"
Oct 30 20:37:26.322806 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.322590    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-etc-openvswitch\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:37:26.322806 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.322592    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-multus-conf-dir\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:37:26.322806 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.322622    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-host-run-ovn-kubernetes\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:37:26.322806 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.322634    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-etc-openvswitch\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:37:26.322806 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.322686    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-host-run-ovn-kubernetes\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:37:26.322806 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.322733    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/08ddd17c-0134-45dc-8a2f-aca6e3668e9f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4gl7z\" (UID: \"08ddd17c-0134-45dc-8a2f-aca6e3668e9f\") " pod="openshift-multus/multus-additional-cni-plugins-4gl7z"
Oct 30 20:37:26.322806 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.322772    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b7f60156-3771-486a-a9d2-7acd5ab9f475-etc-tuned\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:37:26.322806 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.322813    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6qnn\" (UniqueName: \"kubernetes.io/projected/b7f60156-3771-486a-a9d2-7acd5ab9f475-kube-api-access-k6qnn\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:37:26.323590 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.322835    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftcfw\" (UniqueName: \"kubernetes.io/projected/0fb51194-9a91-4430-8ded-4844ffb6727a-kube-api-access-ftcfw\") pod \"iptables-alerter-bpgsn\" (UID: \"0fb51194-9a91-4430-8ded-4844ffb6727a\") " pod="openshift-network-operator/iptables-alerter-bpgsn"
Oct 30 20:37:26.323590 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.322859    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/242a67ca-f9ea-431d-adee-ece061468a02-metrics-certs\") pod \"network-metrics-daemon-hjssq\" (UID: \"242a67ca-f9ea-431d-adee-ece061468a02\") " pod="openshift-multus/network-metrics-daemon-hjssq"
Oct 30 20:37:26.323590 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.322882    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-host-slash\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:37:26.323590 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.322904    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-run-openvswitch\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:37:26.323590 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.322911    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/88a8166b-aa0c-41dc-8bf3-92708c60df57-serviceca\") pod \"node-ca-5clg5\" (UID: \"88a8166b-aa0c-41dc-8bf3-92708c60df57\") " pod="openshift-image-registry/node-ca-5clg5"
Oct 30 20:37:26.323590 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.322926    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-host-cni-netd\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:37:26.323590 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.322963    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-host-slash\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:37:26.323590 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.322960    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a7b1237b-e022-4570-9c48-4a8449996c69-socket-dir\") pod \"aws-ebs-csi-driver-node-nvcbb\" (UID: \"a7b1237b-e022-4570-9c48-4a8449996c69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvcbb"
Oct 30 20:37:26.323590 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.323008    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a7b1237b-e022-4570-9c48-4a8449996c69-device-dir\") pod \"aws-ebs-csi-driver-node-nvcbb\" (UID: \"a7b1237b-e022-4570-9c48-4a8449996c69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvcbb"
Oct 30 20:37:26.323590 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.323015    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-run-openvswitch\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:37:26.323590 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:26.322986    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:37:26.323590 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.323032    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z96wz\" (UniqueName: \"kubernetes.io/projected/88a8166b-aa0c-41dc-8bf3-92708c60df57-kube-api-access-z96wz\") pod \"node-ca-5clg5\" (UID: \"88a8166b-aa0c-41dc-8bf3-92708c60df57\") " pod="openshift-image-registry/node-ca-5clg5"
Oct 30 20:37:26.323590 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.323057    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0fb51194-9a91-4430-8ded-4844ffb6727a-iptables-alerter-script\") pod \"iptables-alerter-bpgsn\" (UID: \"0fb51194-9a91-4430-8ded-4844ffb6727a\") " pod="openshift-network-operator/iptables-alerter-bpgsn"
Oct 30 20:37:26.323590 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.323062    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-multus-daemon-config\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:37:26.323590 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.323083    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a7b1237b-e022-4570-9c48-4a8449996c69-device-dir\") pod \"aws-ebs-csi-driver-node-nvcbb\" (UID: \"a7b1237b-e022-4570-9c48-4a8449996c69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvcbb"
Oct 30 20:37:26.323590 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:26.323109    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/242a67ca-f9ea-431d-adee-ece061468a02-metrics-certs podName:242a67ca-f9ea-431d-adee-ece061468a02 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:26.823067865 +0000 UTC m=+2.240342541 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/242a67ca-f9ea-431d-adee-ece061468a02-metrics-certs") pod "network-metrics-daemon-hjssq" (UID: "242a67ca-f9ea-431d-adee-ece061468a02") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:37:26.323590 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.323027    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-host-cni-netd\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:37:26.324418 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.323120    2576 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 20:37:26.324418 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.323145    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0fb51194-9a91-4430-8ded-4844ffb6727a-host-slash\") pod \"iptables-alerter-bpgsn\" (UID: \"0fb51194-9a91-4430-8ded-4844ffb6727a\") " pod="openshift-network-operator/iptables-alerter-bpgsn"
Oct 30 20:37:26.324418 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.323124    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a7b1237b-e022-4570-9c48-4a8449996c69-socket-dir\") pod \"aws-ebs-csi-driver-node-nvcbb\" (UID: \"a7b1237b-e022-4570-9c48-4a8449996c69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvcbb"
Oct 30 20:37:26.324418 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.323129    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0fb51194-9a91-4430-8ded-4844ffb6727a-host-slash\") pod \"iptables-alerter-bpgsn\" (UID: \"0fb51194-9a91-4430-8ded-4844ffb6727a\") " pod="openshift-network-operator/iptables-alerter-bpgsn"
Oct 30 20:37:26.324418 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.323189    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-run-systemd\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:37:26.324418 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.323210    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-node-log\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:37:26.324418 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.323230    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/764461d5-e0b2-4856-a97e-0c5d3d34a497-env-overrides\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:37:26.324418 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.323234    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-run-systemd\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:37:26.324418 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.323253    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c3704770-6fb6-4086-8a22-9b29468d5a70-konnectivity-ca\") pod \"konnectivity-agent-lkxfb\" (UID: \"c3704770-6fb6-4086-8a22-9b29468d5a70\") " pod="kube-system/konnectivity-agent-lkxfb"
Oct 30 20:37:26.324418 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.323264    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-node-log\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:37:26.324418 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.323275    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b7f60156-3771-486a-a9d2-7acd5ab9f475-etc-modprobe-d\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:37:26.324418 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.323299    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7f60156-3771-486a-a9d2-7acd5ab9f475-var-lib-kubelet\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:37:26.324418 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.323320    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-host-run-netns\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:37:26.324418 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.323345    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-host-var-lib-cni-multus\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:37:26.324418 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.323355    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b7f60156-3771-486a-a9d2-7acd5ab9f475-etc-modprobe-d\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:37:26.324418 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.323370    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a7b1237b-e022-4570-9c48-4a8449996c69-etc-selinux\") pod \"aws-ebs-csi-driver-node-nvcbb\" (UID: \"a7b1237b-e022-4570-9c48-4a8449996c69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvcbb"
Oct 30 20:37:26.324418 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.323394    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqm5k\" (UniqueName: \"kubernetes.io/projected/c7a31d20-2bf3-4c38-8816-2181571a1fda-kube-api-access-jqm5k\") pod \"node-resolver-d69bq\" (UID: \"c7a31d20-2bf3-4c38-8816-2181571a1fda\") " pod="openshift-dns/node-resolver-d69bq"
Oct 30 20:37:26.324418 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.323402    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-host-run-netns\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:37:26.325268 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.323416    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a74da60-0284-410b-9d69-0d5192ce0928-original-pull-secret\") pod \"global-pull-secret-syncer-xj584\" (UID: \"1a74da60-0284-410b-9d69-0d5192ce0928\") " pod="kube-system/global-pull-secret-syncer-xj584"
Oct 30 20:37:26.325268 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.323441    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/764461d5-e0b2-4856-a97e-0c5d3d34a497-ovnkube-config\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:37:26.325268 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.323449    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7f60156-3771-486a-a9d2-7acd5ab9f475-var-lib-kubelet\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:37:26.325268 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.323503    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/764461d5-e0b2-4856-a97e-0c5d3d34a497-ovn-node-metrics-cert\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:37:26.325268 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.323518    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a7b1237b-e022-4570-9c48-4a8449996c69-etc-selinux\") pod \"aws-ebs-csi-driver-node-nvcbb\" (UID: \"a7b1237b-e022-4570-9c48-4a8449996c69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvcbb"
Oct 30 20:37:26.325268 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.323531    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b7f60156-3771-486a-a9d2-7acd5ab9f475-etc-sysconfig\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:37:26.325268 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.323559    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-host-run-netns\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:37:26.325268 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.323570    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0fb51194-9a91-4430-8ded-4844ffb6727a-iptables-alerter-script\") pod \"iptables-alerter-bpgsn\" (UID: \"0fb51194-9a91-4430-8ded-4844ffb6727a\") " pod="openshift-network-operator/iptables-alerter-bpgsn"
Oct 30 20:37:26.325268 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.323586    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-hostroot\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:37:26.325268 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.323598    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-host-var-lib-cni-multus\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:37:26.325268 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.323626    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-etc-kubernetes\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:37:26.325268 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.323643    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/764461d5-e0b2-4856-a97e-0c5d3d34a497-env-overrides\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:37:26.325268 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.323655    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-hostroot\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:37:26.325268 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.323664    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkjkh\" (UniqueName: \"kubernetes.io/projected/a7b1237b-e022-4570-9c48-4a8449996c69-kube-api-access-xkjkh\") pod \"aws-ebs-csi-driver-node-nvcbb\" (UID: \"a7b1237b-e022-4570-9c48-4a8449996c69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvcbb"
Oct 30 20:37:26.325268 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.323686    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-etc-kubernetes\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:37:26.325268 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.323696    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/764461d5-e0b2-4856-a97e-0c5d3d34a497-ovnkube-script-lib\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:37:26.325268 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.323725    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08ddd17c-0134-45dc-8a2f-aca6e3668e9f-system-cni-dir\") pod \"multus-additional-cni-plugins-4gl7z\" (UID: \"08ddd17c-0134-45dc-8a2f-aca6e3668e9f\") " pod="openshift-multus/multus-additional-cni-plugins-4gl7z"
Oct 30 20:37:26.326034 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.323733    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b7f60156-3771-486a-a9d2-7acd5ab9f475-etc-sysconfig\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:37:26.326034 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.323754    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7a31d20-2bf3-4c38-8816-2181571a1fda-tmp-dir\") pod \"node-resolver-d69bq\" (UID: \"c7a31d20-2bf3-4c38-8816-2181571a1fda\") " pod="openshift-dns/node-resolver-d69bq"
Oct 30 20:37:26.326034 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:26.323771    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 20:37:26.326034 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.323808    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-run-ovn\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:37:26.326034 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.323840    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-multus-cni-dir\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:37:26.326034 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:26.323855    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a74da60-0284-410b-9d69-0d5192ce0928-original-pull-secret podName:1a74da60-0284-410b-9d69-0d5192ce0928 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:26.823835066 +0000 UTC m=+2.241109732 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a74da60-0284-410b-9d69-0d5192ce0928-original-pull-secret") pod "global-pull-secret-syncer-xj584" (UID: "1a74da60-0284-410b-9d69-0d5192ce0928") : object "kube-system"/"original-pull-secret" not registered
Oct 30 20:37:26.326034 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.323885    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-host-run-netns\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:37:26.326034 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.323904    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08ddd17c-0134-45dc-8a2f-aca6e3668e9f-system-cni-dir\") pod \"multus-additional-cni-plugins-4gl7z\" (UID: \"08ddd17c-0134-45dc-8a2f-aca6e3668e9f\") " pod="openshift-multus/multus-additional-cni-plugins-4gl7z"
Oct 30 20:37:26.326034 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.323917    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-cnibin\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:37:26.326034 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.323924    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c3704770-6fb6-4086-8a22-9b29468d5a70-konnectivity-ca\") pod \"konnectivity-agent-lkxfb\" (UID: \"c3704770-6fb6-4086-8a22-9b29468d5a70\") " pod="kube-system/konnectivity-agent-lkxfb"
Oct 30 20:37:26.326034 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.323931    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-multus-cni-dir\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:37:26.326034 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.323943    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1a74da60-0284-410b-9d69-0d5192ce0928-dbus\") pod \"global-pull-secret-syncer-xj584\" (UID: \"1a74da60-0284-410b-9d69-0d5192ce0928\") " pod="kube-system/global-pull-secret-syncer-xj584"
Oct 30 20:37:26.326034 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.323981    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-host-cni-bin\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:37:26.326034 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.323996    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-run-ovn\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:37:26.326034 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.324008    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b7f60156-3771-486a-a9d2-7acd5ab9f475-host\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:37:26.326034 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.324034    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/08ddd17c-0134-45dc-8a2f-aca6e3668e9f-cni-binary-copy\") pod \"multus-additional-cni-plugins-4gl7z\" (UID: \"08ddd17c-0134-45dc-8a2f-aca6e3668e9f\") " pod="openshift-multus/multus-additional-cni-plugins-4gl7z"
Oct 30 20:37:26.326034 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.324059    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/08ddd17c-0134-45dc-8a2f-aca6e3668e9f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4gl7z\" (UID: \"08ddd17c-0134-45dc-8a2f-aca6e3668e9f\") " pod="openshift-multus/multus-additional-cni-plugins-4gl7z"
Oct 30 20:37:26.326590 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.324082    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-host-cni-bin\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:37:26.326590 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.324089    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-host-kubelet\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:37:26.326590 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.324062    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1a74da60-0284-410b-9d69-0d5192ce0928-dbus\") pod \"global-pull-secret-syncer-xj584\" (UID: \"1a74da60-0284-410b-9d69-0d5192ce0928\") " pod="kube-system/global-pull-secret-syncer-xj584"
Oct 30 20:37:26.326590 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.324141    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b7f60156-3771-486a-a9d2-7acd5ab9f475-host\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:37:26.326590 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.324147    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-systemd-units\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:37:26.326590 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.324171    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-var-lib-openvswitch\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:37:26.326590 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.324181    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-host-kubelet\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:37:26.326590 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.324194    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c3704770-6fb6-4086-8a22-9b29468d5a70-agent-certs\") pod \"konnectivity-agent-lkxfb\" (UID: \"c3704770-6fb6-4086-8a22-9b29468d5a70\") " pod="kube-system/konnectivity-agent-lkxfb"
Oct 30 20:37:26.326590 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.324213    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7a31d20-2bf3-4c38-8816-2181571a1fda-tmp-dir\") pod \"node-resolver-d69bq\" (UID: \"c7a31d20-2bf3-4c38-8816-2181571a1fda\") " pod="openshift-dns/node-resolver-d69bq"
Oct 30 20:37:26.326590 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.324246    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b7f60156-3771-486a-a9d2-7acd5ab9f475-run\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:37:26.326590 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.324249    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-cnibin\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:37:26.326590 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.324266    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b7f60156-3771-486a-a9d2-7acd5ab9f475-tmp\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:37:26.326590 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.324292    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-cni-binary-copy\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:37:26.326590 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.324290    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-var-lib-openvswitch\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:37:26.326590 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.324339    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frzdn\" (UniqueName: \"kubernetes.io/projected/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-kube-api-access-frzdn\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:37:26.326590 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.324363    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a7b1237b-e022-4570-9c48-4a8449996c69-registration-dir\") pod \"aws-ebs-csi-driver-node-nvcbb\" (UID: \"a7b1237b-e022-4570-9c48-4a8449996c69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvcbb"
Oct 30 20:37:26.326590 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.324371    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/764461d5-e0b2-4856-a97e-0c5d3d34a497-ovnkube-config\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:37:26.326590 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.324387    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1a74da60-0284-410b-9d69-0d5192ce0928-kubelet-config\") pod \"global-pull-secret-syncer-xj584\" (UID: \"1a74da60-0284-410b-9d69-0d5192ce0928\") " pod="kube-system/global-pull-secret-syncer-xj584"
Oct 30 20:37:26.327256 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.324342    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-systemd-units\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:37:26.327256 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.324366    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b7f60156-3771-486a-a9d2-7acd5ab9f475-run\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:37:26.327256 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.324446    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b7f60156-3771-486a-a9d2-7acd5ab9f475-etc-kubernetes\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:37:26.327256 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.324472    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7f60156-3771-486a-a9d2-7acd5ab9f475-sys\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:37:26.327256 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.324492    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a7b1237b-e022-4570-9c48-4a8449996c69-registration-dir\") pod \"aws-ebs-csi-driver-node-nvcbb\" (UID: \"a7b1237b-e022-4570-9c48-4a8449996c69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvcbb"
Oct 30 20:37:26.327256 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.324497    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-host-var-lib-kubelet\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:37:26.327256 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.324578    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1a74da60-0284-410b-9d69-0d5192ce0928-kubelet-config\") pod \"global-pull-secret-syncer-xj584\" (UID: \"1a74da60-0284-410b-9d69-0d5192ce0928\") " pod="kube-system/global-pull-secret-syncer-xj584"
Oct 30 20:37:26.327256 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.324705    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-host-var-lib-kubelet\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:37:26.327256 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.324760    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/08ddd17c-0134-45dc-8a2f-aca6e3668e9f-cni-binary-copy\") pod \"multus-additional-cni-plugins-4gl7z\" (UID: \"08ddd17c-0134-45dc-8a2f-aca6e3668e9f\") " pod="openshift-multus/multus-additional-cni-plugins-4gl7z"
Oct 30 20:37:26.327256 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.324809    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c7a31d20-2bf3-4c38-8816-2181571a1fda-hosts-file\") pod \"node-resolver-d69bq\" (UID: \"c7a31d20-2bf3-4c38-8816-2181571a1fda\") " pod="openshift-dns/node-resolver-d69bq"
Oct 30 20:37:26.327256 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.324838    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4m4z\" (UniqueName: \"kubernetes.io/projected/242a67ca-f9ea-431d-adee-ece061468a02-kube-api-access-k4m4z\") pod \"network-metrics-daemon-hjssq\" (UID: \"242a67ca-f9ea-431d-adee-ece061468a02\") " pod="openshift-multus/network-metrics-daemon-hjssq"
Oct 30 20:37:26.327256 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.324878    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b7f60156-3771-486a-a9d2-7acd5ab9f475-etc-sysctl-d\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:37:26.327256 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.324907    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b7f60156-3771-486a-a9d2-7acd5ab9f475-etc-sysctl-conf\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:37:26.327256 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.324933    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b7f60156-3771-486a-a9d2-7acd5ab9f475-lib-modules\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:37:26.327256 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.324961    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-multus-socket-dir-parent\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:37:26.327256 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.325002    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a7b1237b-e022-4570-9c48-4a8449996c69-sys-fs\") pod \"aws-ebs-csi-driver-node-nvcbb\" (UID: \"a7b1237b-e022-4570-9c48-4a8449996c69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvcbb"
Oct 30 20:37:26.327256 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.325028    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl5v9\" (UniqueName: \"kubernetes.io/projected/08ddd17c-0134-45dc-8a2f-aca6e3668e9f-kube-api-access-sl5v9\") pod \"multus-additional-cni-plugins-4gl7z\" (UID: \"08ddd17c-0134-45dc-8a2f-aca6e3668e9f\") " pod="openshift-multus/multus-additional-cni-plugins-4gl7z"
Oct 30 20:37:26.327865 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.325054    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-system-cni-dir\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:37:26.327865 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.325116    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-host-var-lib-cni-bin\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:37:26.327865 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.325161    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/08ddd17c-0134-45dc-8a2f-aca6e3668e9f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4gl7z\" (UID: \"08ddd17c-0134-45dc-8a2f-aca6e3668e9f\") " pod="openshift-multus/multus-additional-cni-plugins-4gl7z"
Oct 30 20:37:26.327865 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.325190    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:37:26.327865 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.325232    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-host-run-k8s-cni-cncf-io\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:37:26.327865 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.325302    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7f60156-3771-486a-a9d2-7acd5ab9f475-sys\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:37:26.327865 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.325300    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:37:26.327865 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.325345    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c7a31d20-2bf3-4c38-8816-2181571a1fda-hosts-file\") pod \"node-resolver-d69bq\" (UID: \"c7a31d20-2bf3-4c38-8816-2181571a1fda\") " pod="openshift-dns/node-resolver-d69bq"
Oct 30 20:37:26.327865 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.325357    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b7f60156-3771-486a-a9d2-7acd5ab9f475-etc-sysctl-d\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:37:26.327865 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.325408    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b7f60156-3771-486a-a9d2-7acd5ab9f475-etc-kubernetes\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:37:26.327865 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.325418    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b7f60156-3771-486a-a9d2-7acd5ab9f475-etc-sysctl-conf\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:37:26.327865 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.325421    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-host-var-lib-cni-bin\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:37:26.327865 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.325446    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-system-cni-dir\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:37:26.327865 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.325444    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a7b1237b-e022-4570-9c48-4a8449996c69-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nvcbb\" (UID: \"a7b1237b-e022-4570-9c48-4a8449996c69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvcbb"
Oct 30 20:37:26.327865 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.325484    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl4qd\" (UniqueName: \"kubernetes.io/projected/764461d5-e0b2-4856-a97e-0c5d3d34a497-kube-api-access-nl4qd\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:37:26.327865 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.325487    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b7f60156-3771-486a-a9d2-7acd5ab9f475-lib-modules\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:37:26.327865 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.325492    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a7b1237b-e022-4570-9c48-4a8449996c69-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nvcbb\" (UID: \"a7b1237b-e022-4570-9c48-4a8449996c69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvcbb"
Oct 30 20:37:26.328373 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.325519    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a7b1237b-e022-4570-9c48-4a8449996c69-sys-fs\") pod \"aws-ebs-csi-driver-node-nvcbb\" (UID: \"a7b1237b-e022-4570-9c48-4a8449996c69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvcbb"
Oct 30 20:37:26.328373 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.325603    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-multus-socket-dir-parent\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:37:26.328373 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.325820    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-host-run-k8s-cni-cncf-io\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:37:26.328373 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.325925    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/08ddd17c-0134-45dc-8a2f-aca6e3668e9f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4gl7z\" (UID: \"08ddd17c-0134-45dc-8a2f-aca6e3668e9f\") " pod="openshift-multus/multus-additional-cni-plugins-4gl7z"
Oct 30 20:37:26.328373 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.326302    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-cni-binary-copy\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:37:26.328373 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.326327    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/08ddd17c-0134-45dc-8a2f-aca6e3668e9f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4gl7z\" (UID: \"08ddd17c-0134-45dc-8a2f-aca6e3668e9f\") " pod="openshift-multus/multus-additional-cni-plugins-4gl7z"
Oct 30 20:37:26.328373 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.327662    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b7f60156-3771-486a-a9d2-7acd5ab9f475-etc-tuned\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:37:26.328373 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.327695    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b7f60156-3771-486a-a9d2-7acd5ab9f475-tmp\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:37:26.328373 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.327840    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/764461d5-e0b2-4856-a97e-0c5d3d34a497-ovn-node-metrics-cert\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:37:26.328373 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.327956    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c3704770-6fb6-4086-8a22-9b29468d5a70-agent-certs\") pod \"konnectivity-agent-lkxfb\" (UID: \"c3704770-6fb6-4086-8a22-9b29468d5a70\") " pod="kube-system/konnectivity-agent-lkxfb"
Oct 30 20:37:26.362051 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.362018    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nvzwj\""
Oct 30 20:37:26.362200 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.362060    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 20:37:26.362200 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.362085    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bmpds\""
Oct 30 20:37:26.362200 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.362126    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 20:37:26.365668 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.365648    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/764461d5-e0b2-4856-a97e-0c5d3d34a497-ovnkube-script-lib\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:37:26.377033 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:26.377005    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 20:37:26.377033 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:26.377027    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 20:37:26.377142 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:26.377039    2576 projected.go:196] Error preparing data for projected volume kube-api-access-bzhsg for pod openshift-network-diagnostics/network-check-target-kbpjj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:37:26.377142 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:26.377093    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e54d0918-84e8-40f3-9b18-6e2fafb3d500-kube-api-access-bzhsg podName:e54d0918-84e8-40f3-9b18-6e2fafb3d500 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:26.87707672 +0000 UTC m=+2.294351393 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-bzhsg" (UniqueName: "kubernetes.io/projected/e54d0918-84e8-40f3-9b18-6e2fafb3d500-kube-api-access-bzhsg") pod "network-check-target-kbpjj" (UID: "e54d0918-84e8-40f3-9b18-6e2fafb3d500") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:37:26.380238 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.380155    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftcfw\" (UniqueName: \"kubernetes.io/projected/0fb51194-9a91-4430-8ded-4844ffb6727a-kube-api-access-ftcfw\") pod \"iptables-alerter-bpgsn\" (UID: \"0fb51194-9a91-4430-8ded-4844ffb6727a\") " pod="openshift-network-operator/iptables-alerter-bpgsn"
Oct 30 20:37:26.380238 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.380166    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z96wz\" (UniqueName: \"kubernetes.io/projected/88a8166b-aa0c-41dc-8bf3-92708c60df57-kube-api-access-z96wz\") pod \"node-ca-5clg5\" (UID: \"88a8166b-aa0c-41dc-8bf3-92708c60df57\") " pod="openshift-image-registry/node-ca-5clg5"
Oct 30 20:37:26.380430 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.380331    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqm5k\" (UniqueName: \"kubernetes.io/projected/c7a31d20-2bf3-4c38-8816-2181571a1fda-kube-api-access-jqm5k\") pod \"node-resolver-d69bq\" (UID: \"c7a31d20-2bf3-4c38-8816-2181571a1fda\") " pod="openshift-dns/node-resolver-d69bq"
Oct 30 20:37:26.380609 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.380582    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkjkh\" (UniqueName: \"kubernetes.io/projected/a7b1237b-e022-4570-9c48-4a8449996c69-kube-api-access-xkjkh\") pod \"aws-ebs-csi-driver-node-nvcbb\" (UID: \"a7b1237b-e022-4570-9c48-4a8449996c69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvcbb"
Oct 30 20:37:26.381431 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.381403    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6qnn\" (UniqueName: \"kubernetes.io/projected/b7f60156-3771-486a-a9d2-7acd5ab9f475-kube-api-access-k6qnn\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:37:26.381533 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.381508    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4m4z\" (UniqueName: \"kubernetes.io/projected/242a67ca-f9ea-431d-adee-ece061468a02-kube-api-access-k4m4z\") pod \"network-metrics-daemon-hjssq\" (UID: \"242a67ca-f9ea-431d-adee-ece061468a02\") " pod="openshift-multus/network-metrics-daemon-hjssq"
Oct 30 20:37:26.381590 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.381574    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl5v9\" (UniqueName: \"kubernetes.io/projected/08ddd17c-0134-45dc-8a2f-aca6e3668e9f-kube-api-access-sl5v9\") pod \"multus-additional-cni-plugins-4gl7z\" (UID: \"08ddd17c-0134-45dc-8a2f-aca6e3668e9f\") " pod="openshift-multus/multus-additional-cni-plugins-4gl7z"
Oct 30 20:37:26.381817 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.381782    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frzdn\" (UniqueName: \"kubernetes.io/projected/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-kube-api-access-frzdn\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:37:26.382117 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.382078    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl4qd\" (UniqueName: \"kubernetes.io/projected/764461d5-e0b2-4856-a97e-0c5d3d34a497-kube-api-access-nl4qd\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:37:26.489609 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.489576    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:37:26.503970 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.503730    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-h4rs7"
Oct 30 20:37:26.514317 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.514292    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvcbb"
Oct 30 20:37:26.523945 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.523925    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5clg5"
Oct 30 20:37:26.534682 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.534661    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bpgsn"
Oct 30 20:37:26.552209 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.552192    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4gl7z"
Oct 30 20:37:26.558198 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.558183    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-d69bq"
Oct 30 20:37:26.587659 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.587640    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lkxfb"
Oct 30 20:37:26.592671 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:26.592649    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7a31d20_2bf3_4c38_8816_2181571a1fda.slice/crio-37919f769bfa6e93adece45c43c99fdd08bf244037b64270406b3d0b42f8541d WatchSource:0}: Error finding container 37919f769bfa6e93adece45c43c99fdd08bf244037b64270406b3d0b42f8541d: Status 404 returned error can't find the container with id 37919f769bfa6e93adece45c43c99fdd08bf244037b64270406b3d0b42f8541d
Oct 30 20:37:26.593389 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.593370    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:37:26.593626 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:26.593605    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccc9dd125588c9f50e9b9a79ea47623e.slice/crio-72b8abeeb9582f7dfef26e301ff0fd3b5f4a2c94b647dbbbac2bf7695ae45efa WatchSource:0}: Error finding container 72b8abeeb9582f7dfef26e301ff0fd3b5f4a2c94b647dbbbac2bf7695ae45efa: Status 404 returned error can't find the container with id 72b8abeeb9582f7dfef26e301ff0fd3b5f4a2c94b647dbbbac2bf7695ae45efa
Oct 30 20:37:26.594221 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:26.594203    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40ad3a6a5fd37a90a3238925c9fca019.slice/crio-1ea7e0ed662156243f8ea7572a5da1f21a253dc7697c46faf8bfc5712d171b76 WatchSource:0}: Error finding container 1ea7e0ed662156243f8ea7572a5da1f21a253dc7697c46faf8bfc5712d171b76: Status 404 returned error can't find the container with id 1ea7e0ed662156243f8ea7572a5da1f21a253dc7697c46faf8bfc5712d171b76
Oct 30 20:37:26.598491 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.598475    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:37:26.599175 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:26.599157    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3704770_6fb6_4086_8a22_9b29468d5a70.slice/crio-52d55cc2ca4f2a532260dd37c06837c41cff4ecde4c800d447ba3d1f90fed26f WatchSource:0}: Error finding container 52d55cc2ca4f2a532260dd37c06837c41cff4ecde4c800d447ba3d1f90fed26f: Status 404 returned error can't find the container with id 52d55cc2ca4f2a532260dd37c06837c41cff4ecde4c800d447ba3d1f90fed26f
Oct 30 20:37:26.603431 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:26.603413    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod764461d5_e0b2_4856_a97e_0c5d3d34a497.slice/crio-9bb2d18fcfaf283a4fcc45f58c023f04ae3298c1663aa836bc1e8145054137f4 WatchSource:0}: Error finding container 9bb2d18fcfaf283a4fcc45f58c023f04ae3298c1663aa836bc1e8145054137f4: Status 404 returned error can't find the container with id 9bb2d18fcfaf283a4fcc45f58c023f04ae3298c1663aa836bc1e8145054137f4
Oct 30 20:37:26.828090 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.828060    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/242a67ca-f9ea-431d-adee-ece061468a02-metrics-certs\") pod \"network-metrics-daemon-hjssq\" (UID: \"242a67ca-f9ea-431d-adee-ece061468a02\") " pod="openshift-multus/network-metrics-daemon-hjssq"
Oct 30 20:37:26.828255 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.828098    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a74da60-0284-410b-9d69-0d5192ce0928-original-pull-secret\") pod \"global-pull-secret-syncer-xj584\" (UID: \"1a74da60-0284-410b-9d69-0d5192ce0928\") " pod="kube-system/global-pull-secret-syncer-xj584"
Oct 30 20:37:26.828255 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:26.828199    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 20:37:26.828255 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:26.828203    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:37:26.828255 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:26.828254    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a74da60-0284-410b-9d69-0d5192ce0928-original-pull-secret podName:1a74da60-0284-410b-9d69-0d5192ce0928 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:27.828237929 +0000 UTC m=+3.245512608 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a74da60-0284-410b-9d69-0d5192ce0928-original-pull-secret") pod "global-pull-secret-syncer-xj584" (UID: "1a74da60-0284-410b-9d69-0d5192ce0928") : object "kube-system"/"original-pull-secret" not registered
Oct 30 20:37:26.828426 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:26.828268    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/242a67ca-f9ea-431d-adee-ece061468a02-metrics-certs podName:242a67ca-f9ea-431d-adee-ece061468a02 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:27.828261978 +0000 UTC m=+3.245536635 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/242a67ca-f9ea-431d-adee-ece061468a02-metrics-certs") pod "network-metrics-daemon-hjssq" (UID: "242a67ca-f9ea-431d-adee-ece061468a02") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:37:26.928950 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.928916    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzhsg\" (UniqueName: \"kubernetes.io/projected/e54d0918-84e8-40f3-9b18-6e2fafb3d500-kube-api-access-bzhsg\") pod \"network-check-target-kbpjj\" (UID: \"e54d0918-84e8-40f3-9b18-6e2fafb3d500\") " pod="openshift-network-diagnostics/network-check-target-kbpjj"
Oct 30 20:37:26.929464 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:26.929074    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 20:37:26.929464 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:26.929092    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 20:37:26.929464 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:26.929101    2576 projected.go:196] Error preparing data for projected volume kube-api-access-bzhsg for pod openshift-network-diagnostics/network-check-target-kbpjj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:37:26.929464 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:26.929154    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e54d0918-84e8-40f3-9b18-6e2fafb3d500-kube-api-access-bzhsg podName:e54d0918-84e8-40f3-9b18-6e2fafb3d500 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:27.929137568 +0000 UTC m=+3.346412230 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-bzhsg" (UniqueName: "kubernetes.io/projected/e54d0918-84e8-40f3-9b18-6e2fafb3d500-kube-api-access-bzhsg") pod "network-check-target-kbpjj" (UID: "e54d0918-84e8-40f3-9b18-6e2fafb3d500") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:37:26.955906 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.955882    2576 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:37:26.959387 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:26.959371    2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:37:27.018340 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:27.018319    2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:37:27.054757 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:27.054704    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-d69bq" event={"ID":"c7a31d20-2bf3-4c38-8816-2181571a1fda","Type":"ContainerStarted","Data":"37919f769bfa6e93adece45c43c99fdd08bf244037b64270406b3d0b42f8541d"}
Oct 30 20:37:27.055944 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:27.055918    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-165.us-west-1.compute.internal" event={"ID":"ccc9dd125588c9f50e9b9a79ea47623e","Type":"ContainerStarted","Data":"72b8abeeb9582f7dfef26e301ff0fd3b5f4a2c94b647dbbbac2bf7695ae45efa"}
Oct 30 20:37:27.057103 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:27.057083    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf" event={"ID":"764461d5-e0b2-4856-a97e-0c5d3d34a497","Type":"ContainerStarted","Data":"9bb2d18fcfaf283a4fcc45f58c023f04ae3298c1663aa836bc1e8145054137f4"}
Oct 30 20:37:27.058109 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:27.058088    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lkxfb" event={"ID":"c3704770-6fb6-4086-8a22-9b29468d5a70","Type":"ContainerStarted","Data":"52d55cc2ca4f2a532260dd37c06837c41cff4ecde4c800d447ba3d1f90fed26f"}
Oct 30 20:37:27.059186 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:27.059164    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-165.us-west-1.compute.internal" event={"ID":"40ad3a6a5fd37a90a3238925c9fca019","Type":"ContainerStarted","Data":"1ea7e0ed662156243f8ea7572a5da1f21a253dc7697c46faf8bfc5712d171b76"}
Oct 30 20:37:27.211352 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:27.211296    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 20:32:26 +0000 UTC" deadline="2028-07-26 01:40:32.451602096 +0000 UTC"
Oct 30 20:37:27.211352 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:27.211345    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15221h3m5.240259861s"
Oct 30 20:37:27.345224 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:27.345183    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7f60156_3771_486a_a9d2_7acd5ab9f475.slice/crio-5597bb7634853533e9000cc5af41efda9d5b9befb622696f777b3adf75905383 WatchSource:0}: Error finding container 5597bb7634853533e9000cc5af41efda9d5b9befb622696f777b3adf75905383: Status 404 returned error can't find the container with id 5597bb7634853533e9000cc5af41efda9d5b9befb622696f777b3adf75905383
Oct 30 20:37:27.392949 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:27.392907    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fb51194_9a91_4430_8ded_4844ffb6727a.slice/crio-4025cb9daa803f5e569efffc0b3dfd1521edd91538cf9a349efa38966ae598bd WatchSource:0}: Error finding container 4025cb9daa803f5e569efffc0b3dfd1521edd91538cf9a349efa38966ae598bd: Status 404 returned error can't find the container with id 4025cb9daa803f5e569efffc0b3dfd1521edd91538cf9a349efa38966ae598bd
Oct 30 20:37:27.430006 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:27.429916    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf5cc97c_3f88_40e2_b6e8_e5f7b279bf85.slice/crio-e1c3c4f48a7972af5d5cab42a2780626c2e8a8e1fbfb446f19b0c7ca3ce86303 WatchSource:0}: Error finding container e1c3c4f48a7972af5d5cab42a2780626c2e8a8e1fbfb446f19b0c7ca3ce86303: Status 404 returned error can't find the container with id e1c3c4f48a7972af5d5cab42a2780626c2e8a8e1fbfb446f19b0c7ca3ce86303
Oct 30 20:37:27.473884 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:27.473845    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08ddd17c_0134_45dc_8a2f_aca6e3668e9f.slice/crio-ec74ba9089bb74e797ab04ab632e021ded2af0dfdb192caf8dcd748f4721cf55 WatchSource:0}: Error finding container ec74ba9089bb74e797ab04ab632e021ded2af0dfdb192caf8dcd748f4721cf55: Status 404 returned error can't find the container with id ec74ba9089bb74e797ab04ab632e021ded2af0dfdb192caf8dcd748f4721cf55
Oct 30 20:37:27.543437 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:27.543400    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7b1237b_e022_4570_9c48_4a8449996c69.slice/crio-ed7caa3657ef2870726c29db54cb3e6618b46d7b01cd55ccd643f888811697de WatchSource:0}: Error finding container ed7caa3657ef2870726c29db54cb3e6618b46d7b01cd55ccd643f888811697de: Status 404 returned error can't find the container with id ed7caa3657ef2870726c29db54cb3e6618b46d7b01cd55ccd643f888811697de
Oct 30 20:37:27.606147 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:37:27.606105    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88a8166b_aa0c_41dc_8bf3_92708c60df57.slice/crio-207c2a8b2761f4dd57850a3ae4352f753b6f6f0011eebdfe6b3c757cabdbe25c WatchSource:0}: Error finding container 207c2a8b2761f4dd57850a3ae4352f753b6f6f0011eebdfe6b3c757cabdbe25c: Status 404 returned error can't find the container with id 207c2a8b2761f4dd57850a3ae4352f753b6f6f0011eebdfe6b3c757cabdbe25c
Oct 30 20:37:27.837869 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:27.837752    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/242a67ca-f9ea-431d-adee-ece061468a02-metrics-certs\") pod \"network-metrics-daemon-hjssq\" (UID: \"242a67ca-f9ea-431d-adee-ece061468a02\") " pod="openshift-multus/network-metrics-daemon-hjssq"
Oct 30 20:37:27.837869 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:27.837842    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a74da60-0284-410b-9d69-0d5192ce0928-original-pull-secret\") pod \"global-pull-secret-syncer-xj584\" (UID: \"1a74da60-0284-410b-9d69-0d5192ce0928\") " pod="kube-system/global-pull-secret-syncer-xj584"
Oct 30 20:37:27.838170 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:27.838115    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 20:37:27.838239 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:27.838223    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a74da60-0284-410b-9d69-0d5192ce0928-original-pull-secret podName:1a74da60-0284-410b-9d69-0d5192ce0928 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:29.838170287 +0000 UTC m=+5.255444954 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a74da60-0284-410b-9d69-0d5192ce0928-original-pull-secret") pod "global-pull-secret-syncer-xj584" (UID: "1a74da60-0284-410b-9d69-0d5192ce0928") : object "kube-system"/"original-pull-secret" not registered
Oct 30 20:37:27.838309 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:27.838301    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:37:27.838369 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:27.838339    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/242a67ca-f9ea-431d-adee-ece061468a02-metrics-certs podName:242a67ca-f9ea-431d-adee-ece061468a02 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:29.8383259 +0000 UTC m=+5.255600563 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/242a67ca-f9ea-431d-adee-ece061468a02-metrics-certs") pod "network-metrics-daemon-hjssq" (UID: "242a67ca-f9ea-431d-adee-ece061468a02") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:37:27.938516 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:27.938474    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzhsg\" (UniqueName: \"kubernetes.io/projected/e54d0918-84e8-40f3-9b18-6e2fafb3d500-kube-api-access-bzhsg\") pod \"network-check-target-kbpjj\" (UID: \"e54d0918-84e8-40f3-9b18-6e2fafb3d500\") " pod="openshift-network-diagnostics/network-check-target-kbpjj"
Oct 30 20:37:27.939215 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:27.938675    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 20:37:27.939215 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:27.938697    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 20:37:27.939215 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:27.938710    2576 projected.go:196] Error preparing data for projected volume kube-api-access-bzhsg for pod openshift-network-diagnostics/network-check-target-kbpjj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:37:27.939215 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:27.938762    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e54d0918-84e8-40f3-9b18-6e2fafb3d500-kube-api-access-bzhsg podName:e54d0918-84e8-40f3-9b18-6e2fafb3d500 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:29.938748494 +0000 UTC m=+5.356023155 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-bzhsg" (UniqueName: "kubernetes.io/projected/e54d0918-84e8-40f3-9b18-6e2fafb3d500-kube-api-access-bzhsg") pod "network-check-target-kbpjj" (UID: "e54d0918-84e8-40f3-9b18-6e2fafb3d500") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:37:28.050996 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:28.050963    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbpjj"
Oct 30 20:37:28.051187 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:28.051104    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbpjj" podUID="e54d0918-84e8-40f3-9b18-6e2fafb3d500"
Oct 30 20:37:28.051255 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:28.051200    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xj584"
Oct 30 20:37:28.051329 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:28.051297    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xj584" podUID="1a74da60-0284-410b-9d69-0d5192ce0928"
Oct 30 20:37:28.051393 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:28.051381    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjssq"
Oct 30 20:37:28.051487 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:28.051465    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hjssq" podUID="242a67ca-f9ea-431d-adee-ece061468a02"
Oct 30 20:37:28.063610 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:28.063513    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvcbb" event={"ID":"a7b1237b-e022-4570-9c48-4a8449996c69","Type":"ContainerStarted","Data":"ed7caa3657ef2870726c29db54cb3e6618b46d7b01cd55ccd643f888811697de"}
Oct 30 20:37:28.068322 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:28.068230    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4gl7z" event={"ID":"08ddd17c-0134-45dc-8a2f-aca6e3668e9f","Type":"ContainerStarted","Data":"ec74ba9089bb74e797ab04ab632e021ded2af0dfdb192caf8dcd748f4721cf55"}
Oct 30 20:37:28.069996 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:28.069972    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-h4rs7" event={"ID":"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85","Type":"ContainerStarted","Data":"e1c3c4f48a7972af5d5cab42a2780626c2e8a8e1fbfb446f19b0c7ca3ce86303"}
Oct 30 20:37:28.073145 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:28.073117    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-brmwl" event={"ID":"b7f60156-3771-486a-a9d2-7acd5ab9f475","Type":"ContainerStarted","Data":"5597bb7634853533e9000cc5af41efda9d5b9befb622696f777b3adf75905383"}
Oct 30 20:37:28.077519 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:28.077491    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5clg5" event={"ID":"88a8166b-aa0c-41dc-8bf3-92708c60df57","Type":"ContainerStarted","Data":"207c2a8b2761f4dd57850a3ae4352f753b6f6f0011eebdfe6b3c757cabdbe25c"}
Oct 30 20:37:28.079489 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:28.079413    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bpgsn" event={"ID":"0fb51194-9a91-4430-8ded-4844ffb6727a","Type":"ContainerStarted","Data":"4025cb9daa803f5e569efffc0b3dfd1521edd91538cf9a349efa38966ae598bd"}
Oct 30 20:37:28.212331 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:28.212264    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 20:32:26 +0000 UTC" deadline="2028-04-27 03:35:35.005074048 +0000 UTC"
Oct 30 20:37:28.212331 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:28.212327    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13062h58m6.792752137s"
Oct 30 20:37:29.874887 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:29.874803    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/242a67ca-f9ea-431d-adee-ece061468a02-metrics-certs\") pod \"network-metrics-daemon-hjssq\" (UID: \"242a67ca-f9ea-431d-adee-ece061468a02\") " pod="openshift-multus/network-metrics-daemon-hjssq"
Oct 30 20:37:29.875545 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:29.874916    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a74da60-0284-410b-9d69-0d5192ce0928-original-pull-secret\") pod \"global-pull-secret-syncer-xj584\" (UID: \"1a74da60-0284-410b-9d69-0d5192ce0928\") " pod="kube-system/global-pull-secret-syncer-xj584"
Oct 30 20:37:29.875545 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:29.875128    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 20:37:29.875545 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:29.875200    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a74da60-0284-410b-9d69-0d5192ce0928-original-pull-secret podName:1a74da60-0284-410b-9d69-0d5192ce0928 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:33.875177511 +0000 UTC m=+9.292452180 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a74da60-0284-410b-9d69-0d5192ce0928-original-pull-secret") pod "global-pull-secret-syncer-xj584" (UID: "1a74da60-0284-410b-9d69-0d5192ce0928") : object "kube-system"/"original-pull-secret" not registered
Oct 30 20:37:29.875742 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:29.875656    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:37:29.875742 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:29.875707    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/242a67ca-f9ea-431d-adee-ece061468a02-metrics-certs podName:242a67ca-f9ea-431d-adee-ece061468a02 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:33.875691925 +0000 UTC m=+9.292966602 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/242a67ca-f9ea-431d-adee-ece061468a02-metrics-certs") pod "network-metrics-daemon-hjssq" (UID: "242a67ca-f9ea-431d-adee-ece061468a02") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:37:29.976105 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:29.976002    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzhsg\" (UniqueName: \"kubernetes.io/projected/e54d0918-84e8-40f3-9b18-6e2fafb3d500-kube-api-access-bzhsg\") pod \"network-check-target-kbpjj\" (UID: \"e54d0918-84e8-40f3-9b18-6e2fafb3d500\") " pod="openshift-network-diagnostics/network-check-target-kbpjj"
Oct 30 20:37:29.976264 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:29.976184    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 20:37:29.976264 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:29.976210    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 20:37:29.976264 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:29.976223    2576 projected.go:196] Error preparing data for projected volume kube-api-access-bzhsg for pod openshift-network-diagnostics/network-check-target-kbpjj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:37:29.976386 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:29.976297    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e54d0918-84e8-40f3-9b18-6e2fafb3d500-kube-api-access-bzhsg podName:e54d0918-84e8-40f3-9b18-6e2fafb3d500 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:33.976278583 +0000 UTC m=+9.393553249 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-bzhsg" (UniqueName: "kubernetes.io/projected/e54d0918-84e8-40f3-9b18-6e2fafb3d500-kube-api-access-bzhsg") pod "network-check-target-kbpjj" (UID: "e54d0918-84e8-40f3-9b18-6e2fafb3d500") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:37:30.051088 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:30.051057    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbpjj"
Oct 30 20:37:30.051287 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:30.051247    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbpjj" podUID="e54d0918-84e8-40f3-9b18-6e2fafb3d500"
Oct 30 20:37:30.051360 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:30.051344    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xj584"
Oct 30 20:37:30.051486 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:30.051426    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xj584" podUID="1a74da60-0284-410b-9d69-0d5192ce0928"
Oct 30 20:37:30.051564 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:30.051515    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjssq"
Oct 30 20:37:30.051661 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:30.051641    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hjssq" podUID="242a67ca-f9ea-431d-adee-ece061468a02"
Oct 30 20:37:32.051804 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:32.051457    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbpjj"
Oct 30 20:37:32.052231 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:32.051847    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbpjj" podUID="e54d0918-84e8-40f3-9b18-6e2fafb3d500"
Oct 30 20:37:32.052897 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:32.052350    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xj584"
Oct 30 20:37:32.052897 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:32.052457    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xj584" podUID="1a74da60-0284-410b-9d69-0d5192ce0928"
Oct 30 20:37:32.052897 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:32.052517    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjssq"
Oct 30 20:37:32.052897 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:32.052599    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hjssq" podUID="242a67ca-f9ea-431d-adee-ece061468a02"
Oct 30 20:37:32.134216 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:32.134175    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-165.us-west-1.compute.internal" event={"ID":"40ad3a6a5fd37a90a3238925c9fca019","Type":"ContainerStarted","Data":"96140bf7b49715a4a17e2b6b944b2a56830caf2610f8b0baa1a7df79e7a09e17"}
Oct 30 20:37:32.360183 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:32.360103    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-165.us-west-1.compute.internal" podStartSLOduration=6.36008203 podStartE2EDuration="6.36008203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:37:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:32.358980983 +0000 UTC m=+7.776255661" watchObservedRunningTime="2026-10-30 20:37:32.36008203 +0000 UTC m=+7.777356717"
Oct 30 20:37:33.140638 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:33.140598    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lkxfb" event={"ID":"c3704770-6fb6-4086-8a22-9b29468d5a70","Type":"ContainerStarted","Data":"264dc21aeec7db52ba21e0a85b58ecbad18ae6eb9dad16735a7fd43a6c84dddf"}
Oct 30 20:37:33.147685 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:33.147648    2576 generic.go:356] "Generic (PLEG): container finished" podID="ccc9dd125588c9f50e9b9a79ea47623e" containerID="12de31f9ba42bfe2aff6d81b3a604a70b24a80ff65ccf3865da3e8cf27ee1868" exitCode=0
Oct 30 20:37:33.147826 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:33.147748    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-165.us-west-1.compute.internal" event={"ID":"ccc9dd125588c9f50e9b9a79ea47623e","Type":"ContainerDied","Data":"12de31f9ba42bfe2aff6d81b3a604a70b24a80ff65ccf3865da3e8cf27ee1868"}
Oct 30 20:37:33.153266 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:33.153222    2576 generic.go:356] "Generic (PLEG): container finished" podID="08ddd17c-0134-45dc-8a2f-aca6e3668e9f" containerID="9bb84dcf2cd17cda09fe673ba6010d41f613352eb4fab6ff7921adff4cce0e58" exitCode=0
Oct 30 20:37:33.154284 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:33.154255    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4gl7z" event={"ID":"08ddd17c-0134-45dc-8a2f-aca6e3668e9f","Type":"ContainerDied","Data":"9bb84dcf2cd17cda09fe673ba6010d41f613352eb4fab6ff7921adff4cce0e58"}
Oct 30 20:37:33.264778 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:33.264351    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-lkxfb" podStartSLOduration=2.543843968 podStartE2EDuration="7.264331798s" totalImagesPullingTime="4.72048783s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:37:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:33.264041761 +0000 UTC m=+8.681316466" watchObservedRunningTime="2026-10-30 20:37:33.264331798 +0000 UTC m=+8.681606490"
Oct 30 20:37:33.908832 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:33.908746    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/242a67ca-f9ea-431d-adee-ece061468a02-metrics-certs\") pod \"network-metrics-daemon-hjssq\" (UID: \"242a67ca-f9ea-431d-adee-ece061468a02\") " pod="openshift-multus/network-metrics-daemon-hjssq"
Oct 30 20:37:33.909081 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:33.908832    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a74da60-0284-410b-9d69-0d5192ce0928-original-pull-secret\") pod \"global-pull-secret-syncer-xj584\" (UID: \"1a74da60-0284-410b-9d69-0d5192ce0928\") " pod="kube-system/global-pull-secret-syncer-xj584"
Oct 30 20:37:33.909081 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:33.908967    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 20:37:33.909081 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:33.909037    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a74da60-0284-410b-9d69-0d5192ce0928-original-pull-secret podName:1a74da60-0284-410b-9d69-0d5192ce0928 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:41.909017654 +0000 UTC m=+17.326292317 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a74da60-0284-410b-9d69-0d5192ce0928-original-pull-secret") pod "global-pull-secret-syncer-xj584" (UID: "1a74da60-0284-410b-9d69-0d5192ce0928") : object "kube-system"/"original-pull-secret" not registered
Oct 30 20:37:33.909283 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:33.909130    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:37:33.909283 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:33.909169    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/242a67ca-f9ea-431d-adee-ece061468a02-metrics-certs podName:242a67ca-f9ea-431d-adee-ece061468a02 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:41.909157297 +0000 UTC m=+17.326431960 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/242a67ca-f9ea-431d-adee-ece061468a02-metrics-certs") pod "network-metrics-daemon-hjssq" (UID: "242a67ca-f9ea-431d-adee-ece061468a02") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:37:34.010224 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:34.009782    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzhsg\" (UniqueName: \"kubernetes.io/projected/e54d0918-84e8-40f3-9b18-6e2fafb3d500-kube-api-access-bzhsg\") pod \"network-check-target-kbpjj\" (UID: \"e54d0918-84e8-40f3-9b18-6e2fafb3d500\") " pod="openshift-network-diagnostics/network-check-target-kbpjj"
Oct 30 20:37:34.010224 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:34.010043    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 20:37:34.010224 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:34.010064    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 20:37:34.010224 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:34.010077    2576 projected.go:196] Error preparing data for projected volume kube-api-access-bzhsg for pod openshift-network-diagnostics/network-check-target-kbpjj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:37:34.010224 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:34.010144    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e54d0918-84e8-40f3-9b18-6e2fafb3d500-kube-api-access-bzhsg podName:e54d0918-84e8-40f3-9b18-6e2fafb3d500 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:42.010122348 +0000 UTC m=+17.427397017 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-bzhsg" (UniqueName: "kubernetes.io/projected/e54d0918-84e8-40f3-9b18-6e2fafb3d500-kube-api-access-bzhsg") pod "network-check-target-kbpjj" (UID: "e54d0918-84e8-40f3-9b18-6e2fafb3d500") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:37:34.050993 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:34.050945    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbpjj"
Oct 30 20:37:34.051209 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:34.051072    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbpjj" podUID="e54d0918-84e8-40f3-9b18-6e2fafb3d500"
Oct 30 20:37:34.051209 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:34.051153    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xj584"
Oct 30 20:37:34.051346 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:34.051228    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xj584" podUID="1a74da60-0284-410b-9d69-0d5192ce0928"
Oct 30 20:37:34.051346 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:34.051287    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjssq"
Oct 30 20:37:34.051454 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:34.051376    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hjssq" podUID="242a67ca-f9ea-431d-adee-ece061468a02"
Oct 30 20:37:34.161284 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:34.161204    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5clg5" event={"ID":"88a8166b-aa0c-41dc-8bf3-92708c60df57","Type":"ContainerStarted","Data":"e6baa19d3fcdf245c0303402659499e034a7ad3e676a6490e6bf7a7bfb74f1a4"}
Oct 30 20:37:34.170376 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:34.170335    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-165.us-west-1.compute.internal" event={"ID":"ccc9dd125588c9f50e9b9a79ea47623e","Type":"ContainerStarted","Data":"f048b02d1812d913c41be3ba7447b2f40aa4b28a9b3cd88efef323c8dfa14d70"}
Oct 30 20:37:34.415411 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:34.415306    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-165.us-west-1.compute.internal" podStartSLOduration=8.415285577 podStartE2EDuration="8.415285577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:37:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:34.415160183 +0000 UTC m=+9.832434868" watchObservedRunningTime="2026-10-30 20:37:34.415285577 +0000 UTC m=+9.832560257"
Oct 30 20:37:34.415680 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:34.415644    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-5clg5" podStartSLOduration=2.380867513 podStartE2EDuration="8.415631779s" totalImagesPullingTime="6.034764266s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:37:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:34.295100962 +0000 UTC m=+9.712375644" watchObservedRunningTime="2026-10-30 20:37:34.415631779 +0000 UTC m=+9.832906495"
Oct 30 20:37:35.175590 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:35.175555    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvcbb" event={"ID":"a7b1237b-e022-4570-9c48-4a8449996c69","Type":"ContainerStarted","Data":"e31f97bf85478178663e69a721a5dea0dd280d3ac11fc9ce6b5aa5f33994fe1f"}
Oct 30 20:37:35.451478 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:35.451454    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-lkxfb"
Oct 30 20:37:36.051352 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:36.051310    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbpjj"
Oct 30 20:37:36.051537 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:36.051460    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbpjj" podUID="e54d0918-84e8-40f3-9b18-6e2fafb3d500"
Oct 30 20:37:36.051947 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:36.051921    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xj584"
Oct 30 20:37:36.052043 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:36.052026    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xj584" podUID="1a74da60-0284-410b-9d69-0d5192ce0928"
Oct 30 20:37:36.052110 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:36.052094    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjssq"
Oct 30 20:37:36.052201 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:36.052180    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hjssq" podUID="242a67ca-f9ea-431d-adee-ece061468a02"
Oct 30 20:37:36.081491 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:36.081467    2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-lkxfb"
Oct 30 20:37:36.082440 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:36.082420    2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-lkxfb"
Oct 30 20:37:36.179308 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:36.179272    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-d69bq" event={"ID":"c7a31d20-2bf3-4c38-8816-2181571a1fda","Type":"ContainerStarted","Data":"345573db5cd9dc6ce38be3f65cb6d00cc131cbcc67ae170a10cf4afe7e07cef3"}
Oct 30 20:37:36.183362 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:36.183333    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-h4rs7" event={"ID":"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85","Type":"ContainerStarted","Data":"5c9dbf82f87624c9de376bcc7b874bb2433ed6b7d9783a4fce5c012cdee90e82"}
Oct 30 20:37:36.183655 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:36.183628    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-lkxfb"
Oct 30 20:37:36.436570 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:36.436537    2576 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 20:37:36.445105 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:36.445052    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-d69bq" podStartSLOduration=1.5911450820000002 podStartE2EDuration="10.445036349s" totalImagesPullingTime="8.853891267s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:37:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:36.324400444 +0000 UTC m=+11.741675157" watchObservedRunningTime="2026-10-30 20:37:36.445036349 +0000 UTC m=+11.862311026"
Oct 30 20:37:36.569852 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:36.569731    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-h4rs7" podStartSLOduration=1.9330258439999999 podStartE2EDuration="10.569712801s" totalImagesPullingTime="8.636686957s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:37:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:36.568870198 +0000 UTC m=+11.986144882" watchObservedRunningTime="2026-10-30 20:37:36.569712801 +0000 UTC m=+11.986987486"
Oct 30 20:37:36.984758 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:36.984437    2576 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T20:37:36.43656549Z","UUID":"4a3bdc8e-e17b-44bb-92ab-23b252b25aa0","Handler":null,"Name":"","Endpoint":""}
Oct 30 20:37:36.986855 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:36.986828    2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 20:37:36.986855 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:36.986863    2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 20:37:37.189090 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:37.189039    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvcbb" event={"ID":"a7b1237b-e022-4570-9c48-4a8449996c69","Type":"ContainerStarted","Data":"5ed22b622e787f059efafda47c1dc365ff118293c6253eb9c1274a0829db6589"}
Oct 30 20:37:38.052933 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:38.052892    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjssq"
Oct 30 20:37:38.053108 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:38.053038    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hjssq" podUID="242a67ca-f9ea-431d-adee-ece061468a02"
Oct 30 20:37:38.053691 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:38.053456    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbpjj"
Oct 30 20:37:38.053691 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:38.053552    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbpjj" podUID="e54d0918-84e8-40f3-9b18-6e2fafb3d500"
Oct 30 20:37:38.053691 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:38.053578    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xj584"
Oct 30 20:37:38.053691 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:38.053646    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xj584" podUID="1a74da60-0284-410b-9d69-0d5192ce0928"
Oct 30 20:37:38.193068 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:38.193031    2576 generic.go:356] "Generic (PLEG): container finished" podID="08ddd17c-0134-45dc-8a2f-aca6e3668e9f" containerID="6eb18861f50bc8b91cfd55eb493f1a8721d10bf15b7929bd52f60be904cdb9ff" exitCode=0
Oct 30 20:37:38.193506 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:38.193131    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4gl7z" event={"ID":"08ddd17c-0134-45dc-8a2f-aca6e3668e9f","Type":"ContainerDied","Data":"6eb18861f50bc8b91cfd55eb493f1a8721d10bf15b7929bd52f60be904cdb9ff"}
Oct 30 20:37:38.197307 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:38.197276    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bpgsn" event={"ID":"0fb51194-9a91-4430-8ded-4844ffb6727a","Type":"ContainerStarted","Data":"218445030b6488ec43da21876e760fad55eb9a6f954123c96ebbe66f32e22b0b"}
Oct 30 20:37:38.450594 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:38.450397    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-bpgsn" podStartSLOduration=3.792332862 podStartE2EDuration="12.450382293s" totalImagesPullingTime="8.658049431s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:37:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:38.449923591 +0000 UTC m=+13.867198271" watchObservedRunningTime="2026-10-30 20:37:38.450382293 +0000 UTC m=+13.867656973"
Oct 30 20:37:39.201287 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:39.201254    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvcbb" event={"ID":"a7b1237b-e022-4570-9c48-4a8449996c69","Type":"ContainerStarted","Data":"685cc167855c9d4047c6188034a513edad6b53fd020700f8a27953ff4227eb20"}
Oct 30 20:37:39.204519 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:39.204116    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-brmwl" event={"ID":"b7f60156-3771-486a-a9d2-7acd5ab9f475","Type":"ContainerStarted","Data":"e171ab873fee893acd2148b04cc7553ebd76d623ec991d6f90d6abb0d256525e"}
Oct 30 20:37:39.322245 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:39.321985    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvcbb" podStartSLOduration=3.070113267 podStartE2EDuration="13.321966575s" totalImagesPullingTime="10.251853308s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:37:26 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:39.321229836 +0000 UTC m=+14.738504517" watchObservedRunningTime="2026-10-30 20:37:39.321966575 +0000 UTC m=+14.739241256"
Oct 30 20:37:39.440310 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:39.437950    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-brmwl" podStartSLOduration=2.4206361100000002 podStartE2EDuration="13.437933725s" totalImagesPullingTime="11.017297615s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:37:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:39.4378256 +0000 UTC m=+14.855100280" watchObservedRunningTime="2026-10-30 20:37:39.437933725 +0000 UTC m=+14.855208406"
Oct 30 20:37:40.050996 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:40.050957    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbpjj"
Oct 30 20:37:40.051150 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:40.051038    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjssq"
Oct 30 20:37:40.051150 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:40.051056    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xj584"
Oct 30 20:37:40.051247 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:40.051146    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbpjj" podUID="e54d0918-84e8-40f3-9b18-6e2fafb3d500"
Oct 30 20:37:40.051247 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:40.051204    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hjssq" podUID="242a67ca-f9ea-431d-adee-ece061468a02"
Oct 30 20:37:40.051338 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:40.051304    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xj584" podUID="1a74da60-0284-410b-9d69-0d5192ce0928"
Oct 30 20:37:40.208731 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:40.208695    2576 generic.go:356] "Generic (PLEG): container finished" podID="08ddd17c-0134-45dc-8a2f-aca6e3668e9f" containerID="01133ca7ce0770488f70d4afcf53a63b728580317ef4b11ae2789b598f459d96" exitCode=0
Oct 30 20:37:40.209451 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:40.208823    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4gl7z" event={"ID":"08ddd17c-0134-45dc-8a2f-aca6e3668e9f","Type":"ContainerDied","Data":"01133ca7ce0770488f70d4afcf53a63b728580317ef4b11ae2789b598f459d96"}
Oct 30 20:37:41.972930 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:41.972896    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a74da60-0284-410b-9d69-0d5192ce0928-original-pull-secret\") pod \"global-pull-secret-syncer-xj584\" (UID: \"1a74da60-0284-410b-9d69-0d5192ce0928\") " pod="kube-system/global-pull-secret-syncer-xj584"
Oct 30 20:37:41.973434 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:41.972979    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/242a67ca-f9ea-431d-adee-ece061468a02-metrics-certs\") pod \"network-metrics-daemon-hjssq\" (UID: \"242a67ca-f9ea-431d-adee-ece061468a02\") " pod="openshift-multus/network-metrics-daemon-hjssq"
Oct 30 20:37:41.973434 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:41.973050    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 20:37:41.973434 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:41.973071    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:37:41.973434 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:41.973112    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/242a67ca-f9ea-431d-adee-ece061468a02-metrics-certs podName:242a67ca-f9ea-431d-adee-ece061468a02 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:57.973099477 +0000 UTC m=+33.390374148 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/242a67ca-f9ea-431d-adee-ece061468a02-metrics-certs") pod "network-metrics-daemon-hjssq" (UID: "242a67ca-f9ea-431d-adee-ece061468a02") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:37:41.973434 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:41.973126    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a74da60-0284-410b-9d69-0d5192ce0928-original-pull-secret podName:1a74da60-0284-410b-9d69-0d5192ce0928 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:57.973120194 +0000 UTC m=+33.390394852 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a74da60-0284-410b-9d69-0d5192ce0928-original-pull-secret") pod "global-pull-secret-syncer-xj584" (UID: "1a74da60-0284-410b-9d69-0d5192ce0928") : object "kube-system"/"original-pull-secret" not registered
Oct 30 20:37:42.051289 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:42.051251    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbpjj"
Oct 30 20:37:42.051289 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:42.051266    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xj584"
Oct 30 20:37:42.051488 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:42.051268    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjssq"
Oct 30 20:37:42.051488 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:42.051361    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbpjj" podUID="e54d0918-84e8-40f3-9b18-6e2fafb3d500"
Oct 30 20:37:42.051488 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:42.051438    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xj584" podUID="1a74da60-0284-410b-9d69-0d5192ce0928"
Oct 30 20:37:42.051644 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:42.051508    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hjssq" podUID="242a67ca-f9ea-431d-adee-ece061468a02"
Oct 30 20:37:42.073613 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:42.073579    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzhsg\" (UniqueName: \"kubernetes.io/projected/e54d0918-84e8-40f3-9b18-6e2fafb3d500-kube-api-access-bzhsg\") pod \"network-check-target-kbpjj\" (UID: \"e54d0918-84e8-40f3-9b18-6e2fafb3d500\") " pod="openshift-network-diagnostics/network-check-target-kbpjj"
Oct 30 20:37:42.073802 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:42.073761    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 20:37:42.073895 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:42.073804    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 20:37:42.073895 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:42.073818    2576 projected.go:196] Error preparing data for projected volume kube-api-access-bzhsg for pod openshift-network-diagnostics/network-check-target-kbpjj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:37:42.073895 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:42.073879    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e54d0918-84e8-40f3-9b18-6e2fafb3d500-kube-api-access-bzhsg podName:e54d0918-84e8-40f3-9b18-6e2fafb3d500 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:58.073861507 +0000 UTC m=+33.491136168 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-bzhsg" (UniqueName: "kubernetes.io/projected/e54d0918-84e8-40f3-9b18-6e2fafb3d500-kube-api-access-bzhsg") pod "network-check-target-kbpjj" (UID: "e54d0918-84e8-40f3-9b18-6e2fafb3d500") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:37:42.215870 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:42.215838    2576 generic.go:356] "Generic (PLEG): container finished" podID="08ddd17c-0134-45dc-8a2f-aca6e3668e9f" containerID="1dae6046ddd9de584cc5207ac196d3f359bbf0c4f9fad84e7f06a0f01b5409fa" exitCode=0
Oct 30 20:37:42.216031 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:42.215890    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4gl7z" event={"ID":"08ddd17c-0134-45dc-8a2f-aca6e3668e9f","Type":"ContainerDied","Data":"1dae6046ddd9de584cc5207ac196d3f359bbf0c4f9fad84e7f06a0f01b5409fa"}
Oct 30 20:37:44.051444 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:44.051285    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbpjj"
Oct 30 20:37:44.051759 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:44.051358    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjssq"
Oct 30 20:37:44.051759 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:44.051525    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbpjj" podUID="e54d0918-84e8-40f3-9b18-6e2fafb3d500"
Oct 30 20:37:44.051759 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:44.051366    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xj584"
Oct 30 20:37:44.051759 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:44.051591    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hjssq" podUID="242a67ca-f9ea-431d-adee-ece061468a02"
Oct 30 20:37:44.051759 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:44.051650    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xj584" podUID="1a74da60-0284-410b-9d69-0d5192ce0928"
Oct 30 20:37:44.223905 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:44.223875    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r7lpf_764461d5-e0b2-4856-a97e-0c5d3d34a497/ovn-acl-logging/0.log"
Oct 30 20:37:44.224314 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:44.224287    2576 generic.go:356] "Generic (PLEG): container finished" podID="764461d5-e0b2-4856-a97e-0c5d3d34a497" containerID="0c20ec4888cc8db97227040556ccefd8f74287d75cdc9a38fea5ff5952c6ecd0" exitCode=1
Oct 30 20:37:44.224410 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:44.224335    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf" event={"ID":"764461d5-e0b2-4856-a97e-0c5d3d34a497","Type":"ContainerStarted","Data":"949d0b9f86b7ce574979b1540130519974df6d39a772df20f8af42f42c92b7a1"}
Oct 30 20:37:44.224465 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:44.224412    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf" event={"ID":"764461d5-e0b2-4856-a97e-0c5d3d34a497","Type":"ContainerStarted","Data":"aec0ff65fb9bf394fc7d5d1a3b77b7e22e673259a402200c0b6c01524517b161"}
Oct 30 20:37:44.224533 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:44.224462    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf" event={"ID":"764461d5-e0b2-4856-a97e-0c5d3d34a497","Type":"ContainerStarted","Data":"eb2c03fba45b2edc673fa72fd090e600ca7b44e647bed33dc398ed10f9681f21"}
Oct 30 20:37:44.224533 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:44.224508    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf" event={"ID":"764461d5-e0b2-4856-a97e-0c5d3d34a497","Type":"ContainerStarted","Data":"274d138c3f8a178255935581a8f4fbbeac0eea1e32b610bda91519185eb6bfa4"}
Oct 30 20:37:44.224625 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:44.224555    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf" event={"ID":"764461d5-e0b2-4856-a97e-0c5d3d34a497","Type":"ContainerDied","Data":"0c20ec4888cc8db97227040556ccefd8f74287d75cdc9a38fea5ff5952c6ecd0"}
Oct 30 20:37:44.224625 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:44.224604    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf" event={"ID":"764461d5-e0b2-4856-a97e-0c5d3d34a497","Type":"ContainerStarted","Data":"4b9e2eb0049b20c553afe52cbf565983a3b216092473925a034596e0f41fb37b"}
Oct 30 20:37:45.229056 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:45.228808    2576 generic.go:356] "Generic (PLEG): container finished" podID="08ddd17c-0134-45dc-8a2f-aca6e3668e9f" containerID="4600938f9a71636c9ee30f66e240f2c3283c1b5f1bcc92ff5ba9bda4b27c9893" exitCode=0
Oct 30 20:37:45.229056 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:45.228887    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4gl7z" event={"ID":"08ddd17c-0134-45dc-8a2f-aca6e3668e9f","Type":"ContainerDied","Data":"4600938f9a71636c9ee30f66e240f2c3283c1b5f1bcc92ff5ba9bda4b27c9893"}
Oct 30 20:37:46.051590 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:46.051551    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xj584"
Oct 30 20:37:46.051590 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:46.051585    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbpjj"
Oct 30 20:37:46.051864 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:46.051682    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xj584" podUID="1a74da60-0284-410b-9d69-0d5192ce0928"
Oct 30 20:37:46.051864 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:46.051700    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjssq"
Oct 30 20:37:46.051864 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:46.051820    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbpjj" podUID="e54d0918-84e8-40f3-9b18-6e2fafb3d500"
Oct 30 20:37:46.052007 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:46.051919    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hjssq" podUID="242a67ca-f9ea-431d-adee-ece061468a02"
Oct 30 20:37:46.234999 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:46.234965    2576 generic.go:356] "Generic (PLEG): container finished" podID="08ddd17c-0134-45dc-8a2f-aca6e3668e9f" containerID="8f9e03eae0151b90ef9428446d71ac4fb85fa96451d5819b2299e6d78003c12e" exitCode=0
Oct 30 20:37:46.235535 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:46.235059    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4gl7z" event={"ID":"08ddd17c-0134-45dc-8a2f-aca6e3668e9f","Type":"ContainerDied","Data":"8f9e03eae0151b90ef9428446d71ac4fb85fa96451d5819b2299e6d78003c12e"}
Oct 30 20:37:46.239024 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:46.239006    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r7lpf_764461d5-e0b2-4856-a97e-0c5d3d34a497/ovn-acl-logging/0.log"
Oct 30 20:37:46.239411 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:46.239389    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf" event={"ID":"764461d5-e0b2-4856-a97e-0c5d3d34a497","Type":"ContainerStarted","Data":"7a885bbf925ff42e35e74718a6b2d5fa8ceef8a7b395da17b0e20976c35d47f7"}
Oct 30 20:37:46.336314 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:46.336296    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-d69bq_c7a31d20-2bf3-4c38-8816-2181571a1fda/dns-node-resolver/0.log"
Oct 30 20:37:47.166630 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:47.166601    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5clg5_88a8166b-aa0c-41dc-8bf3-92708c60df57/node-ca/0.log"
Oct 30 20:37:47.245255 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:47.245219    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4gl7z" event={"ID":"08ddd17c-0134-45dc-8a2f-aca6e3668e9f","Type":"ContainerStarted","Data":"5b035711a92964f90f63d36e49bfdd0618afd5363265fe07783af71ceb91b5d7"}
Oct 30 20:37:47.368537 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:47.368491    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4gl7z" podStartSLOduration=8.845985701 podStartE2EDuration="21.368476942s" totalImagesPullingTime="12.522491241s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:37:26 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:47.368462069 +0000 UTC m=+22.785736749" watchObservedRunningTime="2026-10-30 20:37:47.368476942 +0000 UTC m=+22.785751612"
Oct 30 20:37:48.051604 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:48.051560    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbpjj"
Oct 30 20:37:48.051871 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:48.051579    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjssq"
Oct 30 20:37:48.051871 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:48.051671    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xj584"
Oct 30 20:37:48.051871 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:48.051683    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbpjj" podUID="e54d0918-84e8-40f3-9b18-6e2fafb3d500"
Oct 30 20:37:48.051871 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:48.051744    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xj584" podUID="1a74da60-0284-410b-9d69-0d5192ce0928"
Oct 30 20:37:48.051871 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:48.051843    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hjssq" podUID="242a67ca-f9ea-431d-adee-ece061468a02"
Oct 30 20:37:49.255546 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:49.255519    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r7lpf_764461d5-e0b2-4856-a97e-0c5d3d34a497/ovn-acl-logging/0.log"
Oct 30 20:37:49.256234 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:49.255861    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf" event={"ID":"764461d5-e0b2-4856-a97e-0c5d3d34a497","Type":"ContainerStarted","Data":"c606021a58cf3a06764c3bc420fee714561fa148843ce14e67e53a1ae7990c77"}
Oct 30 20:37:49.256234 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:49.256150    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:37:49.256234 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:49.256209    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:37:49.256366 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:49.256238    2576 scope.go:122] "RemoveContainer" containerID="0c20ec4888cc8db97227040556ccefd8f74287d75cdc9a38fea5ff5952c6ecd0"
Oct 30 20:37:49.272292 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:49.272266    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:37:50.050914 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:50.050881    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbpjj"
Oct 30 20:37:50.050914 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:50.050900    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xj584"
Oct 30 20:37:50.051160 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:50.050984    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbpjj" podUID="e54d0918-84e8-40f3-9b18-6e2fafb3d500"
Oct 30 20:37:50.051160 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:50.051035    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjssq"
Oct 30 20:37:50.051160 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:50.051149    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xj584" podUID="1a74da60-0284-410b-9d69-0d5192ce0928"
Oct 30 20:37:50.051305 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:50.051232    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hjssq" podUID="242a67ca-f9ea-431d-adee-ece061468a02"
Oct 30 20:37:50.262359 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:50.262170    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r7lpf_764461d5-e0b2-4856-a97e-0c5d3d34a497/ovn-acl-logging/0.log"
Oct 30 20:37:50.262752 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:50.262699    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf" event={"ID":"764461d5-e0b2-4856-a97e-0c5d3d34a497","Type":"ContainerStarted","Data":"51fea07031da16951cffa98df4d34e60aad4e6a79d9249b8492db482e6dd71a9"}
Oct 30 20:37:50.262928 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:50.262908    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:37:50.278604 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:50.278579    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:37:50.385945 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:50.385838    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf" podStartSLOduration=7.3542296480000005 podStartE2EDuration="24.385825382s" totalImagesPullingTime="17.031595734s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:37:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:50.385305812 +0000 UTC m=+25.802580491" watchObservedRunningTime="2026-10-30 20:37:50.385825382 +0000 UTC m=+25.803100061"
Oct 30 20:37:52.050658 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:52.050631    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbpjj"
Oct 30 20:37:52.051138 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:52.050725    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xj584"
Oct 30 20:37:52.051138 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:52.050738    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbpjj" podUID="e54d0918-84e8-40f3-9b18-6e2fafb3d500"
Oct 30 20:37:52.051138 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:52.050816    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xj584" podUID="1a74da60-0284-410b-9d69-0d5192ce0928"
Oct 30 20:37:52.051138 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:52.050827    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjssq"
Oct 30 20:37:52.051138 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:52.050921    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hjssq" podUID="242a67ca-f9ea-431d-adee-ece061468a02"
Oct 30 20:37:54.051670 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:54.051636    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xj584"
Oct 30 20:37:54.052175 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:54.051691    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjssq"
Oct 30 20:37:54.052175 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:54.051700    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbpjj"
Oct 30 20:37:54.052175 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:54.051772    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hjssq" podUID="242a67ca-f9ea-431d-adee-ece061468a02"
Oct 30 20:37:54.052175 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:54.051835    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbpjj" podUID="e54d0918-84e8-40f3-9b18-6e2fafb3d500"
Oct 30 20:37:54.052175 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:54.051852    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xj584" podUID="1a74da60-0284-410b-9d69-0d5192ce0928"
Oct 30 20:37:56.051025 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:56.050993    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjssq"
Oct 30 20:37:56.051426 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:56.051050    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbpjj"
Oct 30 20:37:56.051426 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:56.051060    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xj584"
Oct 30 20:37:56.051426 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:56.051121    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbpjj" podUID="e54d0918-84e8-40f3-9b18-6e2fafb3d500"
Oct 30 20:37:56.051426 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:56.051219    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hjssq" podUID="242a67ca-f9ea-431d-adee-ece061468a02"
Oct 30 20:37:56.051426 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:56.051290    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xj584" podUID="1a74da60-0284-410b-9d69-0d5192ce0928"
Oct 30 20:37:57.983104 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:57.983077    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/242a67ca-f9ea-431d-adee-ece061468a02-metrics-certs\") pod \"network-metrics-daemon-hjssq\" (UID: \"242a67ca-f9ea-431d-adee-ece061468a02\") " pod="openshift-multus/network-metrics-daemon-hjssq"
Oct 30 20:37:57.983478 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:57.983110    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a74da60-0284-410b-9d69-0d5192ce0928-original-pull-secret\") pod \"global-pull-secret-syncer-xj584\" (UID: \"1a74da60-0284-410b-9d69-0d5192ce0928\") " pod="kube-system/global-pull-secret-syncer-xj584"
Oct 30 20:37:57.983478 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:57.983208    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 20:37:57.983478 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:57.983208    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:37:57.983478 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:57.983264    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/242a67ca-f9ea-431d-adee-ece061468a02-metrics-certs podName:242a67ca-f9ea-431d-adee-ece061468a02 nodeName:}" failed. No retries permitted until 2026-10-30 20:38:29.983250303 +0000 UTC m=+65.400524966 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/242a67ca-f9ea-431d-adee-ece061468a02-metrics-certs") pod "network-metrics-daemon-hjssq" (UID: "242a67ca-f9ea-431d-adee-ece061468a02") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 20:37:57.983478 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:57.983276    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a74da60-0284-410b-9d69-0d5192ce0928-original-pull-secret podName:1a74da60-0284-410b-9d69-0d5192ce0928 nodeName:}" failed. No retries permitted until 2026-10-30 20:38:29.98327049 +0000 UTC m=+65.400545151 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a74da60-0284-410b-9d69-0d5192ce0928-original-pull-secret") pod "global-pull-secret-syncer-xj584" (UID: "1a74da60-0284-410b-9d69-0d5192ce0928") : object "kube-system"/"original-pull-secret" not registered
Oct 30 20:37:58.051567 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:58.051541    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbpjj"
Oct 30 20:37:58.051705 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:58.051620    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xj584"
Oct 30 20:37:58.051705 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:58.051625    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbpjj" podUID="e54d0918-84e8-40f3-9b18-6e2fafb3d500"
Oct 30 20:37:58.051705 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:58.051646    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjssq"
Oct 30 20:37:58.051845 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:58.051707    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xj584" podUID="1a74da60-0284-410b-9d69-0d5192ce0928"
Oct 30 20:37:58.051845 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:58.051749    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hjssq" podUID="242a67ca-f9ea-431d-adee-ece061468a02"
Oct 30 20:37:58.083514 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:37:58.083491    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzhsg\" (UniqueName: \"kubernetes.io/projected/e54d0918-84e8-40f3-9b18-6e2fafb3d500-kube-api-access-bzhsg\") pod \"network-check-target-kbpjj\" (UID: \"e54d0918-84e8-40f3-9b18-6e2fafb3d500\") " pod="openshift-network-diagnostics/network-check-target-kbpjj"
Oct 30 20:37:58.083638 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:58.083625    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 20:37:58.083676 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:58.083642    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 20:37:58.083676 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:58.083651    2576 projected.go:196] Error preparing data for projected volume kube-api-access-bzhsg for pod openshift-network-diagnostics/network-check-target-kbpjj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:37:58.083739 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:37:58.083694    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e54d0918-84e8-40f3-9b18-6e2fafb3d500-kube-api-access-bzhsg podName:e54d0918-84e8-40f3-9b18-6e2fafb3d500 nodeName:}" failed. No retries permitted until 2026-10-30 20:38:30.083679137 +0000 UTC m=+65.500953796 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-bzhsg" (UniqueName: "kubernetes.io/projected/e54d0918-84e8-40f3-9b18-6e2fafb3d500-kube-api-access-bzhsg") pod "network-check-target-kbpjj" (UID: "e54d0918-84e8-40f3-9b18-6e2fafb3d500") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 20:38:00.050837 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:00.050776    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbpjj"
Oct 30 20:38:00.051268 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:00.050887    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjssq"
Oct 30 20:38:00.051268 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:00.050906    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xj584"
Oct 30 20:38:00.051268 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:38:00.051025    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbpjj" podUID="e54d0918-84e8-40f3-9b18-6e2fafb3d500"
Oct 30 20:38:00.051268 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:38:00.051163    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hjssq" podUID="242a67ca-f9ea-431d-adee-ece061468a02"
Oct 30 20:38:00.051268 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:38:00.051236    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xj584" podUID="1a74da60-0284-410b-9d69-0d5192ce0928"
Oct 30 20:38:02.050952 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:02.050916    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbpjj"
Oct 30 20:38:02.051391 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:02.051024    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xj584"
Oct 30 20:38:02.051391 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:38:02.051039    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbpjj" podUID="e54d0918-84e8-40f3-9b18-6e2fafb3d500"
Oct 30 20:38:02.051391 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:02.051051    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjssq"
Oct 30 20:38:02.051391 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:38:02.051111    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xj584" podUID="1a74da60-0284-410b-9d69-0d5192ce0928"
Oct 30 20:38:02.051391 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:38:02.051176    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hjssq" podUID="242a67ca-f9ea-431d-adee-ece061468a02"
Oct 30 20:38:04.051478 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:04.051441    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xj584"
Oct 30 20:38:04.051963 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:04.051543    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjssq"
Oct 30 20:38:04.051963 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:04.051632    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbpjj"
Oct 30 20:38:04.051963 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:38:04.051656    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xj584" podUID="1a74da60-0284-410b-9d69-0d5192ce0928"
Oct 30 20:38:04.051963 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:38:04.051713    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbpjj" podUID="e54d0918-84e8-40f3-9b18-6e2fafb3d500"
Oct 30 20:38:04.051963 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:38:04.051775    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hjssq" podUID="242a67ca-f9ea-431d-adee-ece061468a02"
Oct 30 20:38:05.109937 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:05.109911    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kbpjj"]
Oct 30 20:38:05.110400 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:05.110005    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbpjj"
Oct 30 20:38:05.110400 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:38:05.110088    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbpjj" podUID="e54d0918-84e8-40f3-9b18-6e2fafb3d500"
Oct 30 20:38:05.114025 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:05.113994    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xj584"]
Oct 30 20:38:05.114132 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:05.114098    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xj584"
Oct 30 20:38:05.114210 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:38:05.114190    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xj584" podUID="1a74da60-0284-410b-9d69-0d5192ce0928"
Oct 30 20:38:05.158954 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:05.158923    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hjssq"]
Oct 30 20:38:05.159108 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:05.159027    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjssq"
Oct 30 20:38:05.159155 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:38:05.159111    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hjssq" podUID="242a67ca-f9ea-431d-adee-ece061468a02"
Oct 30 20:38:07.051124 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:07.051091    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjssq"
Oct 30 20:38:07.051124 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:07.051121    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xj584"
Oct 30 20:38:07.051546 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:07.051152    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbpjj"
Oct 30 20:38:07.051546 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:38:07.051247    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xj584" podUID="1a74da60-0284-410b-9d69-0d5192ce0928"
Oct 30 20:38:07.051546 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:38:07.051305    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbpjj" podUID="e54d0918-84e8-40f3-9b18-6e2fafb3d500"
Oct 30 20:38:07.051546 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:38:07.051346    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hjssq" podUID="242a67ca-f9ea-431d-adee-ece061468a02"
Oct 30 20:38:09.051495 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:09.051467    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjssq"
Oct 30 20:38:09.051867 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:09.051521    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbpjj"
Oct 30 20:38:09.051867 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:09.051528    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xj584"
Oct 30 20:38:09.051867 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:38:09.051597    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbpjj" podUID="e54d0918-84e8-40f3-9b18-6e2fafb3d500"
Oct 30 20:38:09.051867 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:38:09.051677    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hjssq" podUID="242a67ca-f9ea-431d-adee-ece061468a02"
Oct 30 20:38:09.051867 ip-10-0-128-165 kubenswrapper[2576]: E1030 20:38:09.051753    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xj584" podUID="1a74da60-0284-410b-9d69-0d5192ce0928"
Oct 30 20:38:10.268662 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:10.268499    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-165.us-west-1.compute.internal" event="NodeReady"
Oct 30 20:38:10.269032 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:10.268748    2576 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 20:38:10.381326 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:10.381292    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jgvrx"]
Oct 30 20:38:10.383325 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:10.383302    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jgvrx"
Oct 30 20:38:10.423753 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:10.423722    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8khvs"]
Oct 30 20:38:10.425403 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:10.425387    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8khvs"
Oct 30 20:38:10.430398 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:10.430373    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-k58xq"]
Oct 30 20:38:10.432262 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:10.432244    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-k58xq"
Oct 30 20:38:10.434860 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:10.434837    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 20:38:10.434915 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:10.434889    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 20:38:10.434958 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:10.434932    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-h8n42\""
Oct 30 20:38:10.442549 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:10.442529    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jgvrx"]
Oct 30 20:38:10.457329 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:10.457301    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-k58xq"]
Oct 30 20:38:10.468783 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:10.468761    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6b559cf8-14bb-4eaf-9bea-47a956284d20-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8khvs\" (UID: \"6b559cf8-14bb-4eaf-9bea-47a956284d20\") " pod="openshift-insights/insights-runtime-extractor-8khvs"
Oct 30 20:38:10.468890 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:10.468812    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m476q\" (UniqueName: \"kubernetes.io/projected/6b559cf8-14bb-4eaf-9bea-47a956284d20-kube-api-access-m476q\") pod \"insights-runtime-extractor-8khvs\" (UID: \"6b559cf8-14bb-4eaf-9bea-47a956284d20\") " pod="openshift-insights/insights-runtime-extractor-8khvs"
Oct 30 20:38:10.468933 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:10.468886    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b25a9500-23b1-42d8-8a77-ca75dabc6ef5-metrics-tls\") pod \"dns-default-k58xq\" (UID: \"b25a9500-23b1-42d8-8a77-ca75dabc6ef5\") " pod="openshift-dns/dns-default-k58xq"
Oct 30 20:38:10.468933 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:10.468908    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/520d950c-2ec5-457f-b55f-51bd29207757-cert\") pod \"ingress-canary-jgvrx\" (UID: \"520d950c-2ec5-457f-b55f-51bd29207757\") " pod="openshift-ingress-canary/ingress-canary-jgvrx"
Oct 30 20:38:10.468933 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:10.468927    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b25a9500-23b1-42d8-8a77-ca75dabc6ef5-config-volume\") pod \"dns-default-k58xq\" (UID: \"b25a9500-23b1-42d8-8a77-ca75dabc6ef5\") " pod="openshift-dns/dns-default-k58xq"
Oct 30 20:38:10.469029 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:10.468941    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b25a9500-23b1-42d8-8a77-ca75dabc6ef5-tmp-dir\") pod \"dns-default-k58xq\" (UID: \"b25a9500-23b1-42d8-8a77-ca75dabc6ef5\") " pod="openshift-dns/dns-default-k58xq"
Oct 30 20:38:10.469060 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:10.469022    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgzz6\" (UniqueName: \"kubernetes.io/projected/520d950c-2ec5-457f-b55f-51bd29207757-kube-api-access-xgzz6\") pod \"ingress-canary-jgvrx\" (UID: \"520d950c-2ec5-457f-b55f-51bd29207757\") " pod="openshift-ingress-canary/ingress-canary-jgvrx"
Oct 30 20:38:10.469111 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:10.469098    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6b559cf8-14bb-4eaf-9bea-47a956284d20-data-volume\") pod \"insights-runtime-extractor-8khvs\" (UID: \"6b559cf8-14bb-4eaf-9bea-47a956284d20\") " pod="openshift-insights/insights-runtime-extractor-8khvs"
Oct 30 20:38:10.469165 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:10.469154    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6b559cf8-14bb-4eaf-9bea-47a956284d20-crio-socket\") pod \"insights-runtime-extractor-8khvs\" (UID: \"6b559cf8-14bb-4eaf-9bea-47a956284d20\") " pod="openshift-insights/insights-runtime-extractor-8khvs"
Oct 30 20:38:10.469200 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:10.469177    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6b559cf8-14bb-4eaf-9bea-47a956284d20-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8khvs\" (UID: \"6b559cf8-14bb-4eaf-9bea-47a956284d20\") " pod="openshift-insights/insights-runtime-extractor-8khvs"
Oct 30 20:38:10.469238 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:10.469225    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cjbx\" (UniqueName: \"kubernetes.io/projected/b25a9500-23b1-42d8-8a77-ca75dabc6ef5-kube-api-access-9cjbx\") pod \"dns-default-k58xq\" (UID: \"b25a9500-23b1-42d8-8a77-ca75dabc6ef5\") " pod="openshift-dns/dns-default-k58xq"
Oct 30 20:38:10.504903 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:10.504876    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8khvs"]
Oct 30 20:38:10.535303 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:10.535228    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wsjv8\""
Oct 30 20:38:10.535303 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:10.535234    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2srwc\""
Oct 30 20:38:10.535303 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:10.535254    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 20:38:10.535303 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:10.535274    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 20:38:10.537275 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:10.537261    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 20:38:10.537315 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:10.537287    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 20:38:10.537315 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:10.537295    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 20:38:10.537315 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:10.537291    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 20:38:10.537410 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:10.537318    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 20:38:10.569679 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:10.569661    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m476q\" (UniqueName: \"kubernetes.io/projected/6b559cf8-14bb-4eaf-9bea-47a956284d20-kube-api-access-m476q\") pod \"insights-runtime-extractor-8khvs\" (UID: \"6b559cf8-14bb-4eaf-9bea-47a956284d20\") " pod="openshift-insights/insights-runtime-extractor-8khvs"
Oct 30 20:38:10.569728 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:10.569683    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b25a9500-23b1-42d8-8a77-ca75dabc6ef5-metrics-tls\") pod \"dns-default-k58xq\" (UID: \"b25a9500-23b1-42d8-8a77-ca75dabc6ef5\") " pod="openshift-dns/dns-default-k58xq"
Oct 30 20:38:10.569728 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:10.569699    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/520d950c-2ec5-457f-b55f-51bd29207757-cert\") pod \"ingress-canary-jgvrx\" (UID: \"520d950c-2ec5-457f-b55f-51bd29207757\") " pod="openshift-ingress-canary/ingress-canary-jgvrx"
Oct 30 20:38:10.569728 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:10.569716    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b25a9500-23b1-42d8-8a77-ca75dabc6ef5-config-volume\") pod \"dns-default-k58xq\" (UID: \"b25a9500-23b1-42d8-8a77-ca75dabc6ef5\") " pod="openshift-dns/dns-default-k58xq"
Oct 30 20:38:10.569885 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:10.569730    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b25a9500-23b1-42d8-8a77-ca75dabc6ef5-tmp-dir\") pod \"dns-default-k58xq\" (UID: \"b25a9500-23b1-42d8-8a77-ca75dabc6ef5\") " pod="openshift-dns/dns-default-k58xq"
Oct 30 20:38:10.569885 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:10.569770    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgzz6\" (UniqueName: \"kubernetes.io/projected/520d950c-2ec5-457f-b55f-51bd29207757-kube-api-access-xgzz6\") pod \"ingress-canary-jgvrx\" (UID: \"520d950c-2ec5-457f-b55f-51bd29207757\") " pod="openshift-ingress-canary/ingress-canary-jgvrx"
Oct 30 20:38:10.569885 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:10.569828    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6b559cf8-14bb-4eaf-9bea-47a956284d20-data-volume\") pod \"insights-runtime-extractor-8khvs\" (UID: \"6b559cf8-14bb-4eaf-9bea-47a956284d20\") " pod="openshift-insights/insights-runtime-extractor-8khvs"
Oct 30 20:38:10.569885 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:10.569876    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6b559cf8-14bb-4eaf-9bea-47a956284d20-crio-socket\") pod \"insights-runtime-extractor-8khvs\" (UID: \"6b559cf8-14bb-4eaf-9bea-47a956284d20\") " pod="openshift-insights/insights-runtime-extractor-8khvs"
Oct 30 20:38:10.570065 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:10.569902    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6b559cf8-14bb-4eaf-9bea-47a956284d20-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8khvs\" (UID: \"6b559cf8-14bb-4eaf-9bea-47a956284d20\") " pod="openshift-insights/insights-runtime-extractor-8khvs"
Oct 30 20:38:10.570065 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:10.569929    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cjbx\" (UniqueName: \"kubernetes.io/projected/b25a9500-23b1-42d8-8a77-ca75dabc6ef5-kube-api-access-9cjbx\") pod \"dns-default-k58xq\" (UID: \"b25a9500-23b1-42d8-8a77-ca75dabc6ef5\") " pod="openshift-dns/dns-default-k58xq"
Oct 30 20:38:10.570065 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:10.569954    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6b559cf8-14bb-4eaf-9bea-47a956284d20-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8khvs\" (UID: \"6b559cf8-14bb-4eaf-9bea-47a956284d20\") " pod="openshift-insights/insights-runtime-extractor-8khvs"
Oct 30 20:38:10.570231 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:10.570210    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6b559cf8-14bb-4eaf-9bea-47a956284d20-crio-socket\") pod \"insights-runtime-extractor-8khvs\" (UID: \"6b559cf8-14bb-4eaf-9bea-47a956284d20\") " pod="openshift-insights/insights-runtime-extractor-8khvs"
Oct 30 20:38:10.570400 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:10.570378    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b25a9500-23b1-42d8-8a77-ca75dabc6ef5-tmp-dir\") pod \"dns-default-k58xq\" (UID: \"b25a9500-23b1-42d8-8a77-ca75dabc6ef5\") " pod="openshift-dns/dns-default-k58xq"
Oct 30 20:38:10.570623 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:10.570595    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6b559cf8-14bb-4eaf-9bea-47a956284d20-data-volume\") pod \"insights-runtime-extractor-8khvs\" (UID: \"6b559cf8-14bb-4eaf-9bea-47a956284d20\") " pod="openshift-insights/insights-runtime-extractor-8khvs"
Oct 30 20:38:10.570702 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:10.570624    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b25a9500-23b1-42d8-8a77-ca75dabc6ef5-config-volume\") pod \"dns-default-k58xq\" (UID: \"b25a9500-23b1-42d8-8a77-ca75dabc6ef5\") " pod="openshift-dns/dns-default-k58xq"
Oct 30 20:38:10.570702 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:10.570666    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6b559cf8-14bb-4eaf-9bea-47a956284d20-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8khvs\" (UID: \"6b559cf8-14bb-4eaf-9bea-47a956284d20\") " pod="openshift-insights/insights-runtime-extractor-8khvs"
Oct 30 20:38:10.574045 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:10.574025    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b25a9500-23b1-42d8-8a77-ca75dabc6ef5-metrics-tls\") pod \"dns-default-k58xq\" (UID: \"b25a9500-23b1-42d8-8a77-ca75dabc6ef5\") " pod="openshift-dns/dns-default-k58xq"
Oct 30 20:38:10.574123 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:10.574025    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6b559cf8-14bb-4eaf-9bea-47a956284d20-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8khvs\" (UID: \"6b559cf8-14bb-4eaf-9bea-47a956284d20\") " pod="openshift-insights/insights-runtime-extractor-8khvs"
Oct 30 20:38:10.574123 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:10.574028    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/520d950c-2ec5-457f-b55f-51bd29207757-cert\") pod \"ingress-canary-jgvrx\" (UID: \"520d950c-2ec5-457f-b55f-51bd29207757\") " pod="openshift-ingress-canary/ingress-canary-jgvrx"
Oct 30 20:38:10.625701 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:10.625675    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m476q\" (UniqueName: \"kubernetes.io/projected/6b559cf8-14bb-4eaf-9bea-47a956284d20-kube-api-access-m476q\") pod \"insights-runtime-extractor-8khvs\" (UID: \"6b559cf8-14bb-4eaf-9bea-47a956284d20\") " pod="openshift-insights/insights-runtime-extractor-8khvs"
Oct 30 20:38:10.625950 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:10.625933    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgzz6\" (UniqueName: \"kubernetes.io/projected/520d950c-2ec5-457f-b55f-51bd29207757-kube-api-access-xgzz6\") pod \"ingress-canary-jgvrx\" (UID: \"520d950c-2ec5-457f-b55f-51bd29207757\") " pod="openshift-ingress-canary/ingress-canary-jgvrx"
Oct 30 20:38:10.626110 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:10.626091    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cjbx\" (UniqueName: \"kubernetes.io/projected/b25a9500-23b1-42d8-8a77-ca75dabc6ef5-kube-api-access-9cjbx\") pod \"dns-default-k58xq\" (UID: \"b25a9500-23b1-42d8-8a77-ca75dabc6ef5\") " pod="openshift-dns/dns-default-k58xq"
Oct 30 20:38:10.693756 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:10.693718    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jgvrx"
Oct 30 20:38:10.735990 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:10.735951    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8khvs"
Oct 30 20:38:10.742096 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:10.742072    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-k58xq"
Oct 30 20:38:11.051418 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:11.051350    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjssq"
Oct 30 20:38:11.051418 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:11.051378    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xj584"
Oct 30 20:38:11.051574 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:11.051442    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbpjj"
Oct 30 20:38:11.074291 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:11.074260    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jgvrx"]
Oct 30 20:38:11.103498 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:11.103387    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 20:38:11.103498 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:11.103457    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 20:38:11.103498 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:11.103468    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-hjmcj\""
Oct 30 20:38:11.103754 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:11.103511    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-49f2v\""
Oct 30 20:38:11.103754 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:11.103583    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4lpn7\""
Oct 30 20:38:11.103754 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:11.103599    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 20:38:11.103754 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:11.103618    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 20:38:11.104188 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:38:11.104168    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod520d950c_2ec5_457f_b55f_51bd29207757.slice/crio-fb474b3b1951057e4d840268aa15a635b39e4d8e8512b25b8d781e34fec82255 WatchSource:0}: Error finding container fb474b3b1951057e4d840268aa15a635b39e4d8e8512b25b8d781e34fec82255: Status 404 returned error can't find the container with id fb474b3b1951057e4d840268aa15a635b39e4d8e8512b25b8d781e34fec82255
Oct 30 20:38:11.127002 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:11.126983    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8khvs"]
Oct 30 20:38:11.130069 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:38:11.130051    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b559cf8_14bb_4eaf_9bea_47a956284d20.slice/crio-dd1344076d2e9fdfd6fb1d182509921d27b82e16709f2a47f113c478e72c814b WatchSource:0}: Error finding container dd1344076d2e9fdfd6fb1d182509921d27b82e16709f2a47f113c478e72c814b: Status 404 returned error can't find the container with id dd1344076d2e9fdfd6fb1d182509921d27b82e16709f2a47f113c478e72c814b
Oct 30 20:38:11.160773 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:11.160642    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-k58xq"]
Oct 30 20:38:11.163564 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:38:11.163543    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb25a9500_23b1_42d8_8a77_ca75dabc6ef5.slice/crio-0216b8abddf058a5e8c11ab3d4247a9bc0861a991792b0e9c5185b0457313f5a WatchSource:0}: Error finding container 0216b8abddf058a5e8c11ab3d4247a9bc0861a991792b0e9c5185b0457313f5a: Status 404 returned error can't find the container with id 0216b8abddf058a5e8c11ab3d4247a9bc0861a991792b0e9c5185b0457313f5a
Oct 30 20:38:11.307197 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:11.307107    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jgvrx" event={"ID":"520d950c-2ec5-457f-b55f-51bd29207757","Type":"ContainerStarted","Data":"fb474b3b1951057e4d840268aa15a635b39e4d8e8512b25b8d781e34fec82255"}
Oct 30 20:38:11.308262 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:11.308244    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-k58xq" event={"ID":"b25a9500-23b1-42d8-8a77-ca75dabc6ef5","Type":"ContainerStarted","Data":"0216b8abddf058a5e8c11ab3d4247a9bc0861a991792b0e9c5185b0457313f5a"}
Oct 30 20:38:11.309675 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:11.309657    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8khvs" event={"ID":"6b559cf8-14bb-4eaf-9bea-47a956284d20","Type":"ContainerStarted","Data":"676af471b73158a5095d647983fe521e95e00783c087f7196478e5f01b98cc27"}
Oct 30 20:38:11.309767 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:11.309692    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8khvs" event={"ID":"6b559cf8-14bb-4eaf-9bea-47a956284d20","Type":"ContainerStarted","Data":"dd1344076d2e9fdfd6fb1d182509921d27b82e16709f2a47f113c478e72c814b"}
Oct 30 20:38:13.319335 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:13.319263    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8khvs" event={"ID":"6b559cf8-14bb-4eaf-9bea-47a956284d20","Type":"ContainerStarted","Data":"b0e612e221def115c50011b5cc427cee7fe05c345c91550fb4e84a94cb38fe43"}
Oct 30 20:38:14.324726 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.324683    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-k58xq" event={"ID":"b25a9500-23b1-42d8-8a77-ca75dabc6ef5","Type":"ContainerStarted","Data":"d01d3069f1549c6db838000905df52d6ff88d27dbb130a6c801ad578ba67b2dc"}
Oct 30 20:38:14.325219 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.324770    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-k58xq" event={"ID":"b25a9500-23b1-42d8-8a77-ca75dabc6ef5","Type":"ContainerStarted","Data":"b30131a1cf448fe51b193c88d78571f05e224573fd5652e48a90c90fb0a06319"}
Oct 30 20:38:14.325219 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.324915    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-k58xq"
Oct 30 20:38:14.446708 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.446641    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-k58xq" podStartSLOduration=2.185881129 podStartE2EDuration="4.446617316s" totalImagesPullingTime="2.260736187s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:38:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:14.445527236 +0000 UTC m=+49.862801918" watchObservedRunningTime="2026-10-30 20:38:14.446617316 +0000 UTC m=+49.863891998"
Oct 30 20:38:14.561864 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.561828    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-c4xbh"]
Oct 30 20:38:14.563812 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.563774    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c4xbh"
Oct 30 20:38:14.599812 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.599695    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvnz6\" (UniqueName: \"kubernetes.io/projected/7787041b-34dc-4567-8989-22bfd95456e0-kube-api-access-bvnz6\") pod \"openshift-state-metrics-7bd7489957-c4xbh\" (UID: \"7787041b-34dc-4567-8989-22bfd95456e0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c4xbh"
Oct 30 20:38:14.599812 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.599733    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7787041b-34dc-4567-8989-22bfd95456e0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-c4xbh\" (UID: \"7787041b-34dc-4567-8989-22bfd95456e0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c4xbh"
Oct 30 20:38:14.599812 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.599750    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7787041b-34dc-4567-8989-22bfd95456e0-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-c4xbh\" (UID: \"7787041b-34dc-4567-8989-22bfd95456e0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c4xbh"
Oct 30 20:38:14.600109 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.599890    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7787041b-34dc-4567-8989-22bfd95456e0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-c4xbh\" (UID: \"7787041b-34dc-4567-8989-22bfd95456e0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c4xbh"
Oct 30 20:38:14.617017 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.616979    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 20:38:14.617274 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.617253    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Oct 30 20:38:14.617274 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.617273    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 20:38:14.617441 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.617290    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 20:38:14.617441 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.617381    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Oct 30 20:38:14.617535 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.617478    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-nl57c\""
Oct 30 20:38:14.639618 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.639588    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-c4xbh"]
Oct 30 20:38:14.674671 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.674646    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfcqd"]
Oct 30 20:38:14.676820 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.676801    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfcqd"
Oct 30 20:38:14.701237 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.701206    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvnz6\" (UniqueName: \"kubernetes.io/projected/7787041b-34dc-4567-8989-22bfd95456e0-kube-api-access-bvnz6\") pod \"openshift-state-metrics-7bd7489957-c4xbh\" (UID: \"7787041b-34dc-4567-8989-22bfd95456e0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c4xbh"
Oct 30 20:38:14.701237 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.701240    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5b900a91-15bc-4019-99f5-4619cea03518-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-cfcqd\" (UID: \"5b900a91-15bc-4019-99f5-4619cea03518\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfcqd"
Oct 30 20:38:14.703941 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.701265    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7787041b-34dc-4567-8989-22bfd95456e0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-c4xbh\" (UID: \"7787041b-34dc-4567-8989-22bfd95456e0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c4xbh"
Oct 30 20:38:14.703941 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.701282    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7787041b-34dc-4567-8989-22bfd95456e0-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-c4xbh\" (UID: \"7787041b-34dc-4567-8989-22bfd95456e0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c4xbh"
Oct 30 20:38:14.703941 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.701313    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jdjz\" (UniqueName: \"kubernetes.io/projected/5b900a91-15bc-4019-99f5-4619cea03518-kube-api-access-4jdjz\") pod \"kube-state-metrics-5dc8d7d89b-cfcqd\" (UID: \"5b900a91-15bc-4019-99f5-4619cea03518\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfcqd"
Oct 30 20:38:14.703941 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.701404    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5b900a91-15bc-4019-99f5-4619cea03518-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-cfcqd\" (UID: \"5b900a91-15bc-4019-99f5-4619cea03518\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfcqd"
Oct 30 20:38:14.703941 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.701470    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7787041b-34dc-4567-8989-22bfd95456e0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-c4xbh\" (UID: \"7787041b-34dc-4567-8989-22bfd95456e0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c4xbh"
Oct 30 20:38:14.703941 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.701505    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5b900a91-15bc-4019-99f5-4619cea03518-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-cfcqd\" (UID: \"5b900a91-15bc-4019-99f5-4619cea03518\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfcqd"
Oct 30 20:38:14.703941 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.701537    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5b900a91-15bc-4019-99f5-4619cea03518-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-cfcqd\" (UID: \"5b900a91-15bc-4019-99f5-4619cea03518\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfcqd"
Oct 30 20:38:14.703941 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.701566    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5b900a91-15bc-4019-99f5-4619cea03518-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-cfcqd\" (UID: \"5b900a91-15bc-4019-99f5-4619cea03518\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfcqd"
Oct 30 20:38:14.703941 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.703766    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7787041b-34dc-4567-8989-22bfd95456e0-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-c4xbh\" (UID: \"7787041b-34dc-4567-8989-22bfd95456e0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c4xbh"
Oct 30 20:38:14.705920 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.705894    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7787041b-34dc-4567-8989-22bfd95456e0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-c4xbh\" (UID: \"7787041b-34dc-4567-8989-22bfd95456e0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c4xbh"
Oct 30 20:38:14.706026 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.705988    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7787041b-34dc-4567-8989-22bfd95456e0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-c4xbh\" (UID: \"7787041b-34dc-4567-8989-22bfd95456e0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c4xbh"
Oct 30 20:38:14.730489 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.730464    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-lwhq8"]
Oct 30 20:38:14.732510 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.732492    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 30 20:38:14.732627 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.732574    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lwhq8"
Oct 30 20:38:14.732702 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.732690    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-jt52d\""
Oct 30 20:38:14.732702 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.732693    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 30 20:38:14.732844 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.732693    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 30 20:38:14.746004 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.745977    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfcqd"]
Oct 30 20:38:14.769109 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.769079    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvnz6\" (UniqueName: \"kubernetes.io/projected/7787041b-34dc-4567-8989-22bfd95456e0-kube-api-access-bvnz6\") pod \"openshift-state-metrics-7bd7489957-c4xbh\" (UID: \"7787041b-34dc-4567-8989-22bfd95456e0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c4xbh"
Oct 30 20:38:14.802696 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.802665    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5b900a91-15bc-4019-99f5-4619cea03518-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-cfcqd\" (UID: \"5b900a91-15bc-4019-99f5-4619cea03518\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfcqd"
Oct 30 20:38:14.802696 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.802701    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/53aaf696-7925-43b6-86fd-1b536b3095d3-node-exporter-wtmp\") pod \"node-exporter-lwhq8\" (UID: \"53aaf696-7925-43b6-86fd-1b536b3095d3\") " pod="openshift-monitoring/node-exporter-lwhq8"
Oct 30 20:38:14.802917 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.802721    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/53aaf696-7925-43b6-86fd-1b536b3095d3-metrics-client-ca\") pod \"node-exporter-lwhq8\" (UID: \"53aaf696-7925-43b6-86fd-1b536b3095d3\") " pod="openshift-monitoring/node-exporter-lwhq8"
Oct 30 20:38:14.802917 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.802832    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5b900a91-15bc-4019-99f5-4619cea03518-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-cfcqd\" (UID: \"5b900a91-15bc-4019-99f5-4619cea03518\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfcqd"
Oct 30 20:38:14.802917 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.802856    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/53aaf696-7925-43b6-86fd-1b536b3095d3-sys\") pod \"node-exporter-lwhq8\" (UID: \"53aaf696-7925-43b6-86fd-1b536b3095d3\") " pod="openshift-monitoring/node-exporter-lwhq8"
Oct 30 20:38:14.802917 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.802870    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/53aaf696-7925-43b6-86fd-1b536b3095d3-root\") pod \"node-exporter-lwhq8\" (UID: \"53aaf696-7925-43b6-86fd-1b536b3095d3\") " pod="openshift-monitoring/node-exporter-lwhq8"
Oct 30 20:38:14.802917 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.802891    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jdjz\" (UniqueName: \"kubernetes.io/projected/5b900a91-15bc-4019-99f5-4619cea03518-kube-api-access-4jdjz\") pod \"kube-state-metrics-5dc8d7d89b-cfcqd\" (UID: \"5b900a91-15bc-4019-99f5-4619cea03518\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfcqd"
Oct 30 20:38:14.803152 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.802972    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5b900a91-15bc-4019-99f5-4619cea03518-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-cfcqd\" (UID: \"5b900a91-15bc-4019-99f5-4619cea03518\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfcqd"
Oct 30 20:38:14.803152 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.803010    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/53aaf696-7925-43b6-86fd-1b536b3095d3-node-exporter-accelerators-collector-config\") pod \"node-exporter-lwhq8\" (UID: \"53aaf696-7925-43b6-86fd-1b536b3095d3\") " pod="openshift-monitoring/node-exporter-lwhq8"
Oct 30 20:38:14.803152 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.803048    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpbnb\" (UniqueName: \"kubernetes.io/projected/53aaf696-7925-43b6-86fd-1b536b3095d3-kube-api-access-kpbnb\") pod \"node-exporter-lwhq8\" (UID: \"53aaf696-7925-43b6-86fd-1b536b3095d3\") " pod="openshift-monitoring/node-exporter-lwhq8"
Oct 30 20:38:14.803152 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.803068    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5b900a91-15bc-4019-99f5-4619cea03518-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-cfcqd\" (UID: \"5b900a91-15bc-4019-99f5-4619cea03518\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfcqd"
Oct 30 20:38:14.803152 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.803111    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/53aaf696-7925-43b6-86fd-1b536b3095d3-node-exporter-textfile\") pod \"node-exporter-lwhq8\" (UID: \"53aaf696-7925-43b6-86fd-1b536b3095d3\") " pod="openshift-monitoring/node-exporter-lwhq8"
Oct 30 20:38:14.803401 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.803269    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/53aaf696-7925-43b6-86fd-1b536b3095d3-node-exporter-tls\") pod \"node-exporter-lwhq8\" (UID: \"53aaf696-7925-43b6-86fd-1b536b3095d3\") " pod="openshift-monitoring/node-exporter-lwhq8"
Oct 30 20:38:14.803401 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.803298    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5b900a91-15bc-4019-99f5-4619cea03518-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-cfcqd\" (UID: \"5b900a91-15bc-4019-99f5-4619cea03518\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfcqd"
Oct 30 20:38:14.803401 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.803317    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/53aaf696-7925-43b6-86fd-1b536b3095d3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lwhq8\" (UID: \"53aaf696-7925-43b6-86fd-1b536b3095d3\") " pod="openshift-monitoring/node-exporter-lwhq8"
Oct 30 20:38:14.803401 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.803357    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5b900a91-15bc-4019-99f5-4619cea03518-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-cfcqd\" (UID: \"5b900a91-15bc-4019-99f5-4619cea03518\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfcqd"
Oct 30 20:38:14.803682 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.803625    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5b900a91-15bc-4019-99f5-4619cea03518-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-cfcqd\" (UID: \"5b900a91-15bc-4019-99f5-4619cea03518\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfcqd"
Oct 30 20:38:14.803849 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.803715    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5b900a91-15bc-4019-99f5-4619cea03518-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-cfcqd\" (UID: \"5b900a91-15bc-4019-99f5-4619cea03518\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfcqd"
Oct 30 20:38:14.805459 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.805435    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5b900a91-15bc-4019-99f5-4619cea03518-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-cfcqd\" (UID: \"5b900a91-15bc-4019-99f5-4619cea03518\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfcqd"
Oct 30 20:38:14.805621 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.805607    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5b900a91-15bc-4019-99f5-4619cea03518-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-cfcqd\" (UID: \"5b900a91-15bc-4019-99f5-4619cea03518\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfcqd"
Oct 30 20:38:14.832895 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.832857    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 20:38:14.832895 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.832883    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dmr5l\""
Oct 30 20:38:14.832895 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.832857    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 20:38:14.833093 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.832936    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 20:38:14.860914 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.860852    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jdjz\" (UniqueName: \"kubernetes.io/projected/5b900a91-15bc-4019-99f5-4619cea03518-kube-api-access-4jdjz\") pod \"kube-state-metrics-5dc8d7d89b-cfcqd\" (UID: \"5b900a91-15bc-4019-99f5-4619cea03518\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfcqd"
Oct 30 20:38:14.876646 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.876625    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c4xbh"
Oct 30 20:38:14.903991 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.903964    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/53aaf696-7925-43b6-86fd-1b536b3095d3-node-exporter-accelerators-collector-config\") pod \"node-exporter-lwhq8\" (UID: \"53aaf696-7925-43b6-86fd-1b536b3095d3\") " pod="openshift-monitoring/node-exporter-lwhq8"
Oct 30 20:38:14.903991 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.903995    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpbnb\" (UniqueName: \"kubernetes.io/projected/53aaf696-7925-43b6-86fd-1b536b3095d3-kube-api-access-kpbnb\") pod \"node-exporter-lwhq8\" (UID: \"53aaf696-7925-43b6-86fd-1b536b3095d3\") " pod="openshift-monitoring/node-exporter-lwhq8"
Oct 30 20:38:14.904218 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.904026    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/53aaf696-7925-43b6-86fd-1b536b3095d3-node-exporter-textfile\") pod \"node-exporter-lwhq8\" (UID: \"53aaf696-7925-43b6-86fd-1b536b3095d3\") " pod="openshift-monitoring/node-exporter-lwhq8"
Oct 30 20:38:14.904218 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.904043    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/53aaf696-7925-43b6-86fd-1b536b3095d3-node-exporter-tls\") pod \"node-exporter-lwhq8\" (UID: \"53aaf696-7925-43b6-86fd-1b536b3095d3\") " pod="openshift-monitoring/node-exporter-lwhq8"
Oct 30 20:38:14.904218 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.904060    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/53aaf696-7925-43b6-86fd-1b536b3095d3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lwhq8\" (UID: \"53aaf696-7925-43b6-86fd-1b536b3095d3\") " pod="openshift-monitoring/node-exporter-lwhq8"
Oct 30 20:38:14.904218 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.904119    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/53aaf696-7925-43b6-86fd-1b536b3095d3-node-exporter-wtmp\") pod \"node-exporter-lwhq8\" (UID: \"53aaf696-7925-43b6-86fd-1b536b3095d3\") " pod="openshift-monitoring/node-exporter-lwhq8"
Oct 30 20:38:14.904218 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.904145    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/53aaf696-7925-43b6-86fd-1b536b3095d3-metrics-client-ca\") pod \"node-exporter-lwhq8\" (UID: \"53aaf696-7925-43b6-86fd-1b536b3095d3\") " pod="openshift-monitoring/node-exporter-lwhq8"
Oct 30 20:38:14.904218 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.904196    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/53aaf696-7925-43b6-86fd-1b536b3095d3-sys\") pod \"node-exporter-lwhq8\" (UID: \"53aaf696-7925-43b6-86fd-1b536b3095d3\") " pod="openshift-monitoring/node-exporter-lwhq8"
Oct 30 20:38:14.904218 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.904212    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/53aaf696-7925-43b6-86fd-1b536b3095d3-root\") pod \"node-exporter-lwhq8\" (UID: \"53aaf696-7925-43b6-86fd-1b536b3095d3\") " pod="openshift-monitoring/node-exporter-lwhq8"
Oct 30 20:38:14.904527 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.904268    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/53aaf696-7925-43b6-86fd-1b536b3095d3-root\") pod \"node-exporter-lwhq8\" (UID: \"53aaf696-7925-43b6-86fd-1b536b3095d3\") " pod="openshift-monitoring/node-exporter-lwhq8"
Oct 30 20:38:14.904527 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.904317    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/53aaf696-7925-43b6-86fd-1b536b3095d3-sys\") pod \"node-exporter-lwhq8\" (UID: \"53aaf696-7925-43b6-86fd-1b536b3095d3\") " pod="openshift-monitoring/node-exporter-lwhq8"
Oct 30 20:38:14.904527 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.904422    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/53aaf696-7925-43b6-86fd-1b536b3095d3-node-exporter-wtmp\") pod \"node-exporter-lwhq8\" (UID: \"53aaf696-7925-43b6-86fd-1b536b3095d3\") " pod="openshift-monitoring/node-exporter-lwhq8"
Oct 30 20:38:14.904527 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.904475    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/53aaf696-7925-43b6-86fd-1b536b3095d3-node-exporter-textfile\") pod \"node-exporter-lwhq8\" (UID: \"53aaf696-7925-43b6-86fd-1b536b3095d3\") " pod="openshift-monitoring/node-exporter-lwhq8"
Oct 30 20:38:14.904819 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.904773    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/53aaf696-7925-43b6-86fd-1b536b3095d3-node-exporter-accelerators-collector-config\") pod \"node-exporter-lwhq8\" (UID: \"53aaf696-7925-43b6-86fd-1b536b3095d3\") " pod="openshift-monitoring/node-exporter-lwhq8"
Oct 30 20:38:14.904901 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.904877    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/53aaf696-7925-43b6-86fd-1b536b3095d3-metrics-client-ca\") pod \"node-exporter-lwhq8\" (UID: \"53aaf696-7925-43b6-86fd-1b536b3095d3\") " pod="openshift-monitoring/node-exporter-lwhq8"
Oct 30 20:38:14.906811 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.906769    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/53aaf696-7925-43b6-86fd-1b536b3095d3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lwhq8\" (UID: \"53aaf696-7925-43b6-86fd-1b536b3095d3\") " pod="openshift-monitoring/node-exporter-lwhq8"
Oct 30 20:38:14.906949 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.906924    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/53aaf696-7925-43b6-86fd-1b536b3095d3-node-exporter-tls\") pod \"node-exporter-lwhq8\" (UID: \"53aaf696-7925-43b6-86fd-1b536b3095d3\") " pod="openshift-monitoring/node-exporter-lwhq8"
Oct 30 20:38:14.961215 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.961186    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpbnb\" (UniqueName: \"kubernetes.io/projected/53aaf696-7925-43b6-86fd-1b536b3095d3-kube-api-access-kpbnb\") pod \"node-exporter-lwhq8\" (UID: \"53aaf696-7925-43b6-86fd-1b536b3095d3\") " pod="openshift-monitoring/node-exporter-lwhq8"
Oct 30 20:38:14.988750 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:14.988721    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfcqd"
Oct 30 20:38:15.044146 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:15.044115    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lwhq8"
Oct 30 20:38:15.054160 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:38:15.054131    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53aaf696_7925_43b6_86fd_1b536b3095d3.slice/crio-5c04573be329bbdedea4c49d44b5542bbdca873033dfe32c353d5d813401a9f3 WatchSource:0}: Error finding container 5c04573be329bbdedea4c49d44b5542bbdca873033dfe32c353d5d813401a9f3: Status 404 returned error can't find the container with id 5c04573be329bbdedea4c49d44b5542bbdca873033dfe32c353d5d813401a9f3
Oct 30 20:38:15.296424 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:15.296380    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-c4xbh"]
Oct 30 20:38:15.300409 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:38:15.300386    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7787041b_34dc_4567_8989_22bfd95456e0.slice/crio-cd28db981bd07b2f76f443c22c22f841e728ee45b8044be93908ec23d1c18f9b WatchSource:0}: Error finding container cd28db981bd07b2f76f443c22c22f841e728ee45b8044be93908ec23d1c18f9b: Status 404 returned error can't find the container with id cd28db981bd07b2f76f443c22c22f841e728ee45b8044be93908ec23d1c18f9b
Oct 30 20:38:15.329411 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:15.329377    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8khvs" event={"ID":"6b559cf8-14bb-4eaf-9bea-47a956284d20","Type":"ContainerStarted","Data":"8dbbba9c238c15edbddb2a645912bdf9023cde54198c8ff70d79cddb399b52a5"}
Oct 30 20:38:15.331046 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:15.331011    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c4xbh" event={"ID":"7787041b-34dc-4567-8989-22bfd95456e0","Type":"ContainerStarted","Data":"cd28db981bd07b2f76f443c22c22f841e728ee45b8044be93908ec23d1c18f9b"}
Oct 30 20:38:15.332473 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:15.332448    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jgvrx" event={"ID":"520d950c-2ec5-457f-b55f-51bd29207757","Type":"ContainerStarted","Data":"d4aa10448814039ad47ba385330f92bed334ecd40b0b3a8d6957482207688875"}
Oct 30 20:38:15.333655 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:15.333635    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lwhq8" event={"ID":"53aaf696-7925-43b6-86fd-1b536b3095d3","Type":"ContainerStarted","Data":"5c04573be329bbdedea4c49d44b5542bbdca873033dfe32c353d5d813401a9f3"}
Oct 30 20:38:15.362406 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:15.362375    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfcqd"]
Oct 30 20:38:15.365609 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:38:15.365568    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b900a91_15bc_4019_99f5_4619cea03518.slice/crio-bb61b79694f7acf56a2fe0d9befede9258908bca9bccf0c60271880a71e5820e WatchSource:0}: Error finding container bb61b79694f7acf56a2fe0d9befede9258908bca9bccf0c60271880a71e5820e: Status 404 returned error can't find the container with id bb61b79694f7acf56a2fe0d9befede9258908bca9bccf0c60271880a71e5820e
Oct 30 20:38:15.448583 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:15.448535    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-8khvs" podStartSLOduration=2.2361778230000002 podStartE2EDuration="5.44851942s" totalImagesPullingTime="3.212341597s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:38:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:15.447672337 +0000 UTC m=+50.864947017" watchObservedRunningTime="2026-10-30 20:38:15.44851942 +0000 UTC m=+50.865794099"
Oct 30 20:38:15.561808 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:15.561687    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jgvrx" podStartSLOduration=2.199062505 podStartE2EDuration="5.561669195s" totalImagesPullingTime="3.36260669s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:38:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:15.561276488 +0000 UTC m=+50.978551169" watchObservedRunningTime="2026-10-30 20:38:15.561669195 +0000 UTC m=+50.978943874"
Oct 30 20:38:16.338452 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:16.338421    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfcqd" event={"ID":"5b900a91-15bc-4019-99f5-4619cea03518","Type":"ContainerStarted","Data":"bb61b79694f7acf56a2fe0d9befede9258908bca9bccf0c60271880a71e5820e"}
Oct 30 20:38:16.340634 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:16.340604    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c4xbh" event={"ID":"7787041b-34dc-4567-8989-22bfd95456e0","Type":"ContainerStarted","Data":"a4c7764dc5863f85952c767f7ab3b847a299fd1a0f05cf8409372bb38e4ed353"}
Oct 30 20:38:16.340752 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:16.340663    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c4xbh" event={"ID":"7787041b-34dc-4567-8989-22bfd95456e0","Type":"ContainerStarted","Data":"730bb14806e67342d6e7d4d6fb90ef41e32000de2a6bcd7d556f17208bc7e6bd"}
Oct 30 20:38:16.976658 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:16.976622    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 20:38:16.979622 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:16.979600    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:17.021086 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:17.020995    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:17.021238 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:17.021125    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-config-out\") pod \"alertmanager-main-0\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:17.021238 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:17.021153    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:17.021238 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:17.021170    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-web-config\") pod \"alertmanager-main-0\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:17.021400 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:17.021263    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:17.021400 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:17.021298    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-config-volume\") pod \"alertmanager-main-0\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:17.021400 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:17.021332    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:17.021400 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:17.021368    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:17.021594 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:17.021404    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:17.021594 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:17.021432    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:17.021594 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:17.021458    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xq44\" (UniqueName: \"kubernetes.io/projected/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-kube-api-access-8xq44\") pod \"alertmanager-main-0\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:17.021594 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:17.021487    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:17.021594 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:17.021517    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:17.040655 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:17.040634    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 20:38:17.041226 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:17.041204    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 20:38:17.042264 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:17.042237    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-pqmk2\""
Oct 30 20:38:17.043206 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:17.043183    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 20:38:17.047768 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:17.043717    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 20:38:17.047768 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:17.044162    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 20:38:17.066814 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:17.061899    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 20:38:17.066814 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:17.061959    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 20:38:17.124627 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:17.124599    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:17.124778 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:17.124677    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-config-out\") pod \"alertmanager-main-0\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:17.124778 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:17.124707    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:17.124778 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:17.124730    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-web-config\") pod \"alertmanager-main-0\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:17.124778 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:17.124757    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:17.124945 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:17.124781    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-config-volume\") pod \"alertmanager-main-0\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:17.124945 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:17.124831    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:17.124945 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:17.124862    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:17.124945 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:17.124894    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:17.124945 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:17.124922    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:17.125128 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:17.124947    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xq44\" (UniqueName: \"kubernetes.io/projected/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-kube-api-access-8xq44\") pod \"alertmanager-main-0\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:17.125128 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:17.124970    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:17.125128 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:17.124988    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:17.125498 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:17.125469    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:17.126247 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:17.126225    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:17.128075 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:17.128050    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-config-out\") pod \"alertmanager-main-0\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:17.129608 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:17.129581    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:17.130231 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:17.130209    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:17.131225 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:17.131202    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-config-volume\") pod \"alertmanager-main-0\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:17.131342 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:17.131296    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-web-config\") pod \"alertmanager-main-0\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:17.132196 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:17.132173    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:17.132632 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:17.132602    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:17.136451 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:17.136407    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:17.139964 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:17.139929    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 20:38:17.140357 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:17.140335    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 20:38:17.141469 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:17.141447    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 20:38:17.146902 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:17.146879    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:17.148209 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:17.148187    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:17.187111 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:17.187081    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xq44\" (UniqueName: \"kubernetes.io/projected/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-kube-api-access-8xq44\") pod \"alertmanager-main-0\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:17.300144 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:17.300077    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:17.347730 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:17.347698    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c4xbh" event={"ID":"7787041b-34dc-4567-8989-22bfd95456e0","Type":"ContainerStarted","Data":"11117900a1612b10eebe4b9a660f485dbfe24bd8031af5e12d52eb726742153c"}
Oct 30 20:38:17.349735 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:17.349712    2576 generic.go:356] "Generic (PLEG): container finished" podID="53aaf696-7925-43b6-86fd-1b536b3095d3" containerID="0c531114f8d8872186e4de8004fefa552bc40dfb17f210071f9c69ed15238a3c" exitCode=0
Oct 30 20:38:17.349875 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:17.349764    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lwhq8" event={"ID":"53aaf696-7925-43b6-86fd-1b536b3095d3","Type":"ContainerDied","Data":"0c531114f8d8872186e4de8004fefa552bc40dfb17f210071f9c69ed15238a3c"}
Oct 30 20:38:17.505476 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:17.505423    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c4xbh" podStartSLOduration=1.861813487 podStartE2EDuration="3.505405183s" totalImagesPullingTime="1.643591696s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:38:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:17.504409275 +0000 UTC m=+52.921683956" watchObservedRunningTime="2026-10-30 20:38:17.505405183 +0000 UTC m=+52.922679864"
Oct 30 20:38:17.811684 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:17.811654    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 20:38:17.814565 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:38:17.814530    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c4609ce_58a7_47f3_8eaa_0f19cea5c6db.slice/crio-7ff1c0abadbf0f85df09a48cb328c77e7e85b622024858da9d8c89ef20a49824 WatchSource:0}: Error finding container 7ff1c0abadbf0f85df09a48cb328c77e7e85b622024858da9d8c89ef20a49824: Status 404 returned error can't find the container with id 7ff1c0abadbf0f85df09a48cb328c77e7e85b622024858da9d8c89ef20a49824
Oct 30 20:38:18.354531 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:18.354490    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db","Type":"ContainerStarted","Data":"7ff1c0abadbf0f85df09a48cb328c77e7e85b622024858da9d8c89ef20a49824"}
Oct 30 20:38:18.357024 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:18.356988    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfcqd" event={"ID":"5b900a91-15bc-4019-99f5-4619cea03518","Type":"ContainerStarted","Data":"fca01ac1334820ddf7e03be0f1076205814048a264c8bdc07ecee4f09654e79d"}
Oct 30 20:38:18.357180 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:18.357044    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfcqd" event={"ID":"5b900a91-15bc-4019-99f5-4619cea03518","Type":"ContainerStarted","Data":"1ffcf7b11307a5d2d34c480d1b1667116f6caedee5ddffd81a2857486e30fbc0"}
Oct 30 20:38:18.357180 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:18.357071    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfcqd" event={"ID":"5b900a91-15bc-4019-99f5-4619cea03518","Type":"ContainerStarted","Data":"dfbb622fb80a653fdec4d898b070ea4b15222ad7f7884de305b1e8216933cfa6"}
Oct 30 20:38:18.359208 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:18.359180    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lwhq8" event={"ID":"53aaf696-7925-43b6-86fd-1b536b3095d3","Type":"ContainerStarted","Data":"383fc3e01b4a8d3d6d8ba2b50ed81bacd593233438aa44020544cffc4c05aa22"}
Oct 30 20:38:18.359329 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:18.359237    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lwhq8" event={"ID":"53aaf696-7925-43b6-86fd-1b536b3095d3","Type":"ContainerStarted","Data":"7219e9e53f2de8f4f315bda829560df0c2d48e6e1f4ca4891612a8f407f96c42"}
Oct 30 20:38:18.484623 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:18.484567    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfcqd" podStartSLOduration=2.535391392 podStartE2EDuration="4.484551409s" totalImagesPullingTime="1.949160017s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:38:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:18.483675063 +0000 UTC m=+53.900949745" watchObservedRunningTime="2026-10-30 20:38:18.484551409 +0000 UTC m=+53.901826090"
Oct 30 20:38:18.614833 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:18.614757    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-lwhq8" podStartSLOduration=2.903523748 podStartE2EDuration="4.614741704s" totalImagesPullingTime="1.617302678s" totalInitContainerRuntime="93.915278ms" isStatefulPod=true podCreationTimestamp="2026-10-30 20:38:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:18.613982529 +0000 UTC m=+54.031257210" watchObservedRunningTime="2026-10-30 20:38:18.614741704 +0000 UTC m=+54.032016384"
Oct 30 20:38:19.609975 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:19.609900    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-bfd9f4b99-p95mj"]
Oct 30 20:38:19.611984 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:19.611959    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-bfd9f4b99-p95mj"
Oct 30 20:38:19.648152 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:19.648121    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1004eede-9d90-42b0-bd11-42a32056f2ae-secret-metrics-server-client-certs\") pod \"metrics-server-bfd9f4b99-p95mj\" (UID: \"1004eede-9d90-42b0-bd11-42a32056f2ae\") " pod="openshift-monitoring/metrics-server-bfd9f4b99-p95mj"
Oct 30 20:38:19.648299 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:19.648172    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1004eede-9d90-42b0-bd11-42a32056f2ae-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-bfd9f4b99-p95mj\" (UID: \"1004eede-9d90-42b0-bd11-42a32056f2ae\") " pod="openshift-monitoring/metrics-server-bfd9f4b99-p95mj"
Oct 30 20:38:19.648299 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:19.648198    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1004eede-9d90-42b0-bd11-42a32056f2ae-audit-log\") pod \"metrics-server-bfd9f4b99-p95mj\" (UID: \"1004eede-9d90-42b0-bd11-42a32056f2ae\") " pod="openshift-monitoring/metrics-server-bfd9f4b99-p95mj"
Oct 30 20:38:19.648493 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:19.648300    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1004eede-9d90-42b0-bd11-42a32056f2ae-client-ca-bundle\") pod \"metrics-server-bfd9f4b99-p95mj\" (UID: \"1004eede-9d90-42b0-bd11-42a32056f2ae\") " pod="openshift-monitoring/metrics-server-bfd9f4b99-p95mj"
Oct 30 20:38:19.648493 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:19.648347    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1004eede-9d90-42b0-bd11-42a32056f2ae-metrics-server-audit-profiles\") pod \"metrics-server-bfd9f4b99-p95mj\" (UID: \"1004eede-9d90-42b0-bd11-42a32056f2ae\") " pod="openshift-monitoring/metrics-server-bfd9f4b99-p95mj"
Oct 30 20:38:19.648493 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:19.648392    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgsh5\" (UniqueName: \"kubernetes.io/projected/1004eede-9d90-42b0-bd11-42a32056f2ae-kube-api-access-tgsh5\") pod \"metrics-server-bfd9f4b99-p95mj\" (UID: \"1004eede-9d90-42b0-bd11-42a32056f2ae\") " pod="openshift-monitoring/metrics-server-bfd9f4b99-p95mj"
Oct 30 20:38:19.648493 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:19.648425    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1004eede-9d90-42b0-bd11-42a32056f2ae-secret-metrics-server-tls\") pod \"metrics-server-bfd9f4b99-p95mj\" (UID: \"1004eede-9d90-42b0-bd11-42a32056f2ae\") " pod="openshift-monitoring/metrics-server-bfd9f4b99-p95mj"
Oct 30 20:38:19.664140 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:19.664115    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 30 20:38:19.664465 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:19.664446    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 30 20:38:19.664536 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:19.664478    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 30 20:38:19.664536 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:19.664515    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 30 20:38:19.664666 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:19.664652    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-envd3r50lfh67\""
Oct 30 20:38:19.664666 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:19.664659    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-kr8d5\""
Oct 30 20:38:19.673209 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:19.673183    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-bfd9f4b99-p95mj"]
Oct 30 20:38:19.749777 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:19.749740    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1004eede-9d90-42b0-bd11-42a32056f2ae-client-ca-bundle\") pod \"metrics-server-bfd9f4b99-p95mj\" (UID: \"1004eede-9d90-42b0-bd11-42a32056f2ae\") " pod="openshift-monitoring/metrics-server-bfd9f4b99-p95mj"
Oct 30 20:38:19.749993 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:19.749827    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1004eede-9d90-42b0-bd11-42a32056f2ae-metrics-server-audit-profiles\") pod \"metrics-server-bfd9f4b99-p95mj\" (UID: \"1004eede-9d90-42b0-bd11-42a32056f2ae\") " pod="openshift-monitoring/metrics-server-bfd9f4b99-p95mj"
Oct 30 20:38:19.749993 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:19.749859    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgsh5\" (UniqueName: \"kubernetes.io/projected/1004eede-9d90-42b0-bd11-42a32056f2ae-kube-api-access-tgsh5\") pod \"metrics-server-bfd9f4b99-p95mj\" (UID: \"1004eede-9d90-42b0-bd11-42a32056f2ae\") " pod="openshift-monitoring/metrics-server-bfd9f4b99-p95mj"
Oct 30 20:38:19.750094 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:19.750002    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1004eede-9d90-42b0-bd11-42a32056f2ae-secret-metrics-server-tls\") pod \"metrics-server-bfd9f4b99-p95mj\" (UID: \"1004eede-9d90-42b0-bd11-42a32056f2ae\") " pod="openshift-monitoring/metrics-server-bfd9f4b99-p95mj"
Oct 30 20:38:19.750094 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:19.750040    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1004eede-9d90-42b0-bd11-42a32056f2ae-secret-metrics-server-client-certs\") pod \"metrics-server-bfd9f4b99-p95mj\" (UID: \"1004eede-9d90-42b0-bd11-42a32056f2ae\") " pod="openshift-monitoring/metrics-server-bfd9f4b99-p95mj"
Oct 30 20:38:19.750200 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:19.750100    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1004eede-9d90-42b0-bd11-42a32056f2ae-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-bfd9f4b99-p95mj\" (UID: \"1004eede-9d90-42b0-bd11-42a32056f2ae\") " pod="openshift-monitoring/metrics-server-bfd9f4b99-p95mj"
Oct 30 20:38:19.750200 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:19.750153    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1004eede-9d90-42b0-bd11-42a32056f2ae-audit-log\") pod \"metrics-server-bfd9f4b99-p95mj\" (UID: \"1004eede-9d90-42b0-bd11-42a32056f2ae\") " pod="openshift-monitoring/metrics-server-bfd9f4b99-p95mj"
Oct 30 20:38:19.750549 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:19.750509    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1004eede-9d90-42b0-bd11-42a32056f2ae-audit-log\") pod \"metrics-server-bfd9f4b99-p95mj\" (UID: \"1004eede-9d90-42b0-bd11-42a32056f2ae\") " pod="openshift-monitoring/metrics-server-bfd9f4b99-p95mj"
Oct 30 20:38:19.751009 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:19.750983    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1004eede-9d90-42b0-bd11-42a32056f2ae-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-bfd9f4b99-p95mj\" (UID: \"1004eede-9d90-42b0-bd11-42a32056f2ae\") " pod="openshift-monitoring/metrics-server-bfd9f4b99-p95mj"
Oct 30 20:38:19.751158 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:19.751138    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1004eede-9d90-42b0-bd11-42a32056f2ae-metrics-server-audit-profiles\") pod \"metrics-server-bfd9f4b99-p95mj\" (UID: \"1004eede-9d90-42b0-bd11-42a32056f2ae\") " pod="openshift-monitoring/metrics-server-bfd9f4b99-p95mj"
Oct 30 20:38:19.752548 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:19.752526    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1004eede-9d90-42b0-bd11-42a32056f2ae-secret-metrics-server-client-certs\") pod \"metrics-server-bfd9f4b99-p95mj\" (UID: \"1004eede-9d90-42b0-bd11-42a32056f2ae\") " pod="openshift-monitoring/metrics-server-bfd9f4b99-p95mj"
Oct 30 20:38:19.752621 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:19.752566    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1004eede-9d90-42b0-bd11-42a32056f2ae-secret-metrics-server-tls\") pod \"metrics-server-bfd9f4b99-p95mj\" (UID: \"1004eede-9d90-42b0-bd11-42a32056f2ae\") " pod="openshift-monitoring/metrics-server-bfd9f4b99-p95mj"
Oct 30 20:38:19.752687 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:19.752628    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1004eede-9d90-42b0-bd11-42a32056f2ae-client-ca-bundle\") pod \"metrics-server-bfd9f4b99-p95mj\" (UID: \"1004eede-9d90-42b0-bd11-42a32056f2ae\") " pod="openshift-monitoring/metrics-server-bfd9f4b99-p95mj"
Oct 30 20:38:19.806618 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:19.806586    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgsh5\" (UniqueName: \"kubernetes.io/projected/1004eede-9d90-42b0-bd11-42a32056f2ae-kube-api-access-tgsh5\") pod \"metrics-server-bfd9f4b99-p95mj\" (UID: \"1004eede-9d90-42b0-bd11-42a32056f2ae\") " pod="openshift-monitoring/metrics-server-bfd9f4b99-p95mj"
Oct 30 20:38:19.922280 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:19.922182    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-bfd9f4b99-p95mj"
Oct 30 20:38:20.344702 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:20.344663    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-bfd9f4b99-p95mj"]
Oct 30 20:38:20.348094 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:38:20.348067    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1004eede_9d90_42b0_bd11_42a32056f2ae.slice/crio-0fd6ba6bbeee6529a916683a80a42d504623480e41b88ec7ba4b3c949def3b23 WatchSource:0}: Error finding container 0fd6ba6bbeee6529a916683a80a42d504623480e41b88ec7ba4b3c949def3b23: Status 404 returned error can't find the container with id 0fd6ba6bbeee6529a916683a80a42d504623480e41b88ec7ba4b3c949def3b23
Oct 30 20:38:20.368055 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:20.368028    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-bfd9f4b99-p95mj" event={"ID":"1004eede-9d90-42b0-bd11-42a32056f2ae","Type":"ContainerStarted","Data":"0fd6ba6bbeee6529a916683a80a42d504623480e41b88ec7ba4b3c949def3b23"}
Oct 30 20:38:20.369575 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:20.369549    2576 generic.go:356] "Generic (PLEG): container finished" podID="1c4609ce-58a7-47f3-8eaa-0f19cea5c6db" containerID="d6b9d3c29eb082eb14cbc0dd4664369c791f40d5a75aeeb98689299537f59f75" exitCode=0
Oct 30 20:38:20.369656 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:20.369619    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db","Type":"ContainerDied","Data":"d6b9d3c29eb082eb14cbc0dd4664369c791f40d5a75aeeb98689299537f59f75"}
Oct 30 20:38:20.914876 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:20.914846    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-k58xq_b25a9500-23b1-42d8-8a77-ca75dabc6ef5/dns/0.log"
Oct 30 20:38:21.019521 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:21.019493    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-k58xq_b25a9500-23b1-42d8-8a77-ca75dabc6ef5/kube-rbac-proxy/0.log"
Oct 30 20:38:21.197275 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:21.197249    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-d69bq_c7a31d20-2bf3-4c38-8816-2181571a1fda/dns-node-resolver/0.log"
Oct 30 20:38:22.277772 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:22.277751    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:38:22.378351 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:22.378318    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-bfd9f4b99-p95mj" event={"ID":"1004eede-9d90-42b0-bd11-42a32056f2ae","Type":"ContainerStarted","Data":"3fb12bb932fcd2223956da7a46de4d4ed681a348990210b52cb067809511b911"}
Oct 30 20:38:22.402477 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:22.402451    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jgvrx_520d950c-2ec5-457f-b55f-51bd29207757/serve-healthcheck-canary/0.log"
Oct 30 20:38:22.613598 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:22.613499    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-bfd9f4b99-p95mj" podStartSLOduration=1.7458650329999998 podStartE2EDuration="3.613483526s" totalImagesPullingTime="1.867618493s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:38:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:22.612855564 +0000 UTC m=+58.030130244" watchObservedRunningTime="2026-10-30 20:38:22.613483526 +0000 UTC m=+58.030758205"
Oct 30 20:38:24.336296 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.336265    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-k58xq"
Oct 30 20:38:24.364112 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.364082    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 20:38:24.366606 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.366585    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:24.387396 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.387365    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:24.387630 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.387407    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:24.387630 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.387434    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:24.387630 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.387512    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:24.387630 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.387560    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-web-config\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:24.387920 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.387672    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:24.387920 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.387708    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:24.387920 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.387771    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:24.387920 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.387821    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:24.387920 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.387850    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx4tv\" (UniqueName: \"kubernetes.io/projected/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-kube-api-access-rx4tv\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:24.388131 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.387948    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:24.388131 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.388014    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:24.388131 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.388071    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:24.388131 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.388091    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:24.388320 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.388249    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:24.388320 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.388287    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-config\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:24.388320 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.388313    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:24.388436 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.388331    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:24.388436 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.388370    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:24.388436 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.388427    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-config-out\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:24.419077 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.419048    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 20:38:24.419317 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.419301    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-szlhz\""
Oct 30 20:38:24.419366 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.419323    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 20:38:24.419366 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.419325    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-55jjhefcr4ig6\""
Oct 30 20:38:24.437191 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.437158    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 20:38:24.437191 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.437159    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 20:38:24.437436 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.437215    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 20:38:24.437436 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.437304    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 20:38:24.468464 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.468431    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 20:38:24.468464 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.468435    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 20:38:24.468679 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.468480    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 20:38:24.469694 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.469675    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 20:38:24.489268 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.489231    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-config-out\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:24.489418 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.489269    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:24.489418 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.489297    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:24.489418 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.489317    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:24.489418 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.489335    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:24.489418 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.489364    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-web-config\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:24.489664 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.489530    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:24.489664 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.489558    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:24.489664 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.489588    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:24.489664 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.489614    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:24.490806 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.489636    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx4tv\" (UniqueName: \"kubernetes.io/projected/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-kube-api-access-rx4tv\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:24.490926 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.490844    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:24.491624 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.491602    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:24.491729 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.491653    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:24.491729 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.491686    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:24.491861 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.491727    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:24.491861 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.491755    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-config\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:24.491861 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.491818    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:24.491861 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.491846    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:24.492056 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.491868    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:24.492056 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.490192    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:24.492056 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.491088    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:24.492056 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.490209    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:24.492056 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.490332    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:24.492056 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.491529    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:24.492056 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.490724    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:24.492554 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.492531    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-web-config\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:24.492709 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.492688    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-config-out\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:24.492767 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.492756    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:24.494950 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.494902    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-config\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:24.495445 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.495424    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:24.495807 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.495765    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:24.496043 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.496004    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:24.496256 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.496237    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:24.567537 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.567504    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 20:38:24.567779 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.567754    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 20:38:24.567881 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.567782    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 20:38:24.567881 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.567820    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 20:38:24.567985 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.567880    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 20:38:24.570479 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.570456    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx4tv\" (UniqueName: \"kubernetes.io/projected/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-kube-api-access-rx4tv\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:24.570564 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.570512    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:24.571835 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.571813    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:24.575072 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.575053    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:24.575150 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.575077    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:24.575254 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.575237    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:24.684124 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:24.684096    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:25.160856 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:25.160762    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 20:38:25.165049 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:38:25.165013    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbe18c50_e3ff_4684_b9a1_cb5a83f6d6df.slice/crio-ef111d4318a24f1b18f799bb083297e58132a20086bb28aedf2679e9b3dc1913 WatchSource:0}: Error finding container ef111d4318a24f1b18f799bb083297e58132a20086bb28aedf2679e9b3dc1913: Status 404 returned error can't find the container with id ef111d4318a24f1b18f799bb083297e58132a20086bb28aedf2679e9b3dc1913
Oct 30 20:38:25.393056 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:25.393022    2576 generic.go:356] "Generic (PLEG): container finished" podID="bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" containerID="521dd12d5bf7bf42e12d8d6d5ac01b10c9993cfd8c508bb4c92dea2fc5092310" exitCode=0
Oct 30 20:38:25.393056 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:25.393060    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df","Type":"ContainerDied","Data":"521dd12d5bf7bf42e12d8d6d5ac01b10c9993cfd8c508bb4c92dea2fc5092310"}
Oct 30 20:38:25.393577 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:25.393122    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df","Type":"ContainerStarted","Data":"ef111d4318a24f1b18f799bb083297e58132a20086bb28aedf2679e9b3dc1913"}
Oct 30 20:38:26.400650 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:26.400567    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db","Type":"ContainerStarted","Data":"bd8e64fb9dc49d675e305dfe67dd2ac11c07e5014c92311e685bc19270a55b8b"}
Oct 30 20:38:26.401084 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:26.400668    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db","Type":"ContainerStarted","Data":"a70e118fb15f6bb5edca24ed3dfeb614808b8b6100704b9dbdfb9be04e21f3a7"}
Oct 30 20:38:26.401084 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:26.400695    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db","Type":"ContainerStarted","Data":"f27e0161f269a31de0d92e4b6943ef3c333705a2895885f8d95a3aec9b5a248a"}
Oct 30 20:38:26.401084 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:26.400728    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db","Type":"ContainerStarted","Data":"c7296d272b2a7cdb3a06dff55dcf6811d03f31db5bfc6882918389bfb6e3b6cd"}
Oct 30 20:38:26.401084 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:26.400753    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db","Type":"ContainerStarted","Data":"d38e224686999e89588a3fa2b45d1c29d2c94553d17e7e25e2e3251c0281ab4a"}
Oct 30 20:38:27.408496 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:27.408413    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db","Type":"ContainerStarted","Data":"d140b08b6210f2b3667f398b74f0086088403d96b90c4b000f43ea224ac6b458"}
Oct 30 20:38:27.533628 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:27.533560    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.503552253 podStartE2EDuration="11.533544402s" totalImagesPullingTime="8.017133761s" totalInitContainerRuntime="12.858388ms" isStatefulPod=false podCreationTimestamp="2026-10-30 20:38:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:27.532338891 +0000 UTC m=+62.949613576" watchObservedRunningTime="2026-10-30 20:38:27.533544402 +0000 UTC m=+62.950819084"
Oct 30 20:38:29.423488 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:29.423455    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df","Type":"ContainerStarted","Data":"5d9bc8b61136c18c540be963353fc2c21014d8179a9d7d563d8b939d136c8ce5"}
Oct 30 20:38:29.423892 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:29.423526    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df","Type":"ContainerStarted","Data":"cea18e6abdfde296bb779d83752561bccff0c9547b9829446ca54e81b9c71bad"}
Oct 30 20:38:30.050998 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:30.050961    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/242a67ca-f9ea-431d-adee-ece061468a02-metrics-certs\") pod \"network-metrics-daemon-hjssq\" (UID: \"242a67ca-f9ea-431d-adee-ece061468a02\") " pod="openshift-multus/network-metrics-daemon-hjssq"
Oct 30 20:38:30.051281 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:30.051061    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a74da60-0284-410b-9d69-0d5192ce0928-original-pull-secret\") pod \"global-pull-secret-syncer-xj584\" (UID: \"1a74da60-0284-410b-9d69-0d5192ce0928\") " pod="kube-system/global-pull-secret-syncer-xj584"
Oct 30 20:38:30.102164 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:30.102130    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 20:38:30.114699 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:30.114666    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/242a67ca-f9ea-431d-adee-ece061468a02-metrics-certs\") pod \"network-metrics-daemon-hjssq\" (UID: \"242a67ca-f9ea-431d-adee-ece061468a02\") " pod="openshift-multus/network-metrics-daemon-hjssq"
Oct 30 20:38:30.151350 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:30.151327    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 20:38:30.151634 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:30.151614    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzhsg\" (UniqueName: \"kubernetes.io/projected/e54d0918-84e8-40f3-9b18-6e2fafb3d500-kube-api-access-bzhsg\") pod \"network-check-target-kbpjj\" (UID: \"e54d0918-84e8-40f3-9b18-6e2fafb3d500\") " pod="openshift-network-diagnostics/network-check-target-kbpjj"
Oct 30 20:38:30.164045 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:30.164013    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a74da60-0284-410b-9d69-0d5192ce0928-original-pull-secret\") pod \"global-pull-secret-syncer-xj584\" (UID: \"1a74da60-0284-410b-9d69-0d5192ce0928\") " pod="kube-system/global-pull-secret-syncer-xj584"
Oct 30 20:38:30.202291 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:30.202266    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 20:38:30.262965 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:30.262933    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 20:38:30.283296 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:30.283274    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzhsg\" (UniqueName: \"kubernetes.io/projected/e54d0918-84e8-40f3-9b18-6e2fafb3d500-kube-api-access-bzhsg\") pod \"network-check-target-kbpjj\" (UID: \"e54d0918-84e8-40f3-9b18-6e2fafb3d500\") " pod="openshift-network-diagnostics/network-check-target-kbpjj"
Oct 30 20:38:30.318042 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:30.317980    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4lpn7\""
Oct 30 20:38:30.320904 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:30.320887    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-49f2v\""
Oct 30 20:38:30.327992 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:30.327969    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hjssq"
Oct 30 20:38:30.330496 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:30.330476    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xj584"
Oct 30 20:38:30.626012 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:30.625940    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-hjmcj\""
Oct 30 20:38:30.635913 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:30.635896    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbpjj"
Oct 30 20:38:30.767684 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:30.767661    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xj584"]
Oct 30 20:38:30.770416 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:38:30.770390    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a74da60_0284_410b_9d69_0d5192ce0928.slice/crio-0556d7ef4363eda2c99088bf88103c5f529286eff610cf928bd5fd364d59661f WatchSource:0}: Error finding container 0556d7ef4363eda2c99088bf88103c5f529286eff610cf928bd5fd364d59661f: Status 404 returned error can't find the container with id 0556d7ef4363eda2c99088bf88103c5f529286eff610cf928bd5fd364d59661f
Oct 30 20:38:30.821349 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:38:30.821315    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod242a67ca_f9ea_431d_adee_ece061468a02.slice/crio-50613bde83fde8f4cbc68785d926585e85b6115110d8e3b0e9ec5dbd554007d3 WatchSource:0}: Error finding container 50613bde83fde8f4cbc68785d926585e85b6115110d8e3b0e9ec5dbd554007d3: Status 404 returned error can't find the container with id 50613bde83fde8f4cbc68785d926585e85b6115110d8e3b0e9ec5dbd554007d3
Oct 30 20:38:30.916495 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:30.916426    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hjssq"]
Oct 30 20:38:31.009405 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:31.009320    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kbpjj"]
Oct 30 20:38:31.011692 ip-10-0-128-165 kubenswrapper[2576]: W1030 20:38:31.011665    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode54d0918_84e8_40f3_9b18_6e2fafb3d500.slice/crio-4e21ab7264b551a8ab03a4c48dbf1cd0249e6358baf993bf958430e2be9d8ae5 WatchSource:0}: Error finding container 4e21ab7264b551a8ab03a4c48dbf1cd0249e6358baf993bf958430e2be9d8ae5: Status 404 returned error can't find the container with id 4e21ab7264b551a8ab03a4c48dbf1cd0249e6358baf993bf958430e2be9d8ae5
Oct 30 20:38:31.434581 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:31.434550    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df","Type":"ContainerStarted","Data":"b2305237333dd77b38d54b1bf6d4e64a02247bcb4e8d0baf57949b5b2a0b8893"}
Oct 30 20:38:31.435930 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:31.435902    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kbpjj" event={"ID":"e54d0918-84e8-40f3-9b18-6e2fafb3d500","Type":"ContainerStarted","Data":"4e21ab7264b551a8ab03a4c48dbf1cd0249e6358baf993bf958430e2be9d8ae5"}
Oct 30 20:38:31.437532 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:31.437488    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xj584" event={"ID":"1a74da60-0284-410b-9d69-0d5192ce0928","Type":"ContainerStarted","Data":"0556d7ef4363eda2c99088bf88103c5f529286eff610cf928bd5fd364d59661f"}
Oct 30 20:38:31.438902 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:31.438881    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hjssq" event={"ID":"242a67ca-f9ea-431d-adee-ece061468a02","Type":"ContainerStarted","Data":"50613bde83fde8f4cbc68785d926585e85b6115110d8e3b0e9ec5dbd554007d3"}
Oct 30 20:38:32.462021 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:32.461985    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df","Type":"ContainerStarted","Data":"bada155176449de4f95a9134cc2bcf9d9a6e49153aa34d6b19d69a007ec22a66"}
Oct 30 20:38:32.462504 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:32.462094    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df","Type":"ContainerStarted","Data":"ac90672f57cdaaed8c17ddf63fec27de5e7299ed8d999f071519583ed0c0229b"}
Oct 30 20:38:32.462504 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:32.462139    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df","Type":"ContainerStarted","Data":"9d4a60f970f9c790fb7a15f66d0c939c5cfe097422fc7390b667543e90ec02d7"}
Oct 30 20:38:33.468296 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:33.468246    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hjssq" event={"ID":"242a67ca-f9ea-431d-adee-ece061468a02","Type":"ContainerStarted","Data":"3163cdb3631726ad1f9f76bcc12edbe48a0c859849f7f77e5356c5c14987e072"}
Oct 30 20:38:33.468676 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:33.468326    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hjssq" event={"ID":"242a67ca-f9ea-431d-adee-ece061468a02","Type":"ContainerStarted","Data":"fcbecd9c0d132a16dcdf7c6e2df291e1c10b8bfc1238b40012f150173471f6f0"}
Oct 30 20:38:33.582962 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:33.582901    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.71049756 podStartE2EDuration="9.582878177s" totalImagesPullingTime="5.852503314s" totalInitContainerRuntime="19.877303ms" isStatefulPod=false podCreationTimestamp="2026-10-30 20:38:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:32.638924412 +0000 UTC m=+68.056199096" watchObservedRunningTime="2026-10-30 20:38:33.582878177 +0000 UTC m=+69.000152858"
Oct 30 20:38:33.583182 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:33.583150    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-hjssq" podStartSLOduration=65.798966218 podStartE2EDuration="1m7.583145092s" totalImagesPullingTime="1.784178874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:37:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:33.581349899 +0000 UTC m=+68.998624591" watchObservedRunningTime="2026-10-30 20:38:33.583145092 +0000 UTC m=+69.000419772"
Oct 30 20:38:34.684807 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:34.684763    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:35.478468 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:35.477977    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kbpjj" event={"ID":"e54d0918-84e8-40f3-9b18-6e2fafb3d500","Type":"ContainerStarted","Data":"606220b8056db0e04e838b0a24a1d1373071458184ff378cdb9322f9d70635c4"}
Oct 30 20:38:35.478468 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:35.478137    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kbpjj"
Oct 30 20:38:35.590064 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:35.590004    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-kbpjj" podStartSLOduration=65.902777115 podStartE2EDuration="1m9.589987532s" totalImagesPullingTime="3.687210417s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:37:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:35.588879226 +0000 UTC m=+71.006153907" watchObservedRunningTime="2026-10-30 20:38:35.589987532 +0000 UTC m=+71.007262211"
Oct 30 20:38:36.873593 ip-10-0-128-165 kubenswrapper[2576]: I1030 20:38:36.873477    2576 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 20:38:36.873611 ip-10-0-128-165 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 20:38:36.880649 ip-10-0-128-165 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 20:38:36.880933 ip-10-0-128-165 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 20:38:36.881188 ip-10-0-128-165 systemd[1]: kubelet.service: Consumed 4.340s CPU time.
Oct 30 20:38:36.898466 ip-10-0-128-165 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 20:38:36.967912 ip-10-0-128-165 kubenswrapper[5713]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 20:38:36.967912 ip-10-0-128-165 kubenswrapper[5713]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 20:38:36.967912 ip-10-0-128-165 kubenswrapper[5713]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 20:38:36.967912 ip-10-0-128-165 kubenswrapper[5713]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 20:38:36.970069 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.969977    5713 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:38:36.970069 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970070    5713 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:38:36.970143 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970074    5713 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:38:36.970143 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970077    5713 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:38:36.970143 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970081    5713 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:38:36.970143 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970084    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:38:36.970143 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970087    5713 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:38:36.970143 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970090    5713 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:38:36.970143 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970093    5713 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:38:36.970143 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970095    5713 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:38:36.970143 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970098    5713 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:38:36.970143 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970101    5713 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:38:36.970143 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970103    5713 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:38:36.970143 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970106    5713 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:38:36.970143 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970108    5713 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:38:36.970143 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970111    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:38:36.970143 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970113    5713 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:38:36.970143 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970116    5713 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:38:36.970143 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970119    5713 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:38:36.970143 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970121    5713 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:38:36.970143 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970126    5713 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:38:36.970143 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970129    5713 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:38:36.970644 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970133    5713 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:38:36.970644 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970135    5713 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:38:36.970644 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970137    5713 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:38:36.970644 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970140    5713 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:38:36.970644 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970142    5713 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:38:36.970644 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970145    5713 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:38:36.970644 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970148    5713 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:38:36.970644 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970150    5713 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:38:36.970644 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970153    5713 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:38:36.970644 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970156    5713 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:38:36.970644 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970160    5713 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:38:36.970644 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970162    5713 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:38:36.970644 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970165    5713 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:38:36.970644 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970167    5713 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:38:36.970644 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970170    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:38:36.970644 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970172    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:38:36.970644 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970174    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:38:36.970644 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970178    5713 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:38:36.970644 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970181    5713 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:38:36.971175 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970183    5713 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:38:36.971175 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970186    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:38:36.971175 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970188    5713 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:38:36.971175 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970191    5713 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:38:36.971175 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970193    5713 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:38:36.971175 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970196    5713 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:38:36.971175 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970198    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:38:36.971175 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970201    5713 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:38:36.971175 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970206    5713 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:38:36.971175 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970210    5713 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:38:36.971175 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970214    5713 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:38:36.971175 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970217    5713 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:38:36.971175 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970220    5713 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:38:36.971175 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970223    5713 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:38:36.971175 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970226    5713 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:38:36.971175 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970228    5713 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:38:36.971175 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970231    5713 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:38:36.971175 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970233    5713 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:38:36.971175 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970237    5713 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:38:36.971649 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970241    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:38:36.971649 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970244    5713 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:38:36.971649 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970247    5713 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:38:36.971649 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970249    5713 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:38:36.971649 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970252    5713 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:38:36.971649 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970255    5713 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:38:36.971649 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970258    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:38:36.971649 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970262    5713 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:38:36.971649 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970265    5713 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:38:36.971649 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970267    5713 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:38:36.971649 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970270    5713 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:38:36.971649 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970273    5713 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:38:36.971649 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970276    5713 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:38:36.971649 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970279    5713 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:38:36.971649 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970281    5713 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:38:36.971649 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970284    5713 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:38:36.971649 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970287    5713 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:38:36.971649 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970289    5713 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:38:36.971649 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970292    5713 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:38:36.971649 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970295    5713 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:38:36.972184 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970298    5713 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:38:36.972184 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970301    5713 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:38:36.972184 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970303    5713 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:38:36.972184 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970305    5713 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:38:36.972184 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970308    5713 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:38:36.972184 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970310    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:38:36.972184 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970313    5713 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:38:36.972184 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970315    5713 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:38:36.972184 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970318    5713 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:38:36.972184 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970320    5713 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:38:36.972184 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970324    5713 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:38:36.972184 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970326    5713 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:38:36.972184 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970329    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:38:36.972184 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970332    5713 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:38:36.972184 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970334    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:38:36.972184 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970337    5713 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:38:36.972184 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970339    5713 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:38:36.972184 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970342    5713 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:38:36.972184 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970344    5713 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:38:36.972184 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970347    5713 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:38:36.972749 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970350    5713 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:38:36.972749 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970352    5713 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:38:36.972749 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970356    5713 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:38:36.972749 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970358    5713 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:38:36.972749 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970363    5713 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:38:36.972749 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.970366    5713 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:38:36.972749 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971208    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:38:36.972749 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971215    5713 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:38:36.972749 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971219    5713 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:38:36.972749 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971221    5713 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:38:36.972749 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971224    5713 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:38:36.972749 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971228    5713 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:38:36.972749 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971232    5713 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:38:36.972749 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971235    5713 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:38:36.972749 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971238    5713 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:38:36.972749 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971241    5713 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:38:36.972749 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971245    5713 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:38:36.972749 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971247    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:38:36.972749 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971250    5713 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:38:36.972749 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971253    5713 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:38:36.973267 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971256    5713 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:38:36.973267 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971258    5713 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:38:36.973267 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971261    5713 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:38:36.973267 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971264    5713 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:38:36.973267 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971267    5713 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:38:36.973267 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971269    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:38:36.973267 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971272    5713 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:38:36.973267 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971274    5713 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:38:36.973267 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971277    5713 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:38:36.973267 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971279    5713 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:38:36.973267 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971282    5713 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:38:36.973267 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971285    5713 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:38:36.973267 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971287    5713 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:38:36.973267 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971290    5713 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:38:36.973267 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971292    5713 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:38:36.973267 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971295    5713 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:38:36.973267 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971298    5713 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:38:36.973267 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971300    5713 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:38:36.973267 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971303    5713 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:38:36.973779 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971306    5713 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:38:36.973779 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971309    5713 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:38:36.973779 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971312    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:38:36.973779 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971314    5713 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:38:36.973779 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971317    5713 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:38:36.973779 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971320    5713 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:38:36.973779 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971322    5713 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:38:36.973779 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971324    5713 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:38:36.973779 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971327    5713 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:38:36.973779 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971329    5713 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:38:36.973779 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971332    5713 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:38:36.973779 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971335    5713 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:38:36.973779 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971337    5713 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:38:36.973779 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971340    5713 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:38:36.973779 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971342    5713 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:38:36.973779 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971345    5713 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:38:36.973779 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971347    5713 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:38:36.973779 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971350    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:38:36.973779 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971353    5713 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:38:36.973779 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971356    5713 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:38:36.974283 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971358    5713 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:38:36.974283 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971361    5713 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:38:36.974283 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971363    5713 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:38:36.974283 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971366    5713 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:38:36.974283 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971368    5713 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:38:36.974283 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971371    5713 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:38:36.974283 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971373    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:38:36.974283 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971382    5713 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:38:36.974283 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971385    5713 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:38:36.974283 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971387    5713 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:38:36.974283 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971390    5713 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:38:36.974283 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971393    5713 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:38:36.974283 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971395    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:38:36.974283 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971398    5713 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:38:36.974283 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971401    5713 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:38:36.974283 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971403    5713 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:38:36.974283 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971406    5713 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:38:36.974283 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971408    5713 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:38:36.974283 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971411    5713 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:38:36.974822 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971414    5713 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:38:36.974822 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971416    5713 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:38:36.974822 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971419    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:38:36.974822 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971421    5713 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:38:36.974822 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971424    5713 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:38:36.974822 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971426    5713 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:38:36.974822 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971429    5713 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:38:36.974822 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971431    5713 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:38:36.974822 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971437    5713 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:38:36.974822 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971440    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:38:36.974822 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971442    5713 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:38:36.974822 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971445    5713 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:38:36.974822 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971448    5713 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:38:36.974822 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971450    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:38:36.974822 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971453    5713 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:38:36.974822 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971456    5713 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:38:36.974822 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971458    5713 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:38:36.974822 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971461    5713 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:38:36.974822 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971463    5713 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:38:36.975294 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971466    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:38:36.975294 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971469    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:38:36.975294 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971471    5713 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:38:36.975294 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971474    5713 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:38:36.975294 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971476    5713 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:38:36.975294 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971478    5713 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:38:36.975294 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971483    5713 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:38:36.975294 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971487    5713 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:38:36.975294 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971490    5713 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:38:36.975294 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971493    5713 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:38:36.975294 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971496    5713 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:38:36.975294 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971498    5713 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:38:36.975294 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971501    5713 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:38:36.975294 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971504    5713 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:38:36.975294 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.971506    5713 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:38:36.975294 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972087    5713 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:38:36.975294 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972095    5713 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:38:36.975294 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972098    5713 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:38:36.975294 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972100    5713 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:38:36.975759 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972104    5713 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:38:36.975759 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972109    5713 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:38:36.975759 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972113    5713 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:38:36.975759 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972117    5713 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:38:36.975759 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972121    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:38:36.975759 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972124    5713 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:38:36.975759 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972127    5713 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:38:36.975759 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972130    5713 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:38:36.975759 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972132    5713 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:38:36.975759 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972135    5713 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:38:36.975759 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972137    5713 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:38:36.975759 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972140    5713 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:38:36.975759 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972143    5713 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:38:36.975759 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972145    5713 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:38:36.975759 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972148    5713 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:38:36.975759 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972150    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:38:36.975759 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972153    5713 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:38:36.975759 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972155    5713 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:38:36.976241 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972158    5713 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:38:36.976241 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972161    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:38:36.976241 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972167    5713 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:38:36.976241 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972170    5713 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:38:36.976241 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972172    5713 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:38:36.976241 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972175    5713 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:38:36.976241 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972177    5713 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:38:36.976241 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972180    5713 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:38:36.976241 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972182    5713 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:38:36.976241 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972185    5713 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:38:36.976241 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972187    5713 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:38:36.976241 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972190    5713 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:38:36.976241 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972193    5713 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:38:36.976241 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972195    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:38:36.976241 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972198    5713 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:38:36.976241 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972201    5713 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:38:36.976241 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972204    5713 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:38:36.976241 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972206    5713 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:38:36.976241 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972209    5713 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:38:36.976241 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972211    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:38:36.976763 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972214    5713 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:38:36.976763 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972217    5713 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:38:36.976763 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972219    5713 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:38:36.976763 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972222    5713 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:38:36.976763 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972224    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:38:36.976763 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972226    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:38:36.976763 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972228    5713 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:38:36.976763 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972231    5713 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:38:36.976763 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972233    5713 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:38:36.976763 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972236    5713 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:38:36.976763 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972239    5713 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:38:36.976763 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972241    5713 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:38:36.976763 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972244    5713 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:38:36.976763 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972247    5713 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:38:36.976763 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972250    5713 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:38:36.976763 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972253    5713 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:38:36.976763 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972256    5713 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:38:36.976763 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972258    5713 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:38:36.976763 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972260    5713 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:38:36.976763 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972263    5713 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:38:36.977274 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972266    5713 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:38:36.977274 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972268    5713 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:38:36.977274 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972270    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:38:36.977274 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972273    5713 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:38:36.977274 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972275    5713 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:38:36.977274 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972277    5713 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:38:36.977274 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972280    5713 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:38:36.977274 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972282    5713 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:38:36.977274 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972284    5713 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:38:36.977274 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972287    5713 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:38:36.977274 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972289    5713 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:38:36.977274 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972292    5713 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:38:36.977274 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972294    5713 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:38:36.977274 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972297    5713 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:38:36.977274 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972299    5713 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:38:36.977274 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972301    5713 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:38:36.977274 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972304    5713 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:38:36.977274 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972306    5713 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:38:36.977274 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972309    5713 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:38:36.977274 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972312    5713 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:38:36.977980 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972314    5713 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:38:36.977980 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972317    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:38:36.977980 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972319    5713 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:38:36.977980 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972322    5713 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:38:36.977980 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972324    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:38:36.977980 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972327    5713 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:38:36.977980 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972329    5713 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:38:36.977980 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972331    5713 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:38:36.977980 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972340    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:38:36.977980 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972343    5713 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:38:36.977980 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972345    5713 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:38:36.977980 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972347    5713 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:38:36.977980 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972350    5713 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:38:36.977980 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972352    5713 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:38:36.977980 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972355    5713 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:38:36.977980 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972357    5713 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:38:36.977980 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972360    5713 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:38:36.977980 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972363    5713 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:38:36.977980 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972365    5713 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:38:36.977980 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972367    5713 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:38:36.978663 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972370    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:38:36.978663 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972373    5713 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:38:36.978663 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972375    5713 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:38:36.978663 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.972378    5713 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:38:36.978663 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972486    5713 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 20:38:36.978663 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972499    5713 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 20:38:36.978663 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972514    5713 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 20:38:36.978663 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972519    5713 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 20:38:36.978663 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972523    5713 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 20:38:36.978663 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972527    5713 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 20:38:36.978663 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972531    5713 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 20:38:36.978663 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972535    5713 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 20:38:36.978663 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972541    5713 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 20:38:36.978663 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972544    5713 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 20:38:36.978663 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972547    5713 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 20:38:36.978663 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972550    5713 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 20:38:36.978663 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972554    5713 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 20:38:36.978663 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972557    5713 flags.go:64] FLAG: --cgroup-root=""
Oct 30 20:38:36.978663 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972566    5713 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 20:38:36.978663 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972570    5713 flags.go:64] FLAG: --client-ca-file=""
Oct 30 20:38:36.978663 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972572    5713 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 20:38:36.978663 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972581    5713 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 20:38:36.978663 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972588    5713 flags.go:64] FLAG: --cluster-domain=""
Oct 30 20:38:36.979259 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972591    5713 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 20:38:36.979259 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972594    5713 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 20:38:36.979259 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972597    5713 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 20:38:36.979259 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972600    5713 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 20:38:36.979259 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972604    5713 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 20:38:36.979259 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972607    5713 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 20:38:36.979259 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972610    5713 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 20:38:36.979259 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972613    5713 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 20:38:36.979259 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972616    5713 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 20:38:36.979259 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972619    5713 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 20:38:36.979259 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972623    5713 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 20:38:36.979259 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972626    5713 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 20:38:36.979259 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972629    5713 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 20:38:36.979259 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972634    5713 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 20:38:36.979259 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972637    5713 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 20:38:36.979259 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972640    5713 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 20:38:36.979259 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972643    5713 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 20:38:36.979259 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972646    5713 flags.go:64] FLAG: --enable-server="true"
Oct 30 20:38:36.979259 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972649    5713 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 20:38:36.979259 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972656    5713 flags.go:64] FLAG: --event-burst="100"
Oct 30 20:38:36.979259 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972659    5713 flags.go:64] FLAG: --event-qps="50"
Oct 30 20:38:36.979259 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972664    5713 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 20:38:36.979259 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972667    5713 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 20:38:36.979259 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972670    5713 flags.go:64] FLAG: --eviction-hard=""
Oct 30 20:38:36.979876 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972674    5713 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 20:38:36.979876 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972678    5713 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 20:38:36.979876 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972681    5713 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 20:38:36.979876 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972684    5713 flags.go:64] FLAG: --eviction-soft=""
Oct 30 20:38:36.979876 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972687    5713 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 20:38:36.979876 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972690    5713 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 20:38:36.979876 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972693    5713 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 20:38:36.979876 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972702    5713 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 20:38:36.979876 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972705    5713 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 20:38:36.979876 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972707    5713 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 20:38:36.979876 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972710    5713 flags.go:64] FLAG: --feature-gates=""
Oct 30 20:38:36.979876 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972715    5713 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 20:38:36.979876 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972718    5713 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 20:38:36.979876 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972721    5713 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 20:38:36.979876 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972724    5713 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 20:38:36.979876 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972729    5713 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 20:38:36.979876 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972732    5713 flags.go:64] FLAG: --help="false"
Oct 30 20:38:36.979876 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972735    5713 flags.go:64] FLAG: --hostname-override="ip-10-0-128-165.us-west-1.compute.internal"
Oct 30 20:38:36.979876 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972739    5713 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 20:38:36.979876 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972742    5713 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 20:38:36.979876 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972745    5713 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 20:38:36.979876 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972748    5713 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 20:38:36.979876 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972752    5713 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 20:38:36.980456 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972755    5713 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 20:38:36.980456 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972758    5713 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 20:38:36.980456 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972761    5713 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 20:38:36.980456 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972764    5713 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 20:38:36.980456 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972767    5713 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 20:38:36.980456 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972770    5713 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 20:38:36.980456 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972775    5713 flags.go:64] FLAG: --kube-reserved=""
Oct 30 20:38:36.980456 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972778    5713 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 20:38:36.980456 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972781    5713 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 20:38:36.980456 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972798    5713 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 20:38:36.980456 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972801    5713 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 20:38:36.980456 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972804    5713 flags.go:64] FLAG: --lock-file=""
Oct 30 20:38:36.980456 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972807    5713 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 20:38:36.980456 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972810    5713 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 20:38:36.980456 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972813    5713 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 20:38:36.980456 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972819    5713 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 20:38:36.980456 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972823    5713 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 20:38:36.980456 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972834    5713 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 20:38:36.980456 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972837    5713 flags.go:64] FLAG: --logging-format="text"
Oct 30 20:38:36.980456 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972840    5713 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 20:38:36.980456 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972844    5713 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 20:38:36.980456 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972847    5713 flags.go:64] FLAG: --manifest-url=""
Oct 30 20:38:36.980456 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972850    5713 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 20:38:36.980456 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972854    5713 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 20:38:36.980456 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972860    5713 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 20:38:36.981090 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972864    5713 flags.go:64] FLAG: --max-pods="110"
Oct 30 20:38:36.981090 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972867    5713 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 20:38:36.981090 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972870    5713 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 20:38:36.981090 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972873    5713 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 20:38:36.981090 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972876    5713 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 20:38:36.981090 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972878    5713 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 20:38:36.981090 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972881    5713 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 20:38:36.981090 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972884    5713 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 20:38:36.981090 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972892    5713 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 20:38:36.981090 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972895    5713 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 20:38:36.981090 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972898    5713 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 20:38:36.981090 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972901    5713 flags.go:64] FLAG: --pod-cidr=""
Oct 30 20:38:36.981090 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972904    5713 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 20:38:36.981090 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972908    5713 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 20:38:36.981090 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972912    5713 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 20:38:36.981090 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972915    5713 flags.go:64] FLAG: --port="10250"
Oct 30 20:38:36.981090 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972918    5713 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 20:38:36.981090 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972921    5713 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0153b63859c7ccd86"
Oct 30 20:38:36.981090 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972924    5713 flags.go:64] FLAG: --qos-reserved=""
Oct 30 20:38:36.981090 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972927    5713 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 20:38:36.981090 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972931    5713 flags.go:64] FLAG: --register-node="true"
Oct 30 20:38:36.981090 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972934    5713 flags.go:64] FLAG: --register-with-taints=""
Oct 30 20:38:36.981090 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972938    5713 flags.go:64] FLAG: --registry-burst="10"
Oct 30 20:38:36.981090 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972941    5713 flags.go:64] FLAG: --registry-qps="5"
Oct 30 20:38:36.981090 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972944    5713 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 20:38:36.981090 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972954    5713 flags.go:64] FLAG: --reserved-memory=""
Oct 30 20:38:36.981733 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972959    5713 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 20:38:36.981733 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972962    5713 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 20:38:36.981733 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972965    5713 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 20:38:36.981733 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972968    5713 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 20:38:36.981733 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972971    5713 flags.go:64] FLAG: --runonce="false"
Oct 30 20:38:36.981733 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972975    5713 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 20:38:36.981733 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972978    5713 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 20:38:36.981733 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972982    5713 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 20:38:36.981733 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972985    5713 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 20:38:36.981733 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972988    5713 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 20:38:36.981733 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972991    5713 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 20:38:36.981733 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972994    5713 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 20:38:36.981733 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.972997    5713 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 20:38:36.981733 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.973000    5713 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 20:38:36.981733 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.973003    5713 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 20:38:36.981733 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.973006    5713 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 20:38:36.981733 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.973009    5713 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 20:38:36.981733 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.973012    5713 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 20:38:36.981733 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.973015    5713 flags.go:64] FLAG: --system-cgroups=""
Oct 30 20:38:36.981733 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.973019    5713 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 20:38:36.981733 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.973024    5713 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 20:38:36.981733 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.973027    5713 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 20:38:36.981733 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.973030    5713 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 20:38:36.981733 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.973037    5713 flags.go:64] FLAG: --tls-min-version=""
Oct 30 20:38:36.981733 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.973040    5713 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 20:38:36.982363 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.973043    5713 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 20:38:36.982363 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.973046    5713 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 20:38:36.982363 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.973049    5713 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 20:38:36.982363 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.973052    5713 flags.go:64] FLAG: --v="2"
Oct 30 20:38:36.982363 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.973057    5713 flags.go:64] FLAG: --version="false"
Oct 30 20:38:36.982363 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.973061    5713 flags.go:64] FLAG: --vmodule=""
Oct 30 20:38:36.982363 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.973065    5713 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 20:38:36.982363 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.973074    5713 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 20:38:36.982363 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973263    5713 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:38:36.982363 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973267    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:38:36.982363 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973270    5713 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:38:36.982363 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973273    5713 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:38:36.982363 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973277    5713 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:38:36.982363 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973280    5713 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:38:36.982363 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973283    5713 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:38:36.982363 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973287    5713 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:38:36.982363 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973291    5713 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:38:36.982363 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973294    5713 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:38:36.982363 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973296    5713 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:38:36.982363 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973299    5713 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:38:36.982363 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973303    5713 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:38:36.982932 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973307    5713 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:38:36.982932 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973311    5713 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:38:36.982932 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973314    5713 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:38:36.982932 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973316    5713 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:38:36.982932 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973323    5713 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:38:36.982932 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973328    5713 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:38:36.982932 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973330    5713 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:38:36.982932 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973333    5713 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:38:36.982932 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973335    5713 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:38:36.982932 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973338    5713 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:38:36.982932 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973341    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:38:36.982932 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973343    5713 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:38:36.982932 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973346    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:38:36.982932 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973348    5713 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:38:36.982932 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973351    5713 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:38:36.982932 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973354    5713 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:38:36.982932 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973356    5713 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:38:36.982932 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973358    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:38:36.982932 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973361    5713 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:38:36.982932 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973369    5713 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:38:36.983465 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973372    5713 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:38:36.983465 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973375    5713 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:38:36.983465 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973377    5713 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:38:36.983465 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973381    5713 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:38:36.983465 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973383    5713 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:38:36.983465 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973386    5713 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:38:36.983465 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973389    5713 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:38:36.983465 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973391    5713 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:38:36.983465 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973393    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:38:36.983465 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973396    5713 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:38:36.983465 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973399    5713 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:38:36.983465 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973401    5713 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:38:36.983465 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973403    5713 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:38:36.983465 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973406    5713 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:38:36.983465 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973408    5713 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:38:36.983465 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973411    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:38:36.983465 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973413    5713 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:38:36.983465 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973417    5713 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:38:36.983465 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973419    5713 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:38:36.983465 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973422    5713 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:38:36.984076 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973424    5713 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:38:36.984076 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973427    5713 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:38:36.984076 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973429    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:38:36.984076 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973432    5713 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:38:36.984076 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973435    5713 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:38:36.984076 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973437    5713 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:38:36.984076 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973440    5713 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:38:36.984076 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973442    5713 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:38:36.984076 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973445    5713 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:38:36.984076 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973447    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:38:36.984076 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973450    5713 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:38:36.984076 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973452    5713 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:38:36.984076 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973461    5713 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:38:36.984076 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973464    5713 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:38:36.984076 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973466    5713 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:38:36.984076 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973471    5713 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:38:36.984076 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973473    5713 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:38:36.984076 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973476    5713 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:38:36.984076 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973478    5713 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:38:36.984602 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973481    5713 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:38:36.984602 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973483    5713 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:38:36.984602 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973486    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:38:36.984602 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973488    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:38:36.984602 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973491    5713 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:38:36.984602 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973493    5713 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:38:36.984602 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973496    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:38:36.984602 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973498    5713 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:38:36.984602 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973501    5713 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:38:36.984602 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973503    5713 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:38:36.984602 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973507    5713 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:38:36.984602 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973510    5713 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:38:36.984602 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973512    5713 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:38:36.984602 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973514    5713 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:38:36.984602 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973517    5713 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:38:36.984602 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973519    5713 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:38:36.984602 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973521    5713 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:38:36.984602 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973524    5713 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:38:36.984602 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973526    5713 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:38:36.984602 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973529    5713 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:38:36.985388 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973531    5713 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:38:36.985388 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973533    5713 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:38:36.985388 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973536    5713 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:38:36.985388 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973538    5713 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:38:36.985388 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973541    5713 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:38:36.985388 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973543    5713 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:38:36.985388 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973551    5713 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:38:36.985388 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973554    5713 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:38:36.985388 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973558    5713 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:38:36.985388 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973560    5713 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:38:36.985388 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973563    5713 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:38:36.985388 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973565    5713 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:38:36.985388 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973568    5713 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:38:36.985388 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973570    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:38:36.985388 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.973597    5713 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 20:38:36.985388 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973646    5713 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:38:36.985831 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973651    5713 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:38:36.985831 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973656    5713 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:38:36.985831 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973659    5713 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:38:36.985831 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973662    5713 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:38:36.985831 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973665    5713 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:38:36.985831 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973668    5713 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:38:36.985831 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973672    5713 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:38:36.985831 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973675    5713 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:38:36.985831 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973677    5713 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:38:36.985831 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973680    5713 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:38:36.985831 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973683    5713 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:38:36.985831 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973686    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:38:36.985831 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973688    5713 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:38:36.985831 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973691    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:38:36.985831 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973693    5713 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:38:36.985831 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973696    5713 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:38:36.985831 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973698    5713 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:38:36.985831 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973701    5713 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:38:36.985831 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973703    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:38:36.986470 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973705    5713 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:38:36.986470 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973708    5713 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:38:36.986470 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973710    5713 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:38:36.986470 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973713    5713 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:38:36.986470 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973721    5713 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:38:36.986470 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973724    5713 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:38:36.986470 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973727    5713 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:38:36.986470 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973730    5713 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:38:36.986470 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973732    5713 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:38:36.986470 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973735    5713 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:38:36.986470 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973737    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:38:36.986470 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973739    5713 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:38:36.986470 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973742    5713 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:38:36.986470 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973744    5713 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:38:36.986470 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973747    5713 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:38:36.986470 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973749    5713 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:38:36.986470 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973752    5713 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:38:36.986470 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973755    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:38:36.986470 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973757    5713 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:38:36.986470 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973761    5713 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:38:36.987360 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973763    5713 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:38:36.987360 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973765    5713 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:38:36.987360 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973768    5713 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:38:36.987360 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973770    5713 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:38:36.987360 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973773    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:38:36.987360 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973777    5713 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:38:36.987360 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973779    5713 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:38:36.987360 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973782    5713 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:38:36.987360 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973798    5713 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:38:36.987360 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973801    5713 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:38:36.987360 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973804    5713 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:38:36.987360 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973806    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:38:36.987360 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973815    5713 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:38:36.987360 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973818    5713 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:38:36.987360 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973821    5713 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:38:36.987360 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973823    5713 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:38:36.987360 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973826    5713 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:38:36.987360 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973834    5713 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:38:36.987360 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973839    5713 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:38:36.988200 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973841    5713 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:38:36.988200 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973844    5713 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:38:36.988200 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973847    5713 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:38:36.988200 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973849    5713 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:38:36.988200 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973852    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:38:36.988200 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973854    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:38:36.988200 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973857    5713 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:38:36.988200 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973859    5713 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:38:36.988200 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973862    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:38:36.988200 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973864    5713 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:38:36.988200 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973866    5713 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:38:36.988200 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973869    5713 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:38:36.988200 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973873    5713 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:38:36.988200 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973875    5713 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:38:36.988200 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973878    5713 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:38:36.988200 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973880    5713 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:38:36.988200 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973883    5713 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:38:36.988200 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973886    5713 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:38:36.988200 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973888    5713 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:38:36.988200 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973890    5713 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:38:36.989055 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973893    5713 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:38:36.989055 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973895    5713 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:38:36.989055 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973897    5713 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:38:36.989055 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973900    5713 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:38:36.989055 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973902    5713 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:38:36.989055 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973905    5713 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:38:36.989055 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973907    5713 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:38:36.989055 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973910    5713 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:38:36.989055 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973912    5713 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:38:36.989055 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973914    5713 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:38:36.989055 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973916    5713 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:38:36.989055 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973927    5713 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:38:36.989055 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973930    5713 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:38:36.989055 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973932    5713 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:38:36.989055 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973935    5713 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:38:36.989055 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973937    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:38:36.989055 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973940    5713 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:38:36.989055 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973942    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:38:36.989055 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973945    5713 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:38:36.989055 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973947    5713 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:38:36.990059 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973950    5713 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:38:36.990059 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973952    5713 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:38:36.990059 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973955    5713 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:38:36.990059 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973959    5713 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:38:36.990059 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973968    5713 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:38:36.990059 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973971    5713 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:38:36.990059 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.973974    5713 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:38:36.990059 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.980117    5713 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 20:38:36.990059 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.980137    5713 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 20:38:36.990059 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.980158    5713 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 20:38:36.990059 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.980163    5713 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 20:38:36.990059 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980229    5713 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:38:36.990059 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980235    5713 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:38:36.990059 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980238    5713 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:38:36.990059 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980241    5713 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:38:36.990059 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980244    5713 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:38:36.990059 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980247    5713 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:38:36.990059 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980249    5713 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:38:36.990059 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980252    5713 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:38:36.990832 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980255    5713 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:38:36.990832 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980257    5713 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:38:36.990832 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980260    5713 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:38:36.990832 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980263    5713 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:38:36.990832 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980265    5713 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:38:36.990832 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980268    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:38:36.990832 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980271    5713 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:38:36.990832 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980273    5713 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:38:36.990832 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980276    5713 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:38:36.990832 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980278    5713 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:38:36.990832 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980281    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:38:36.990832 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980284    5713 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:38:36.990832 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980286    5713 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:38:36.990832 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980289    5713 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:38:36.990832 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980291    5713 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:38:36.990832 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980294    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:38:36.990832 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980297    5713 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:38:36.990832 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980299    5713 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:38:36.990832 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980302    5713 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:38:36.991309 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980314    5713 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:38:36.991309 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980317    5713 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:38:36.991309 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980320    5713 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:38:36.991309 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980323    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:38:36.991309 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980326    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:38:36.991309 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980329    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:38:36.991309 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980331    5713 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:38:36.991309 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980334    5713 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:38:36.991309 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980337    5713 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:38:36.991309 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980339    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:38:36.991309 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980342    5713 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:38:36.991309 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980344    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:38:36.991309 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980347    5713 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:38:36.991309 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980350    5713 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:38:36.991309 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980352    5713 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:38:36.991309 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980355    5713 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:38:36.991309 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980357    5713 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:38:36.991309 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980360    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:38:36.991309 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980362    5713 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:38:36.991776 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980371    5713 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:38:36.991776 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980374    5713 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:38:36.991776 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980376    5713 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:38:36.991776 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980379    5713 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:38:36.991776 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980381    5713 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:38:36.991776 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980384    5713 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:38:36.991776 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980386    5713 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:38:36.991776 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980389    5713 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:38:36.991776 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980391    5713 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:38:36.991776 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980395    5713 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:38:36.991776 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980399    5713 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:38:36.991776 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980402    5713 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:38:36.991776 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980405    5713 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:38:36.991776 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980408    5713 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:38:36.991776 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980411    5713 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:38:36.991776 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980414    5713 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:38:36.991776 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980416    5713 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:38:36.991776 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980419    5713 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:38:36.991776 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980421    5713 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:38:36.992398 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980424    5713 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:38:36.992398 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980427    5713 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:38:36.992398 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980429    5713 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:38:36.992398 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980432    5713 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:38:36.992398 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980434    5713 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:38:36.992398 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980437    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:38:36.992398 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980439    5713 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:38:36.992398 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980442    5713 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:38:36.992398 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980444    5713 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:38:36.992398 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980447    5713 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:38:36.992398 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980450    5713 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:38:36.992398 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980455    5713 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:38:36.992398 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980458    5713 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:38:36.992398 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980460    5713 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:38:36.992398 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980469    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:38:36.992398 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980472    5713 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:38:36.992398 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980474    5713 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:38:36.992398 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980477    5713 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:38:36.992398 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980480    5713 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:38:36.992883 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980482    5713 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:38:36.992883 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980485    5713 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:38:36.992883 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980488    5713 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:38:36.992883 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980490    5713 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:38:36.992883 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980493    5713 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:38:36.992883 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980496    5713 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:38:36.992883 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980499    5713 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:38:36.992883 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980501    5713 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:38:36.992883 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980504    5713 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:38:36.992883 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980506    5713 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:38:36.992883 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980508    5713 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:38:36.992883 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980511    5713 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:38:36.992883 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980513    5713 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:38:36.992883 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980515    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:38:36.992883 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980518    5713 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:38:36.992883 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980520    5713 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:38:36.992883 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980523    5713 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:38:36.992883 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980525    5713 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:38:36.992883 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980528    5713 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:38:36.992883 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980530    5713 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:38:36.993422 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980533    5713 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:38:36.993422 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980535    5713 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:38:36.993422 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.980569    5713 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 20:38:36.993422 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980725    5713 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:38:36.993422 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980732    5713 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:38:36.993422 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980735    5713 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:38:36.993422 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980738    5713 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:38:36.993422 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980741    5713 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:38:36.993422 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980744    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:38:36.993422 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980746    5713 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:38:36.993422 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980749    5713 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:38:36.993422 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980752    5713 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:38:36.993422 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980754    5713 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:38:36.993422 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980757    5713 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:38:36.993422 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980759    5713 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:38:36.993422 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980762    5713 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:38:36.993422 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980764    5713 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:38:36.993858 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980767    5713 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:38:36.993858 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980771    5713 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:38:36.993858 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980773    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:38:36.993858 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980776    5713 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:38:36.993858 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980778    5713 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:38:36.993858 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980781    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:38:36.993858 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980783    5713 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:38:36.993858 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980806    5713 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:38:36.993858 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980809    5713 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:38:36.993858 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980812    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:38:36.993858 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980814    5713 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:38:36.993858 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980817    5713 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:38:36.993858 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980820    5713 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:38:36.993858 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980823    5713 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:38:36.993858 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980825    5713 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:38:36.993858 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980828    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:38:36.993858 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980830    5713 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:38:36.993858 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980833    5713 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:38:36.993858 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980835    5713 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:38:36.993858 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980838    5713 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:38:36.994343 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980860    5713 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:38:36.994343 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980865    5713 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:38:36.994343 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980868    5713 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:38:36.994343 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980871    5713 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:38:36.994343 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980875    5713 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:38:36.994343 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980877    5713 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:38:36.994343 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980880    5713 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:38:36.994343 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980882    5713 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:38:36.994343 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980885    5713 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:38:36.994343 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980888    5713 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:38:36.994343 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980890    5713 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:38:36.994343 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980892    5713 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:38:36.994343 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980895    5713 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:38:36.994343 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980897    5713 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:38:36.994343 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980900    5713 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:38:36.994343 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980902    5713 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:38:36.994343 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980905    5713 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:38:36.994343 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980907    5713 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:38:36.994343 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980910    5713 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:38:36.994343 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980912    5713 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:38:36.994847 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980915    5713 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:38:36.994847 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980917    5713 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:38:36.994847 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980920    5713 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:38:36.994847 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980922    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:38:36.994847 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980925    5713 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:38:36.994847 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980927    5713 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:38:36.994847 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980930    5713 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:38:36.994847 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980933    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:38:36.994847 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980936    5713 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:38:36.994847 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980938    5713 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:38:36.994847 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980941    5713 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:38:36.994847 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980943    5713 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:38:36.994847 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980946    5713 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:38:36.994847 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980948    5713 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:38:36.994847 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980951    5713 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:38:36.994847 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980953    5713 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:38:36.994847 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980956    5713 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:38:36.994847 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980959    5713 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:38:36.994847 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980962    5713 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:38:36.995304 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980965    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:38:36.995304 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980967    5713 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:38:36.995304 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980970    5713 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:38:36.995304 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980972    5713 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:38:36.995304 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980975    5713 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:38:36.995304 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980978    5713 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:38:36.995304 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980981    5713 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:38:36.995304 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980984    5713 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:38:36.995304 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980988    5713 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:38:36.995304 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980993    5713 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:38:36.995304 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980996    5713 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:38:36.995304 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.980998    5713 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:38:36.995304 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981000    5713 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:38:36.995304 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981003    5713 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:38:36.995304 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981006    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:38:36.995304 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981008    5713 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:38:36.995304 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981011    5713 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:38:36.995304 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981014    5713 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:38:36.995733 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981016    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:38:36.995733 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981019    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:38:36.995733 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981021    5713 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:38:36.995733 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981024    5713 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:38:36.995733 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981026    5713 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:38:36.995733 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981029    5713 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:38:36.995733 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981031    5713 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:38:36.995733 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981033    5713 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:38:36.995733 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981036    5713 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:38:36.995733 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981040    5713 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:38:36.995733 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981043    5713 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:38:36.995733 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981046    5713 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:38:36.995733 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981049    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:38:36.995733 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981053    5713 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:38:36.995733 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981055    5713 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:38:36.996140 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.981079    5713 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 20:38:36.996140 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981114    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 20:38:36.996140 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981119    5713 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 20:38:36.996140 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981121    5713 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 20:38:36.996140 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981125    5713 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 20:38:36.996140 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981128    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 20:38:36.996140 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981131    5713 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 20:38:36.996140 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981133    5713 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 20:38:36.996140 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981136    5713 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 20:38:36.996140 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981140    5713 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 20:38:36.996140 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981142    5713 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 20:38:36.996140 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981145    5713 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 20:38:36.996140 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981147    5713 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 20:38:36.996140 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981150    5713 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 20:38:36.996140 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981152    5713 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 20:38:36.996140 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981155    5713 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 20:38:36.996539 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981157    5713 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 20:38:36.996539 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981160    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 20:38:36.996539 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981162    5713 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 20:38:36.996539 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981165    5713 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 20:38:36.996539 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981167    5713 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 20:38:36.996539 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981169    5713 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 20:38:36.996539 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981172    5713 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 20:38:36.996539 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981174    5713 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 20:38:36.996539 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981178    5713 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 20:38:36.996539 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981181    5713 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 20:38:36.996539 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981184    5713 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 20:38:36.996539 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981186    5713 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 20:38:36.996539 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981189    5713 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 20:38:36.996539 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981192    5713 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 20:38:36.996539 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981194    5713 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 20:38:36.996539 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981197    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 20:38:36.996539 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981200    5713 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 20:38:36.996539 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981202    5713 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 20:38:36.996539 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981205    5713 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 20:38:36.997092 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981207    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 20:38:36.997092 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981209    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 20:38:36.997092 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981212    5713 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 20:38:36.997092 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981214    5713 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 20:38:36.997092 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981216    5713 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 20:38:36.997092 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981219    5713 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 20:38:36.997092 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981223    5713 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 20:38:36.997092 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981225    5713 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 20:38:36.997092 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981228    5713 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 20:38:36.997092 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981232    5713 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 20:38:36.997092 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981236    5713 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 20:38:36.997092 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981239    5713 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 20:38:36.997092 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981242    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 20:38:36.997092 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981244    5713 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 20:38:36.997092 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981247    5713 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 20:38:36.997092 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981250    5713 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 20:38:36.997092 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981252    5713 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 20:38:36.997092 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981254    5713 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 20:38:36.997092 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981257    5713 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 20:38:36.997610 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981259    5713 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 20:38:36.997610 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981262    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 20:38:36.997610 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981264    5713 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 20:38:36.997610 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981267    5713 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 20:38:36.997610 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981269    5713 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 20:38:36.997610 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981272    5713 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 20:38:36.997610 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981274    5713 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 20:38:36.997610 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981277    5713 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 20:38:36.997610 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981279    5713 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 20:38:36.997610 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981282    5713 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 20:38:36.997610 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981285    5713 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 20:38:36.997610 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981287    5713 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 20:38:36.997610 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981289    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 20:38:36.997610 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981292    5713 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 20:38:36.997610 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981294    5713 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 20:38:36.997610 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981296    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 20:38:36.997610 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981299    5713 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 20:38:36.997610 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981301    5713 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 20:38:36.997610 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981304    5713 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 20:38:36.997610 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981307    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 20:38:36.998157 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981310    5713 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 20:38:36.998157 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981312    5713 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 20:38:36.998157 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981315    5713 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 20:38:36.998157 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981317    5713 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 20:38:36.998157 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981320    5713 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 20:38:36.998157 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981322    5713 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 20:38:36.998157 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981325    5713 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 20:38:36.998157 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981327    5713 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 20:38:36.998157 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981330    5713 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 20:38:36.998157 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981332    5713 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 20:38:36.998157 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981334    5713 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 20:38:36.998157 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981337    5713 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 20:38:36.998157 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981339    5713 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 20:38:36.998157 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981342    5713 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 20:38:36.998157 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981345    5713 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 20:38:36.998157 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981347    5713 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 20:38:36.998157 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981353    5713 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 20:38:36.998157 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981355    5713 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 20:38:36.998157 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981358    5713 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 20:38:36.998157 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981361    5713 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 20:38:36.998661 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981363    5713 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 20:38:36.998661 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981365    5713 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 20:38:36.998661 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981368    5713 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 20:38:36.998661 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981371    5713 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 20:38:36.998661 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981373    5713 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 20:38:36.998661 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981376    5713 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 20:38:36.998661 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981378    5713 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 20:38:36.998661 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981380    5713 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 20:38:36.998661 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981383    5713 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 20:38:36.998661 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981385    5713 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 20:38:36.998661 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981387    5713 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 20:38:36.998661 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981391    5713 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 20:38:36.998661 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:38:36.981393    5713 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 20:38:36.998661 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.981452    5713 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 20:38:36.998661 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.982515    5713 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 20:38:36.998661 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.982581    5713 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 20:38:36.998661 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.982756    5713 server.go:1014] "Starting client certificate rotation"
Oct 30 20:38:36.998661 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.982893    5713 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 20:38:36.999248 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.982945    5713 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 20:32:25 +0000 UTC" deadline="2028-03-29 17:18:57.683170354 +0000 UTC"
Oct 30 20:38:36.999248 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.982974    5713 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12380h40m20.700198631s"
Oct 30 20:38:36.999248 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.983267    5713 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 20:38:36.999248 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.984397    5713 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 20:38:36.999248 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.985735    5713 log.go:25] "Validated CRI v1 runtime API"
Oct 30 20:38:36.999248 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.987762    5713 log.go:25] "Validated CRI v1 image API"
Oct 30 20:38:36.999248 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.988632    5713 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 20:38:36.999248 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.990894    5713 fs.go:135] Filesystem UUIDs: map[402345bb-b82b-4c5c-b016-5c2986310405:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d0a89e01-143a-4f81-b78e-46622c485f15:/dev/nvme0n1p3]
Oct 30 20:38:36.999606 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.990918    5713 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0216b8abddf058a5e8c11ab3d4247a9bc0861a991792b0e9c5185b0457313f5a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0216b8abddf058a5e8c11ab3d4247a9bc0861a991792b0e9c5185b0457313f5a/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0556d7ef4363eda2c99088bf88103c5f529286eff610cf928bd5fd364d59661f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0556d7ef4363eda2c99088bf88103c5f529286eff610cf928bd5fd364d59661f/userdata/shm major:0 minor:403 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0fd6ba6bbeee6529a916683a80a42d504623480e41b88ec7ba4b3c949def3b23/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0fd6ba6bbeee6529a916683a80a42d504623480e41b88ec7ba4b3c949def3b23/userdata/shm major:0 minor:331 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1ea7e0ed662156243f8ea7572a5da1f21a253dc7697c46faf8bfc5712d171b76/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1ea7e0ed662156243f8ea7572a5da1f21a253dc7697c46faf8bfc5712d171b76/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/207c2a8b2761f4dd57850a3ae4352f753b6f6f0011eebdfe6b3c757cabdbe25c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/207c2a8b2761f4dd57850a3ae4352f753b6f6f0011eebdfe6b3c757cabdbe25c/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/37919f769bfa6e93adece45c43c99fdd08bf244037b64270406b3d0b42f8541d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/37919f769bfa6e93adece45c43c99fdd08bf244037b64270406b3d0b42f8541d/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4025cb9daa803f5e569efffc0b3dfd1521edd91538cf9a349efa38966ae598bd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4025cb9daa803f5e569efffc0b3dfd1521edd91538cf9a349efa38966ae598bd/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4e21ab7264b551a8ab03a4c48dbf1cd0249e6358baf993bf958430e2be9d8ae5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4e21ab7264b551a8ab03a4c48dbf1cd0249e6358baf993bf958430e2be9d8ae5/userdata/shm major:0 minor:406 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/50613bde83fde8f4cbc68785d926585e85b6115110d8e3b0e9ec5dbd554007d3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/50613bde83fde8f4cbc68785d926585e85b6115110d8e3b0e9ec5dbd554007d3/userdata/shm major:0 minor:402 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/52d55cc2ca4f2a532260dd37c06837c41cff4ecde4c800d447ba3d1f90fed26f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/52d55cc2ca4f2a532260dd37c06837c41cff4ecde4c800d447ba3d1f90fed26f/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5c04573be329bbdedea4c49d44b5542bbdca873033dfe32c353d5d813401a9f3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5c04573be329bbdedea4c49d44b5542bbdca873033dfe32c353d5d813401a9f3/userdata/shm major:0 minor:263 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/72b8abeeb9582f7dfef26e301ff0fd3b5f4a2c94b647dbbbac2bf7695ae45efa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/72b8abeeb9582f7dfef26e301ff0fd3b5f4a2c94b647dbbbac2bf7695ae45efa/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7ff1c0abadbf0f85df09a48cb328c77e7e85b622024858da9d8c89ef20a49824/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7ff1c0abadbf0f85df09a48cb328c77e7e85b622024858da9d8c89ef20a49824/userdata/shm major:0 minor:297 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9bb2d18fcfaf283a4fcc45f58c023f04ae3298c1663aa836bc1e8145054137f4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9bb2d18fcfaf283a4fcc45f58c023f04ae3298c1663aa836bc1e8145054137f4/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bb61b79694f7acf56a2fe0d9befede9258908bca9bccf0c60271880a71e5820e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bb61b79694f7acf56a2fe0d9befede9258908bca9bccf0c60271880a71e5820e/userdata/shm major:0 minor:261 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cd28db981bd07b2f76f443c22c22f841e728ee45b8044be93908ec23d1c18f9b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cd28db981bd07b2f76f443c22c22f841e728ee45b8044be93908ec23d1c18f9b/userdata/shm major:0 minor:256 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dd1344076d2e9fdfd6fb1d182509921d27b82e16709f2a47f113c478e72c814b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dd1344076d2e9fdfd6fb1d182509921d27b82e16709f2a47f113c478e72c814b/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e1c3c4f48a7972af5d5cab42a2780626c2e8a8e1fbfb446f19b0c7ca3ce86303/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e1c3c4f48a7972af5d5cab42a2780626c2e8a8e1fbfb446f19b0c7ca3ce86303/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ec74ba9089bb74e797ab04ab632e021ded2af0dfdb192caf8dcd748f4721cf55/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ec74ba9089bb74e797ab04ab632e021ded2af0dfdb192caf8dcd748f4721cf55/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ed7caa3657ef2870726c29db54cb3e6618b46d7b01cd55ccd643f888811697de/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ed7caa3657ef2870726c29db54cb3e6618b46d7b01cd55ccd643f888811697de/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ef111d4318a24f1b18f799bb083297e58132a20086bb28aedf2679e9b3dc1913/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ef111d4318a24f1b18f799bb083297e58132a20086bb28aedf2679e9b3dc1913/userdata/shm major:0 minor:353 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fb474b3b1951057e4d840268aa15a635b39e4d8e8512b25b8d781e34fec82255/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fb474b3b1951057e4d840268aa15a635b39e4d8e8512b25b8d781e34fec82255/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/08ddd17c-0134-45dc-8a2f-aca6e3668e9f/volumes/kubernetes.io~projected/kube-api-access-sl5v9:{mountpoint:/var/lib/kubelet/pods/08ddd17c-0134-45dc-8a2f-aca6e3668e9f/volumes/kubernetes.io~projected/kube-api-access-sl5v9 major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0fb51194-9a91-4430-8ded-4844ffb6727a/volumes/kubernetes.io~projected/kube-api-access-ftcfw:{mountpoint:/var/lib/kubelet/pods/0fb51194-9a91-4430-8ded-4844ffb6727a/volumes/kubernetes.io~projected/kube-api-access-ftcfw major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1004eede-9d90-42b0-bd11-42a32056f2ae/volumes/kubernetes.io~projected/kube-api-access-tgsh5:{mountpoint:/var/lib/kubelet/pods/1004eede-9d90-42b0-bd11-42a32056f2ae/volumes/kubernetes.io~projected/kube-api-access-tgsh5 major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1004eede-9d90-42b0-bd11-42a32056f2ae/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/1004eede-9d90-42b0-bd11-42a32056f2ae/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:327 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1004eede-9d90-42b0-bd11-42a32056f2ae/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/1004eede-9d90-42b0-bd11-42a32056f2ae/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1004eede-9d90-42b0-bd11-42a32056f2ae/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/1004eede-9d90-42b0-bd11-42a32056f2ae/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a74da60-0284-410b-9d69-0d5192ce0928/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/1a74da60-0284-410b-9d69-0d5192ce0928/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:400 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db/volume-subpaths/web-config/alertmanager/9 major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db/volume-subpaths/web-config/config-reloader/6 major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db/volumes/kubernetes.io~empty-dir/config-out major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db/volumes/kubernetes.io~projected/kube-api-access-8xq44:{mountpoint:/var/lib/kubelet/pods/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db/volumes/kubernetes.io~projected/kube-api-access-8xq44 major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db/volumes/kubernetes.io~projected/tls-assets major:0 minor:292 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db/volumes/kubernetes.io~secret/config-volume major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db/volumes/kubernetes.io~secret/web-config major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/242a67ca-f9ea-431d-adee-ece061468a02/volumes/kubernetes.io~projected/kube-api-access-k4m4z:{mountpoint:/var/lib/kubelet/pods/242a67ca-f9ea-431d-adee-ece061468a02/volumes/kubernetes.io~projected/kube-api-access-k4m4z major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/242a67ca-f9ea-431d-adee-ece061468a02/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/242a67ca-f9ea-431d-adee-ece061468a02/volumes/kubernetes.io~secret/metrics-certs major:0 minor:396 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/520d950c-2ec5-457f-b55f-51bd29207757/volumes/kubernetes.io~projected/kube-api-access-xgzz6:{mountpoint:/var/lib/kubelet/pods/520d950c-2ec5-457f-b55f-51bd29207757/volumes/kubernetes.io~projected/kube-api-access-xgzz6 major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/520d950c-2ec5-457f-b55f-51bd29207757/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/520d950c-2ec5-457f-b55f-51bd29207757/volumes/kubernetes.io~secret/cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/53aaf696-7925-43b6-86fd-1b536b3095d3/volumes/kubernetes.io~projected/kube-api-access-kpbnb:{mountpoint:/var/lib/kubelet/pods/53aaf696-7925-43b6-86fd-1b536b3095d3/volumes/kubernetes.io~projected/kube-api-access-kpbnb major:0 minor:260 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/53aaf696-7925-43b6-86fd-1b536b3095d3/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/53aaf696-7925-43b6-86fd-1b536b3095d3/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:259 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/53aaf696-7925-43b6-86fd-1b536b3095d3/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/53aaf696-7925-43b6-86fd-1b536b3095d3/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:258 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5b900a91-15bc-4019-99f5-4619cea03518/volumes/kubernetes.io~projected/kube-api-access-4jdjz:{mountpoint:/var/lib/kubelet/pods/5b900a91-15bc-4019-99f5-4619cea03518/volumes/kubernetes.io~projected/kube-api-access-4jdjz major:0 minor:255 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5b900a91-15bc-4019-99f5-4619cea03518/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/5b900a91-15bc-4019-99f5-4619cea03518/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:253 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5b900a91-15bc-4019-99f5-4619cea03518/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/5b900a91-15bc-4019-99f5-4619cea03518/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:254 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6b559cf8-14bb-4eaf-9bea-47a956284d20/volumes/kubernetes.io~projected/kube-api-access-m476q:{mountpoint:/var/lib/kubelet/pods/6b559cf8-14bb-4eaf-9bea-47a956284d20/volumes/kubernetes.io~projected/kube-api-access-m476q major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6b559cf8-14bb-4eaf-9bea-47a956284d20/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/6b559cf8-14bb-4eaf-9bea-47a956284d20/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/764461d5-e0b2-4856-a97e-0c5d3d34a497/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/764461d5-e0b2-4856-a97e-0c5d3d34a497/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/764461d5-e0b2-4856-a97e-0c5d3d34a497/volumes/kubernetes.io~projected/kube-api-access-nl4qd:{mountpoint:/var/lib/kubelet/pods/764461d5-e0b2-4856-a97e-0c5d3d34a497/volumes/kubernetes.io~projected/kube-api-access-nl4qd major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/764461d5-e0b2-4856-a97e-0c5d3d34a497/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/764461d5-e0b2-4856-a97e-0c5d3d34a497/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7787041b-34dc-4567-8989-22bfd95456e0/volumes/kubernetes.io~projected/kube-api-access-bvnz6:{mountpoint:/var/lib/kubelet/pods/7787041b-34dc-4567-8989-22bfd95456e0/volumes/kubernetes.io~projected/kube-api-access-bvnz6 major:0 minor:252 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7787041b-34dc-4567-8989-22bfd95456e0/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/7787041b-34dc-4567-8989-22bfd95456e0/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:244 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7787041b-34dc-4567-8989-22bfd95456e0/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/7787041b-34dc-4567-8989-22bfd95456e0/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:243 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/88a8166b-aa0c-41dc-8bf3-92708c60df57/volumes/kubernetes.io~projected/kube-api-access-z96wz:{mountpoint:/var/lib/kubelet/pods/88a8166b-aa0c-41dc-8bf3-92708c60df57/volumes/kubernetes.io~projected/kube-api-access-z96wz major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a7b1237b-e022-4570-9c48-4a8449996c69/volumes/kubernetes.io~projected/kube-api-access-xkjkh:{mountpoint:/var/lib/kubelet/pods/a7b1237b-e022-4570-9c48-4a8449996c69/volumes/kubernetes.io~projected/kube-api-access-xkjkh major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85/volumes/kubernetes.io~projected/kube-api-access-frzdn:{mountpoint:/var/lib/kubelet/pods/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85/volumes/kubernetes.io~projected/kube-api-access-frzdn major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b25a9500-23b1-42d8-8a77-ca75dabc6ef5/volumes/kubernetes.io~projected/kube-api-access-9cjbx:{mountpoint:/var/lib/kubelet/pods/b25a9500-23b1-42d8-8a77-ca75dabc6ef5/volumes/kubernetes.io~projected/kube-api-access-9cjbx major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b25a9500-23b1-42d8-8a77-ca75dabc6ef5/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/b25a9500-23b1-42d8-8a77-ca75dabc6ef5/volumes/kubernetes.io~secret/metrics-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b7f60156-3771-486a-a9d2-7acd5ab9f475/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/b7f60156-3771-486a-a9d2-7acd5ab9f475/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b7f60156-3771-486a-a9d2-7acd5ab9f475/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/b7f60156-3771-486a-a9d2-7acd5ab9f475/volumes/kubernetes.io~empty-dir/tmp major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b7f60156-3771-486a-a9d2-7acd5ab9f475/volumes/kubernetes.io~projected/kube-api-access-k6qnn:{mountpoint:/var/lib/kubelet/pods/b7f60156-3771-486a-a9d2-7acd5ab9f475/volumes/kubernetes.io~projected/kube-api-access-k6qnn major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df/volume-subpaths/web-config/config-reloader/2 major:0 minor:342 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:342 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df/volume-subpaths/web-config/prometheus/15 major:0 minor:342 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df/volumes/kubernetes.io~empty-dir/config-out major:0 minor:343 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df/volumes/kubernetes.io~projected/kube-api-access-rx4tv:{mountpoint:/var/lib/kubelet/pods/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df/volumes/kubernetes.io~projected/kube-api-access-rx4tv major:0 minor:349 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df/volumes/kubernetes.io~projected/tls-assets major:0 minor:350 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df/volumes/kubernetes.io~secret/config major:0 minor:344 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:345 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:347 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:348 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:352 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:346 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:337 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:351 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df/volumes/kubernetes.io~secret/web-config major:0 minor:342 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c3704770-6fb6-4086-8a22-9b29468d5a70/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/c3704770-6fb6-4086-8a22-9b29468d5a70/volumes/kubernetes.io~secret/agent-certs major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c7a31d20-2bf3-4c38-8816-2181571a1fda/volumes/kubernetes.io~projected/kube-api-access-jqm5k:{mountpoint:/var/lib/kubelet/pods/c7a31d20-2bf3-4c38-8816-2181571a1fda/volumes/kubernetes.io~projected/kube-api-access-jqm5k major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e54d0918-84e8-40f3-9b18-6e2fafb3d500/volumes/kubernetes.io~projected/kube-api-access-bzhsg:{mountpoint:/var/lib/kubelet/pods/e54d0918-84e8-40f3-9b18-6e2fafb3d500/volumes/kubernetes.io~projected/kube-api-access-bzhsg major:0 minor:401 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/e1af0494d7e528aa561b909e58ad0c1c8fce3a2c7ad9ce00e975005a6c7612b5/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/81bf9cd31b9826420761c7d376933e0ec55493b7de25ca20881a70d636043d5f/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/d128d8844a684e2d162ec9cafaa0b04e9e7b6b4ed6b6b45efc4837ac189e6520/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/1ec9dd8bf078e989b1307976ada9bc59a4372347a5095de1304b47f195e93ee3/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-108:{mountpoint:/var/lib/containers/storage/overlay/f794525eb922bdf98406b3955f0714c6fd14db46ff6a252e5e9ffcc36c047817/merged major:0 minor:108 fsType:overlay blockSize:0} overlay_0-116:{mountpoint:/var/lib/containers/storage/overlay/c5b6333bb1c943b54f5d19d3e224986ad60a29415cc713a193f955a5003aa0eb/merged major:0 minor:116 fsType:overlay blockSize:0} overlay_0-121:{mountpoint:/var/lib/containers/storage/overlay/d438e5af48ac15152105f947916f1c37ee18c3942ff8ce96edc9900e111e06b5/merged major:0 minor:121 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/76c14009d5f8c34033563872a203f0a16d01cdbcfcff66733dd08e9dc65b6fce/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/45b7e08cab16690cb24370ce93d568f5596a697e84f518476af4f16255657bc8/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/b87db1a2e83b4f64c5b7a0a506a79a9c457f4c6be8498b2fc713ea70d9cf4cff/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/c03be0ae3ed7f0ba92fce302567a6c21ae4e555fabb270bd2990ce443670a22d/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/47a186b42c41661f8f120b16e1e0258e98e650f5cef4440b02b405d2ca066b54/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/591efd69a1c7f28fb063a8d7ef58aae2d4aaf79a6e6051e6fd320da87ad8e6a8/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/41d66a3fed8333fa0a6932efde49db2dab58f088861f3e0c5c1933d7c79eec4c/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/5e6dee5907065ef57360eeb82f0611073a7965320ede12830c1200613490c045/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/fb8937ffc506b2bc411cfaa56a5b980cfed0d22e431dade7587f8ff74ec8d37a/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/1f57241e8e80cd4722f77b3d67638a3accd5d627e2620fb3a215cc2f2238f8f5/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/a92df3c491223089c41914dc239eadf41b223df38871a9b7604458359cc6c137/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/97d8a5ed8b1ee8ee778ee5e1d01d491ff33880dc039f368afe5009a7e7b35b61/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/18c5473e40468a0e618a6e28f8b2363df5462ed700c79d86e0e9c8e80d252516/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/ce5cb12f7436bfc049b4ed06a037ba3536b74d90bb927b3a8c65347abfb67e05/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/ae1868bd1755b35ddfe34913d8cb8895fc78fa56674f161f74e5ccfc462a4096/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/5e7b200157978d6f98d06c033732db5d6a6ad689593d16ccf6187ac52f7a28c2/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/cbc36d4febf12004f34f91e87d3b81317a114e00647b92ed5fa4d2c031f4f60e/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/c03340e97ce755c29087b9eaad995299f2de96ea7ddcd61533db6655a89248d9/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-218:{mountpoint:/var/lib/containers/storage/overlay/95fd02366144dda0bcd2ac1b8389f737f13d94c68c0a69fdaa3665b2a46a18bb/merged major:0 minor:218 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/911c890e18f8ef65ae0edfc169bccbc232d430fb5f0329254127ed305e6f3c5e/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/3c55ecc65a08f948399b1845d39d12539065612ba6edd85409cfe4af58c23862/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/4cf43f64b5bf409848e90063a60149d0615b3880e823ae22718e52cc93229da1/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/4558c1e515a14cc760e7fea17139b9be9f8072a4a8ede1578b7181b26d637735/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/f3dcd3a4c9c40f5c39dfcc84f9783c54f2b864afbae671dea8291caebfdaaf5c/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/01c5bc55d5c949e14a3aebb52a46770a83604ae59562b69a7542bfa686d81eb8/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/0e177ecbc0c2862b9a67af8c03f24f73127051895c1e2e3a5f224203a678e898/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/1876abdfa64d7715be45f806afd1bcb6b5b6cdd5bce24261b4aac3b8953ac7ab/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/4af256a726a7c85cb507f204794f12b9ea1b7ff13cc0febd1ff30b836baff7b2/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-279:{mountpoint:/var/lib/containers/storage/overlay/03bfa9eaebedeba7df62681bc0ab5aeb8cbb86afac9850a33505357847f32f0f/merged major:0 minor:279 fsType:overlay blockSize:0} overlay_0-299:{mountpoint:/var/lib/containers/storage/overlay/e1e7ab157982c61e852523b1cd629762f1cdc342657b6a4deda01c0307792287/merged major:0 minor:299 fsType:overlay blockSize:0} overlay_0-301:{mountpoint:/var/lib/containers/storage/overlay/02ac921330c55af6570cf42cae91ca94e3e457754a1bdf3c003f01bba42a4022/merged major:0 minor:301 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/39c66d847a22a34ce3428c548d29713121cb984ff73dadfa5f1489c4891e3e15/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/7478806ca83092e7c68f99fe9018bdf6ccc67c3adf98b9eba44c052c5ac9ad14/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-314:{mountpoint:/var/lib/containers/storage/overlay/126c7e2937e20e83c86dc741b9f5cdd953e14d0c63ca59fa52f639fbdafdce65/merged major:0 minor:314 fsType:overlay blockSize:0} overlay_0-318:{mountpoint:/var/lib/containers/storage/overlay/f22e8716eaaa87313cd1d7268c06999002d694fd28bb186f429c7b7941a4e98b/merged major:0 minor:318 fsType:overlay blockSize:0} overlay_0-333:{mountpoint:/var/lib/containers/storage/overlay/9f4837104f34b58d17796c1bed15efc23b10e7ac83a130c69812bf6f621182ee/merged major:0 minor:333 fsType:overlay blockSize:0} overlay_0-335:{mountpoint:/var/lib/containers/storage/overlay/97b15522b67ef1717ebf60133f9f9ce67582680c7030660d42e4f233cf0b0ccd/merged major:0 minor:335 fsType:overlay blockSize:0} overlay_0-355:{mountpoint:/var/lib/containers/storage/overlay/015f95be3c1fa484b3ecc02efc2d26458c195a3336e5a8c2ec18b9bf08470ea5/merged major:0 minor:355 fsType:overlay blockSize:0} overlay_0-357:{mountpoint:/var/lib/containers/storage/overlay/bea2f282fb05357713d402785ec67861a8fe0d54c3321be8fdcd062a244481cb/merged major:0 minor:357 fsType:overlay blockSize:0} overlay_0-359:{mountpoint:/var/lib/containers/storage/overlay/365a83a2a4a3e7342ad45406a93fb1b1038979d5019f8388d0dbf4dea840e2da/merged major:0 minor:359 fsType:overlay blockSize:0} overlay_0-365:{mountpoint:/var/lib/containers/storage/overlay/da0beba6cc0df63ff377da32ba4ed6e68febebd9f24e8299243ff58577e8c7b2/merged major:0 minor:365 fsType:overlay blockSize:0} overlay_0-370:{mountpoint:/var/lib/containers/storage/overlay/b4f5c07979118bc8fa2c8b5199fe2dda5af2170f662e982423806f6d3087e8e3/merged major:0 minor:370 fsType:overlay blockSize:0} overlay_0-375:{mountpoint:/var/lib/containers/storage/overlay/5f4a1d60e3121776ce7cb62aa0b1dbc131961c1171ea722dc1d38b9626cfa160/merged major:0 minor:375 fsType:overlay blockSize:0} overlay_0-380:{mountpoint:/var/lib/containers/storage/overlay/feb09b8c50d008a859a74076991b1011ec10e13debeeb73330cb1082a91f6cae/merged major:0 minor:380 fsType:overlay blockSize:0} overlay_0-385:{mountpoint:/var/lib/containers/storage/overlay/7c2a475ec33e5907cd83ac9eada32f264f74c634414c59f8f308daa2bbcb18c3/merged major:0 minor:385 fsType:overlay blockSize:0} overlay_0-390:{mountpoint:/var/lib/containers/storage/overlay/b0f1702855b241d324eb9e615ea5050304379b93e7797f2ccbfb5d40ba66f0bd/merged major:0 minor:390 fsType:overlay blockSize:0} overlay_0-408:{mountpoint:/var/lib/containers/storage/overlay/bef3684cfb9687f74f92db8f2e9f6d2d6f732bfdb1c5c44cd56e9f97c9676b81/merged major:0 minor:408 fsType:overlay blockSize:0} overlay_0-410:{mountpoint:/var/lib/containers/storage/overlay/c9b2a8addb98d98c7435c5a2868d4bd5a5f2cac55211e132d46cddfda4c559ee/merged major:0 minor:410 fsType:overlay blockSize:0} overlay_0-412:{mountpoint:/var/lib/containers/storage/overlay/19745e94d02d45a697b6c38854828d2dc6e1222e1b99abcdb1a32912e30b6ef1/merged major:0 minor:412 fsType:overlay blockSize:0} overlay_0-414:{mountpoint:/var/lib/containers/storage/overlay/9a9f9ad0c892f0b52227b23e903ede8dace298a5e858ad0957d25303cae55fed/merged major:0 minor:414 fsType:overlay blockSize:0} overlay_0-416:{mountpoint:/var/lib/containers/storage/overlay/fa2950d677282ad1bdc630a7fea891d97196b5ccfc39bab7a9f0f692a28d3fc2/merged major:0 minor:416 fsType:overlay blockSize:0} overlay_0-421:{mountpoint:/var/lib/containers/storage/overlay/0132b0c1b482617f9808e1bc0a37f3b461303bc836afd003011ddea0052e8703/merged major:0 minor:421 fsType:overlay blockSize:0} overlay_0-426:{mountpoint:/var/lib/containers/storage/overlay/8a8db4a46c91750c33011ff069b90406ee59e44cc58f2ee7e67f9b21bc18ea4b/merged major:0 minor:426 fsType:overlay blockSize:0} overlay_0-431:{mountpoint:/var/lib/containers/storage/overlay/6f32981500e2f925e87d2d6ca3fe7699d447ac6fdad4bb4f633eb6d0886f442d/merged major:0 minor:431 fsType:overlay blockSize:0} overlay_0-436:{mountpoint:/var/lib/containers/storage/overlay/18fdc719db106176844411c835e21a52bf6aac6f4333db14f4ad9d11f31af4e4/merged major:0 minor:436 fsType:overlay blockSize:0} overlay_0-442:{mountpoint:/var/lib/containers/storage/overlay/1edf62b96c7b414a27559b46c47e1e8aacf01e5f47b62178730ffb66470d153b/merged major:0 minor:442 fsType:overlay blockSize:0} overlay_0-447:{mountpoint:/var/lib/containers/storage/overlay/53e5c6b568655c331475aa1a21baef06483562648b3eeef55a2b4e5c2a80eac9/merged major:0 minor:447 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/5ae9aaaf5b8239cc9f1b4316dce7e0ab00aa4ac74aa4b7d84be123e5be596535/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/ec211aa23f5faa05056404f9904dc52732db64ed513b0f00d89484570c42fb45/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/900b9103ab46cfffc7bc21660aaf34c94b834c27e2158bb180cd6a82d2bb7652/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/75045bad6e6a6a43a9d319b63d5807b0c19c1d99259167f02fafdee52b464266/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/6cbd1ace9b5e9715717618edaa7f314087ea05ec624800e6f713420b78223ec6/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/24d4ce0be56fcf4e3b42b429396550f2359edf2230ee8941f747501a028d7d05/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/774a8deda6eb33ecebbecab85d68a9495407b14d7d670903fda3b313ed78b01d/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/c151aef7f09264cfff5aae48994d495c1aa71c53915f2fff69bbae67b68b84e3/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/0032352a577e901894541dce762713b9de776ad9ebb5fabd41f2ace6de13d59e/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/7fe955451e080e84aadba9d74b704f14e246803128ed3108063e5c5212321620/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 20:38:37.000017 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:36.998446    5713 manager.go:217] Machine: {Timestamp:2026-10-30 20:38:36.99794891 +0000 UTC m=+0.062725299 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3029543 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2060fc5e4a2a4e0051100d8a2bad43 SystemUUID:ec2060fc-5e4a-2a4e-0051-100d8a2bad43 BootID:84fae1cf-a661-4325-bd4a-15278dc62013 Filesystems:[{Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1004eede-9d90-42b0-bd11-42a32056f2ae/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:328 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:343 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-442 DeviceMajor:0 DeviceMinor:442 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85/volumes/kubernetes.io~projected/kube-api-access-frzdn DeviceMajor:0 DeviceMinor:58 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-121 DeviceMajor:0 DeviceMinor:121 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/764461d5-e0b2-4856-a97e-0c5d3d34a497/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0556d7ef4363eda2c99088bf88103c5f529286eff610cf928bd5fd364d59661f/userdata/shm DeviceMajor:0 DeviceMinor:403 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7787041b-34dc-4567-8989-22bfd95456e0/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:243 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:284 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-333 DeviceMajor:0 DeviceMinor:333 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:337 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:342 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-410 DeviceMajor:0 DeviceMinor:410 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/37919f769bfa6e93adece45c43c99fdd08bf244037b64270406b3d0b42f8541d/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/52d55cc2ca4f2a532260dd37c06837c41cff4ecde4c800d447ba3d1f90fed26f/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6b559cf8-14bb-4eaf-9bea-47a956284d20/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:45 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5b900a91-15bc-4019-99f5-4619cea03518/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:253 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cd28db981bd07b2f76f443c22c22f841e728ee45b8044be93908ec23d1c18f9b/userdata/shm DeviceMajor:0 DeviceMinor:256 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1004eede-9d90-42b0-bd11-42a32056f2ae/volumes/kubernetes.io~projected/kube-api-access-tgsh5 DeviceMajor:0 DeviceMinor:330 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ef111d4318a24f1b18f799bb083297e58132a20086bb28aedf2679e9b3dc1913/userdata/shm DeviceMajor:0 DeviceMinor:353 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5b900a91-15bc-4019-99f5-4619cea03518/volumes/kubernetes.io~projected/kube-api-access-4jdjz DeviceMajor:0 DeviceMinor:255 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:285 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-357 DeviceMajor:0 DeviceMinor:357 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-370 DeviceMajor:0 DeviceMinor:370 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-385 DeviceMajor:0 DeviceMinor:385 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4e21ab7264b551a8ab03a4c48dbf1cd0249e6358baf993bf958430e2be9d8ae5/userdata/shm DeviceMajor:0 DeviceMinor:406 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-416 DeviceMajor:0 DeviceMinor:416 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1ea7e0ed662156243f8ea7572a5da1f21a253dc7697c46faf8bfc5712d171b76/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:347 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/88a8166b-aa0c-41dc-8bf3-92708c60df57/volumes/kubernetes.io~projected/kube-api-access-z96wz DeviceMajor:0 DeviceMinor:51 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-116 DeviceMajor:0 DeviceMinor:116 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:290 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-375 DeviceMajor:0 DeviceMinor:375 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/242a67ca-f9ea-431d-adee-ece061468a02/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:396 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:342 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-414 DeviceMajor:0 DeviceMinor:414 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-447 DeviceMajor:0 DeviceMinor:447 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:290 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/08ddd17c-0134-45dc-8a2f-aca6e3668e9f/volumes/kubernetes.io~projected/kube-api-access-sl5v9 DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/764461d5-e0b2-4856-a97e-0c5d3d34a497/volumes/kubernetes.io~projected/kube-api-access-nl4qd DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6b559cf8-14bb-4eaf-9bea-47a956284d20/volumes/kubernetes.io~projected/kube-api-access-m476q DeviceMajor:0 DeviceMinor:202 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:287 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:293 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-318 DeviceMajor:0 DeviceMinor:318 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/53aaf696-7925-43b6-86fd-1b536b3095d3/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:259 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-380 DeviceMajor:0 DeviceMinor:380 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-301 DeviceMajor:0 DeviceMinor:301 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df/volumes/kubernetes.io~projected/kube-api-access-rx4tv DeviceMajor:0 DeviceMinor:349 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-431 DeviceMajor:0 DeviceMinor:431 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/520d950c-2ec5-457f-b55f-51bd29207757/volumes/kubernetes.io~projected/kube-api-access-xgzz6 DeviceMajor:0 DeviceMinor:203 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/53aaf696-7925-43b6-86fd-1b536b3095d3/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:258 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-335 DeviceMajor:0 DeviceMinor:335 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-218 DeviceMajor:0 DeviceMinor:218 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:344 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9bb2d18fcfaf283a4fcc45f58c023f04ae3298c1663aa836bc1e8145054137f4/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0fb51194-9a91-4430-8ded-4844ffb6727a/volumes/kubernetes.io~projected/kube-api-access-ftcfw DeviceMajor:0 DeviceMinor:52 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ed7caa3657ef2870726c29db54cb3e6618b46d7b01cd55ccd643f888811697de/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-279 DeviceMajor:0 DeviceMinor:279 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4025cb9daa803f5e569efffc0b3dfd1521edd91538cf9a349efa38966ae598bd/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b25a9500-23b1-42d8-8a77-ca75dabc6ef5/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:197 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/520d950c-2ec5-457f-b55f-51bd29207757/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:201 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7787041b-34dc-4567-8989-22bfd95456e0/volumes/kubernetes.io~projected/kube-api-access-bvnz6 DeviceMajor:0 DeviceMinor:252 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-426 DeviceMajor:0 DeviceMinor:426 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/b7f60156-3771-486a-a9d2-7acd5ab9f475/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:50 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/242a67ca-f9ea-431d-adee-ece061468a02/volumes/kubernetes.io~projected/kube-api-access-k4m4z DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dd1344076d2e9fdfd6fb1d182509921d27b82e16709f2a47f113c478e72c814b/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1004eede-9d90-42b0-bd11-42a32056f2ae/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:329 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ec74ba9089bb74e797ab04ab632e021ded2af0dfdb192caf8dcd748f4721cf55/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fb474b3b1951057e4d840268aa15a635b39e4d8e8512b25b8d781e34fec82255/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5b900a91-15bc-4019-99f5-4619cea03518/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:254 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7ff1c0abadbf0f85df09a48cb328c77e7e85b622024858da9d8c89ef20a49824/userdata/shm DeviceMajor:0 DeviceMinor:297 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-390 DeviceMajor:0 DeviceMinor:390 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e54d0918-84e8-40f3-9b18-6e2fafb3d500/volumes/kubernetes.io~projected/kube-api-access-bzhsg DeviceMajor:0 DeviceMinor:401 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7787041b-34dc-4567-8989-22bfd95456e0/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:244 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:348 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:350 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-412 DeviceMajor:0 DeviceMinor:412 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b7f60156-3771-486a-a9d2-7acd5ab9f475/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:49 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:285 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/72b8abeeb9582f7dfef26e301ff0fd3b5f4a2c94b647dbbbac2bf7695ae45efa/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/207c2a8b2761f4dd57850a3ae4352f753b6f6f0011eebdfe6b3c757cabdbe25c/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b25a9500-23b1-42d8-8a77-ca75dabc6ef5/volumes/kubernetes.io~projected/kube-api-access-9cjbx DeviceMajor:0 DeviceMinor:204 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:288 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-108 DeviceMajor:0 DeviceMinor:108 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0216b8abddf058a5e8c11ab3d4247a9bc0861a991792b0e9c5185b0457313f5a/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:289 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:290 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-365 DeviceMajor:0 DeviceMinor:365 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-408 DeviceMajor:0 DeviceMinor:408 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-436 DeviceMajor:0 DeviceMinor:436 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c7a31d20-2bf3-4c38-8816-2181571a1fda/volumes/kubernetes.io~projected/kube-api-access-jqm5k DeviceMajor:0 DeviceMinor:53 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:345 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:352 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c3704770-6fb6-4086-8a22-9b29468d5a70/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:47 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:291 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:342 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1a74da60-0284-410b-9d69-0d5192ce0928/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:400 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e1c3c4f48a7972af5d5cab42a2780626c2e8a8e1fbfb446f19b0c7ca3ce86303/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bb61b79694f7acf56a2fe0d9befede9258908bca9bccf0c60271880a71e5820e/userdata/shm DeviceMajor:0 DeviceMinor:261 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db/volumes/kubernetes.io~projected/kube-api-access-8xq44 DeviceMajor:0 DeviceMinor:286 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-299 DeviceMajor:0 DeviceMinor:299 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-421 DeviceMajor:0 DeviceMinor:421 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/764461d5-e0b2-4856-a97e-0c5d3d34a497/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:48 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b7f60156-3771-486a-a9d2-7acd5ab9f475/volumes/kubernetes.io~projected/kube-api-access-k6qnn DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/53aaf696-7925-43b6-86fd-1b536b3095d3/volumes/kubernetes.io~projected/kube-api-access-kpbnb DeviceMajor:0 DeviceMinor:260 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:292 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-314 DeviceMajor:0 DeviceMinor:314 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1004eede-9d90-42b0-bd11-42a32056f2ae/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:327 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:351 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-355 DeviceMajor:0 DeviceMinor:355 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a7b1237b-e022-4570-9c48-4a8449996c69/volumes/kubernetes.io~projected/kube-api-access-xkjkh DeviceMajor:0 DeviceMinor:54 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5c04573be329bbdedea4c49d44b5542bbdca873033dfe32c353d5d813401a9f3/userdata/shm DeviceMajor:0 DeviceMinor:263 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:290 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:346 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:342 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/50613bde83fde8f4cbc68785d926585e85b6115110d8e3b0e9ec5dbd554007d3/userdata/shm DeviceMajor:0 DeviceMinor:402 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0fd6ba6bbeee6529a916683a80a42d504623480e41b88ec7ba4b3c949def3b23/userdata/shm DeviceMajor:0 DeviceMinor:331 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-359 DeviceMajor:0 DeviceMinor:359 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0216b8abddf058a MacAddress:f6:9c:ac:d1:eb:35 Speed:10000 Mtu:8901} {Name:0556d7ef4363eda MacAddress:a6:29:d8:3b:08:9e Speed:10000 Mtu:8901} {Name:0fd6ba6bbeee652 MacAddress:76:7e:fc:d1:1c:33 Speed:10000 Mtu:8901} {Name:4e21ab7264b551a MacAddress:da:2f:38:81:11:e1 Speed:10000 Mtu:8901} {Name:50613bde83fde8f MacAddress:a6:18:e6:3f:cb:d1 Speed:10000 Mtu:8901} {Name:7ff1c0abadbf0f8 MacAddress:52:88:70:4a:5a:30 Speed:10000 Mtu:8901} {Name:bb61b79694f7acf MacAddress:fe:68:ea:4d:cf:0a Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:3f:87:41:db:bf Speed:0 Mtu:9001} {Name:br-int MacAddress:7e:47:10:77:e3:e8 Speed:0 Mtu:8901} {Name:cd28db981bd07b2 MacAddress:e2:64:f7:39:dc:a0 Speed:10000 Mtu:8901} {Name:dd1344076d2e9fd MacAddress:d6:0b:f1:86:83:ac Speed:10000 Mtu:8901} {Name:ef111d4318a24f1 MacAddress:fa:d9:0b:7c:a5:ff Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:3f:87:41:db:bf Speed:0 Mtu:9001} {Name:fb474b3b1951057 MacAddress:b2:cf:11:86:8f:a4 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:0e:a6:49:42:b3:43 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:f6:c3:91:ce:73:f5 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 20:38:37.000698 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.000532    5713 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 20:38:37.000773 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.000757    5713 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 20:38:37.001145 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.001120    5713 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 20:38:37.001283 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.001147    5713 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-165.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 20:38:37.001331 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.001285    5713 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 20:38:37.001331 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.001294    5713 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 20:38:37.001331 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.001304    5713 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 20:38:37.001331 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.001323    5713 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 20:38:37.001469 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.001432    5713 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 20:38:37.001610 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.001600    5713 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 20:38:37.001727 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.001718    5713 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 20:38:37.001765 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.001757    5713 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 20:38:37.001820 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.001774    5713 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 20:38:37.001820 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.001800    5713 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 20:38:37.001820 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.001810    5713 kubelet.go:405] "Adding apiserver pod source"
Oct 30 20:38:37.001820 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.001819    5713 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 20:38:37.002528 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.002511    5713 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 20:38:37.002528 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.002524    5713 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 20:38:37.003115 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.003096    5713 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 20:38:37.006664 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.006646    5713 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 20:38:37.006664 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.006660    5713 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 20:38:37.006806 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.006776    5713 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 20:38:37.007009 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.006997    5713 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 20:38:37.007041 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.007029    5713 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 20:38:37.007135 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.007126    5713 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 20:38:37.007165 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.007141    5713 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 20:38:37.007165 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.007147    5713 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 20:38:37.007165 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.007153    5713 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 20:38:37.007165 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.007159    5713 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 20:38:37.007165 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.007164    5713 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 20:38:37.007306 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.007172    5713 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 20:38:37.007306 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.007180    5713 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 20:38:37.007306 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.007188    5713 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 20:38:37.007306 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.007193    5713 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 20:38:37.007306 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.007203    5713 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 20:38:37.007306 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.007213    5713 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 20:38:37.007306 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.007242    5713 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 20:38:37.007306 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.007251    5713 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 20:38:37.011052 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.011035    5713 server.go:1263] "Started kubelet"
Oct 30 20:38:37.011222 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.011145    5713 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 20:38:37.011305 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.011267    5713 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 20:38:37.011590 ip-10-0-128-165 systemd[1]: Started Kubernetes Kubelet.
Oct 30 20:38:37.019523 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.013102    5713 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 20:38:37.019523 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.013526    5713 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 20:38:37.019523 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.013905    5713 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 20:38:37.019523 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.016751    5713 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 20:38:37.019523 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.016834    5713 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 20:38:37.019523 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.016893    5713 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 20:32:26 +0000 UTC" deadline="2028-08-10 08:12:52.17612302 +0000 UTC"
Oct 30 20:38:37.019523 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.016927    5713 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15587h34m15.159199895s"
Oct 30 20:38:37.019523 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:38:37.016988    5713 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-165.us-west-1.compute.internal\" not found"
Oct 30 20:38:37.019523 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.017052    5713 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 20:38:37.019523 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.017060    5713 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 20:38:37.019523 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.017086    5713 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 20:38:37.019523 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.019189    5713 factory.go:153] Registering CRI-O factory
Oct 30 20:38:37.019523 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.019214    5713 factory.go:223] Registration of the crio container factory successfully
Oct 30 20:38:37.019523 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.019277    5713 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 20:38:37.019523 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.019287    5713 factory.go:55] Registering systemd factory
Oct 30 20:38:37.019523 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.019294    5713 factory.go:223] Registration of the systemd container factory successfully
Oct 30 20:38:37.019523 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.019313    5713 factory.go:103] Registering Raw factory
Oct 30 20:38:37.019523 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.019326    5713 manager.go:1199] Started watching for new ooms in manager
Oct 30 20:38:37.020508 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.019646    5713 manager.go:319] Starting recovery of all containers
Oct 30 20:38:37.020694 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.020655    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7a31d20-2bf3-4c38-8816-2181571a1fda" volumeName="kubernetes.io/projected/c7a31d20-2bf3-4c38-8816-2181571a1fda-kube-api-access-jqm5k" seLinuxMountContext=""
Oct 30 20:38:37.020749 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.020697    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c4609ce-58a7-47f3-8eaa-0f19cea5c6db" volumeName="kubernetes.io/configmap/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-alertmanager-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 20:38:37.020749 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.020713    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b559cf8-14bb-4eaf-9bea-47a956284d20" volumeName="kubernetes.io/empty-dir/6b559cf8-14bb-4eaf-9bea-47a956284d20-data-volume" seLinuxMountContext=""
Oct 30 20:38:37.020749 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.020725    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="764461d5-e0b2-4856-a97e-0c5d3d34a497" volumeName="kubernetes.io/configmap/764461d5-e0b2-4856-a97e-0c5d3d34a497-ovnkube-config" seLinuxMountContext=""
Oct 30 20:38:37.020749 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.020736    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" volumeName="kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-configmap-serving-certs-ca-bundle" seLinuxMountContext=""
Oct 30 20:38:37.020749 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.020748    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" volumeName="kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 30 20:38:37.021020 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.020759    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c4609ce-58a7-47f3-8eaa-0f19cea5c6db" volumeName="kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 30 20:38:37.021020 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.020771    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b900a91-15bc-4019-99f5-4619cea03518" volumeName="kubernetes.io/empty-dir/5b900a91-15bc-4019-99f5-4619cea03518-volume-directive-shadow" seLinuxMountContext=""
Oct 30 20:38:37.021020 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.020807    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" volumeName="kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-secret-metrics-client-certs" seLinuxMountContext=""
Oct 30 20:38:37.021020 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.020820    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08ddd17c-0134-45dc-8a2f-aca6e3668e9f" volumeName="kubernetes.io/projected/08ddd17c-0134-45dc-8a2f-aca6e3668e9f-kube-api-access-sl5v9" seLinuxMountContext=""
Oct 30 20:38:37.021020 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.020831    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="520d950c-2ec5-457f-b55f-51bd29207757" volumeName="kubernetes.io/secret/520d950c-2ec5-457f-b55f-51bd29207757-cert" seLinuxMountContext=""
Oct 30 20:38:37.021020 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.020838    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="764461d5-e0b2-4856-a97e-0c5d3d34a497" volumeName="kubernetes.io/configmap/764461d5-e0b2-4856-a97e-0c5d3d34a497-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 20:38:37.021020 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.020845    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="764461d5-e0b2-4856-a97e-0c5d3d34a497" volumeName="kubernetes.io/secret/764461d5-e0b2-4856-a97e-0c5d3d34a497-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 20:38:37.021020 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.020853    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b7f60156-3771-486a-a9d2-7acd5ab9f475" volumeName="kubernetes.io/projected/b7f60156-3771-486a-a9d2-7acd5ab9f475-kube-api-access-k6qnn" seLinuxMountContext=""
Oct 30 20:38:37.021020 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.020863    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" volumeName="kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-prometheus-k8s-rulefiles-2" seLinuxMountContext=""
Oct 30 20:38:37.021020 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.020871    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" volumeName="kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-prometheus-k8s-rulefiles-1" seLinuxMountContext=""
Oct 30 20:38:37.021020 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.020879    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b559cf8-14bb-4eaf-9bea-47a956284d20" volumeName="kubernetes.io/configmap/6b559cf8-14bb-4eaf-9bea-47a956284d20-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 30 20:38:37.021020 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.020886    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b559cf8-14bb-4eaf-9bea-47a956284d20" volumeName="kubernetes.io/projected/6b559cf8-14bb-4eaf-9bea-47a956284d20-kube-api-access-m476q" seLinuxMountContext=""
Oct 30 20:38:37.021020 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.020894    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c4609ce-58a7-47f3-8eaa-0f19cea5c6db" volumeName="kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext=""
Oct 30 20:38:37.021020 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.020902    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1004eede-9d90-42b0-bd11-42a32056f2ae" volumeName="kubernetes.io/configmap/1004eede-9d90-42b0-bd11-42a32056f2ae-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Oct 30 20:38:37.021020 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.020910    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1004eede-9d90-42b0-bd11-42a32056f2ae" volumeName="kubernetes.io/secret/1004eede-9d90-42b0-bd11-42a32056f2ae-client-ca-bundle" seLinuxMountContext=""
Oct 30 20:38:37.021020 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.020917    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c4609ce-58a7-47f3-8eaa-0f19cea5c6db" volumeName="kubernetes.io/projected/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-tls-assets" seLinuxMountContext=""
Oct 30 20:38:37.021020 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.020925    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c4609ce-58a7-47f3-8eaa-0f19cea5c6db" volumeName="kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext=""
Oct 30 20:38:37.021020 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.020932    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c4609ce-58a7-47f3-8eaa-0f19cea5c6db" volumeName="kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-cluster-tls-config" seLinuxMountContext=""
Oct 30 20:38:37.021020 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.020951    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c4609ce-58a7-47f3-8eaa-0f19cea5c6db" volumeName="kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-config-volume" seLinuxMountContext=""
Oct 30 20:38:37.021020 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.020963    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="53aaf696-7925-43b6-86fd-1b536b3095d3" volumeName="kubernetes.io/configmap/53aaf696-7925-43b6-86fd-1b536b3095d3-metrics-client-ca" seLinuxMountContext=""
Oct 30 20:38:37.021020 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.020973    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08ddd17c-0134-45dc-8a2f-aca6e3668e9f" volumeName="kubernetes.io/configmap/08ddd17c-0134-45dc-8a2f-aca6e3668e9f-cni-binary-copy" seLinuxMountContext=""
Oct 30 20:38:37.021020 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.020982    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c4609ce-58a7-47f3-8eaa-0f19cea5c6db" volumeName="kubernetes.io/projected/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-kube-api-access-8xq44" seLinuxMountContext=""
Oct 30 20:38:37.021020 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.020993    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="53aaf696-7925-43b6-86fd-1b536b3095d3" volumeName="kubernetes.io/configmap/53aaf696-7925-43b6-86fd-1b536b3095d3-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Oct 30 20:38:37.021020 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.021000    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b900a91-15bc-4019-99f5-4619cea03518" volumeName="kubernetes.io/configmap/5b900a91-15bc-4019-99f5-4619cea03518-metrics-client-ca" seLinuxMountContext=""
Oct 30 20:38:37.021020 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.021008    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88a8166b-aa0c-41dc-8bf3-92708c60df57" volumeName="kubernetes.io/configmap/88a8166b-aa0c-41dc-8bf3-92708c60df57-serviceca" seLinuxMountContext=""
Oct 30 20:38:37.021020 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.021024    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88a8166b-aa0c-41dc-8bf3-92708c60df57" volumeName="kubernetes.io/projected/88a8166b-aa0c-41dc-8bf3-92708c60df57-kube-api-access-z96wz" seLinuxMountContext=""
Oct 30 20:38:37.021020 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.021034    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a74da60-0284-410b-9d69-0d5192ce0928" volumeName="kubernetes.io/secret/1a74da60-0284-410b-9d69-0d5192ce0928-original-pull-secret" seLinuxMountContext=""
Oct 30 20:38:37.022291 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.021042    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b900a91-15bc-4019-99f5-4619cea03518" volumeName="kubernetes.io/secret/5b900a91-15bc-4019-99f5-4619cea03518-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 20:38:37.022291 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.021052    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7787041b-34dc-4567-8989-22bfd95456e0" volumeName="kubernetes.io/projected/7787041b-34dc-4567-8989-22bfd95456e0-kube-api-access-bvnz6" seLinuxMountContext=""
Oct 30 20:38:37.022291 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.021059    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7b1237b-e022-4570-9c48-4a8449996c69" volumeName="kubernetes.io/projected/a7b1237b-e022-4570-9c48-4a8449996c69-kube-api-access-xkjkh" seLinuxMountContext=""
Oct 30 20:38:37.022291 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.021067    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" volumeName="kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-secret-kube-rbac-proxy" seLinuxMountContext=""
Oct 30 20:38:37.022291 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.021085    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c3704770-6fb6-4086-8a22-9b29468d5a70" volumeName="kubernetes.io/configmap/c3704770-6fb6-4086-8a22-9b29468d5a70-konnectivity-ca" seLinuxMountContext=""
Oct 30 20:38:37.022291 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.021097    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e54d0918-84e8-40f3-9b18-6e2fafb3d500" volumeName="kubernetes.io/projected/e54d0918-84e8-40f3-9b18-6e2fafb3d500-kube-api-access-bzhsg" seLinuxMountContext=""
Oct 30 20:38:37.022291 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.021105    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1004eede-9d90-42b0-bd11-42a32056f2ae" volumeName="kubernetes.io/configmap/1004eede-9d90-42b0-bd11-42a32056f2ae-metrics-server-audit-profiles" seLinuxMountContext=""
Oct 30 20:38:37.022291 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.021112    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1004eede-9d90-42b0-bd11-42a32056f2ae" volumeName="kubernetes.io/secret/1004eede-9d90-42b0-bd11-42a32056f2ae-secret-metrics-server-client-certs" seLinuxMountContext=""
Oct 30 20:38:37.022291 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.021120    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af5cc97c-3f88-40e2-b6e8-e5f7b279bf85" volumeName="kubernetes.io/projected/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-kube-api-access-frzdn" seLinuxMountContext=""
Oct 30 20:38:37.022291 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.021128    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b25a9500-23b1-42d8-8a77-ca75dabc6ef5" volumeName="kubernetes.io/configmap/b25a9500-23b1-42d8-8a77-ca75dabc6ef5-config-volume" seLinuxMountContext=""
Oct 30 20:38:37.022291 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.021135    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b25a9500-23b1-42d8-8a77-ca75dabc6ef5" volumeName="kubernetes.io/projected/b25a9500-23b1-42d8-8a77-ca75dabc6ef5-kube-api-access-9cjbx" seLinuxMountContext=""
Oct 30 20:38:37.022291 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.021143    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" volumeName="kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-prometheus-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 20:38:37.022291 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.021151    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" volumeName="kubernetes.io/empty-dir/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-prometheus-k8s-db" seLinuxMountContext=""
Oct 30 20:38:37.022291 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.021158    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c4609ce-58a7-47f3-8eaa-0f19cea5c6db" volumeName="kubernetes.io/empty-dir/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-config-out" seLinuxMountContext=""
Oct 30 20:38:37.022291 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.021168    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="53aaf696-7925-43b6-86fd-1b536b3095d3" volumeName="kubernetes.io/empty-dir/53aaf696-7925-43b6-86fd-1b536b3095d3-node-exporter-textfile" seLinuxMountContext=""
Oct 30 20:38:37.022291 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.021179    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b900a91-15bc-4019-99f5-4619cea03518" volumeName="kubernetes.io/configmap/5b900a91-15bc-4019-99f5-4619cea03518-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext=""
Oct 30 20:38:37.022291 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.021189    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="764461d5-e0b2-4856-a97e-0c5d3d34a497" volumeName="kubernetes.io/projected/764461d5-e0b2-4856-a97e-0c5d3d34a497-kube-api-access-nl4qd" seLinuxMountContext=""
Oct 30 20:38:37.022291 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.021197    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af5cc97c-3f88-40e2-b6e8-e5f7b279bf85" volumeName="kubernetes.io/configmap/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-cni-binary-copy" seLinuxMountContext=""
Oct 30 20:38:37.022291 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.021204    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08ddd17c-0134-45dc-8a2f-aca6e3668e9f" volumeName="kubernetes.io/configmap/08ddd17c-0134-45dc-8a2f-aca6e3668e9f-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 20:38:37.022291 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.021212    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1004eede-9d90-42b0-bd11-42a32056f2ae" volumeName="kubernetes.io/secret/1004eede-9d90-42b0-bd11-42a32056f2ae-secret-metrics-server-tls" seLinuxMountContext=""
Oct 30 20:38:37.022291 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.021219    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c4609ce-58a7-47f3-8eaa-0f19cea5c6db" volumeName="kubernetes.io/empty-dir/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-alertmanager-main-db" seLinuxMountContext=""
Oct 30 20:38:37.022291 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.021227    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="764461d5-e0b2-4856-a97e-0c5d3d34a497" volumeName="kubernetes.io/configmap/764461d5-e0b2-4856-a97e-0c5d3d34a497-env-overrides" seLinuxMountContext=""
Oct 30 20:38:37.022291 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.021235    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b7f60156-3771-486a-a9d2-7acd5ab9f475" volumeName="kubernetes.io/empty-dir/b7f60156-3771-486a-a9d2-7acd5ab9f475-etc-tuned" seLinuxMountContext=""
Oct 30 20:38:37.022291 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.021248    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" volumeName="kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-configmap-metrics-client-ca" seLinuxMountContext=""
Oct 30 20:38:37.022291 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.021256    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" volumeName="kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-secret-grpc-tls" seLinuxMountContext=""
Oct 30 20:38:37.022291 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.021263    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c3704770-6fb6-4086-8a22-9b29468d5a70" volumeName="kubernetes.io/secret/c3704770-6fb6-4086-8a22-9b29468d5a70-agent-certs" seLinuxMountContext=""
Oct 30 20:38:37.022291 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.021270    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08ddd17c-0134-45dc-8a2f-aca6e3668e9f" volumeName="kubernetes.io/configmap/08ddd17c-0134-45dc-8a2f-aca6e3668e9f-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 20:38:37.022291 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.021278    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c4609ce-58a7-47f3-8eaa-0f19cea5c6db" volumeName="kubernetes.io/configmap/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-metrics-client-ca" seLinuxMountContext=""
Oct 30 20:38:37.022291 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.021285    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="242a67ca-f9ea-431d-adee-ece061468a02" volumeName="kubernetes.io/projected/242a67ca-f9ea-431d-adee-ece061468a02-kube-api-access-k4m4z" seLinuxMountContext=""
Oct 30 20:38:37.022291 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.021294    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="242a67ca-f9ea-431d-adee-ece061468a02" volumeName="kubernetes.io/secret/242a67ca-f9ea-431d-adee-ece061468a02-metrics-certs" seLinuxMountContext=""
Oct 30 20:38:37.022291 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.021302    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b559cf8-14bb-4eaf-9bea-47a956284d20" volumeName="kubernetes.io/secret/6b559cf8-14bb-4eaf-9bea-47a956284d20-insights-runtime-extractor-tls" seLinuxMountContext=""
Oct 30 20:38:37.022291 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.021309    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7787041b-34dc-4567-8989-22bfd95456e0" volumeName="kubernetes.io/configmap/7787041b-34dc-4567-8989-22bfd95456e0-metrics-client-ca" seLinuxMountContext=""
Oct 30 20:38:37.022291 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.021317    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" volumeName="kubernetes.io/empty-dir/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-config-out" seLinuxMountContext=""
Oct 30 20:38:37.022291 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.021324    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="53aaf696-7925-43b6-86fd-1b536b3095d3" volumeName="kubernetes.io/secret/53aaf696-7925-43b6-86fd-1b536b3095d3-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 20:38:37.022291 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.021331    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7a31d20-2bf3-4c38-8816-2181571a1fda" volumeName="kubernetes.io/empty-dir/c7a31d20-2bf3-4c38-8816-2181571a1fda-tmp-dir" seLinuxMountContext=""
Oct 30 20:38:37.022291 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.021340    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7787041b-34dc-4567-8989-22bfd95456e0" volumeName="kubernetes.io/secret/7787041b-34dc-4567-8989-22bfd95456e0-openshift-state-metrics-tls" seLinuxMountContext=""
Oct 30 20:38:37.022291 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.021349    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="53aaf696-7925-43b6-86fd-1b536b3095d3" volumeName="kubernetes.io/projected/53aaf696-7925-43b6-86fd-1b536b3095d3-kube-api-access-kpbnb" seLinuxMountContext=""
Oct 30 20:38:37.022291 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.021356    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b900a91-15bc-4019-99f5-4619cea03518" volumeName="kubernetes.io/projected/5b900a91-15bc-4019-99f5-4619cea03518-kube-api-access-4jdjz" seLinuxMountContext=""
Oct 30 20:38:37.022291 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.021365    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" volumeName="kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Oct 30 20:38:37.022291 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.021375    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7787041b-34dc-4567-8989-22bfd95456e0" volumeName="kubernetes.io/secret/7787041b-34dc-4567-8989-22bfd95456e0-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 20:38:37.022291 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.021383    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1004eede-9d90-42b0-bd11-42a32056f2ae" volumeName="kubernetes.io/projected/1004eede-9d90-42b0-bd11-42a32056f2ae-kube-api-access-tgsh5" seLinuxMountContext=""
Oct 30 20:38:37.022291 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.021391    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b900a91-15bc-4019-99f5-4619cea03518" volumeName="kubernetes.io/secret/5b900a91-15bc-4019-99f5-4619cea03518-kube-state-metrics-tls" seLinuxMountContext=""
Oct 30 20:38:37.022291 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.021399    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b25a9500-23b1-42d8-8a77-ca75dabc6ef5" volumeName="kubernetes.io/secret/b25a9500-23b1-42d8-8a77-ca75dabc6ef5-metrics-tls" seLinuxMountContext=""
Oct 30 20:38:37.022291 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.021406    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" volumeName="kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-prometheus-k8s-rulefiles-0" seLinuxMountContext=""
Oct 30 20:38:37.022291 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.021414    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" volumeName="kubernetes.io/projected/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-kube-api-access-rx4tv" seLinuxMountContext=""
Oct 30 20:38:37.022291 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.021421    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" volumeName="kubernetes.io/projected/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-tls-assets" seLinuxMountContext=""
Oct 30 20:38:37.022291 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.021430    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" volumeName="kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-web-config" seLinuxMountContext=""
Oct 30 20:38:37.022291 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.021438    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fb51194-9a91-4430-8ded-4844ffb6727a" volumeName="kubernetes.io/configmap/0fb51194-9a91-4430-8ded-4844ffb6727a-iptables-alerter-script" seLinuxMountContext=""
Oct 30 20:38:37.022291 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.021452    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c4609ce-58a7-47f3-8eaa-0f19cea5c6db" volumeName="kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-web-config" seLinuxMountContext=""
Oct 30 20:38:37.022291 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.021460    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="520d950c-2ec5-457f-b55f-51bd29207757" volumeName="kubernetes.io/projected/520d950c-2ec5-457f-b55f-51bd29207757-kube-api-access-xgzz6" seLinuxMountContext=""
Oct 30 20:38:37.022291 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.021471    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b25a9500-23b1-42d8-8a77-ca75dabc6ef5" volumeName="kubernetes.io/empty-dir/b25a9500-23b1-42d8-8a77-ca75dabc6ef5-tmp-dir" seLinuxMountContext=""
Oct 30 20:38:37.022291 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.021478    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b7f60156-3771-486a-a9d2-7acd5ab9f475" volumeName="kubernetes.io/empty-dir/b7f60156-3771-486a-a9d2-7acd5ab9f475-tmp" seLinuxMountContext=""
Oct 30 20:38:37.022291 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.021485    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" volumeName="kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-secret-prometheus-k8s-tls" seLinuxMountContext=""
Oct 30 20:38:37.022291 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.021493    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" volumeName="kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-config" seLinuxMountContext=""
Oct 30 20:38:37.022291 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.021501    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" volumeName="kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-thanos-prometheus-http-client-file" seLinuxMountContext=""
Oct 30 20:38:37.022291 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.021508    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1004eede-9d90-42b0-bd11-42a32056f2ae" volumeName="kubernetes.io/empty-dir/1004eede-9d90-42b0-bd11-42a32056f2ae-audit-log" seLinuxMountContext=""
Oct 30 20:38:37.022291 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.021516    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="53aaf696-7925-43b6-86fd-1b536b3095d3" volumeName="kubernetes.io/secret/53aaf696-7925-43b6-86fd-1b536b3095d3-node-exporter-tls" seLinuxMountContext=""
Oct 30 20:38:37.022291 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.021524    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fb51194-9a91-4430-8ded-4844ffb6727a" volumeName="kubernetes.io/projected/0fb51194-9a91-4430-8ded-4844ffb6727a-kube-api-access-ftcfw" seLinuxMountContext=""
Oct 30 20:38:37.022291 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.021532    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c4609ce-58a7-47f3-8eaa-0f19cea5c6db" volumeName="kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-secret-alertmanager-main-tls" seLinuxMountContext=""
Oct 30 20:38:37.022291 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.021541    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af5cc97c-3f88-40e2-b6e8-e5f7b279bf85" volumeName="kubernetes.io/configmap/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-multus-daemon-config" seLinuxMountContext=""
Oct 30 20:38:37.022291 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.021551    5713 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" volumeName="kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext=""
Oct 30 20:38:37.022291 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.021562    5713 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 20:38:37.022291 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.021568    5713 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 20:38:37.053379 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.053344    5713 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 20:38:37.053610 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.053585    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r7lpf_764461d5-e0b2-4856-a97e-0c5d3d34a497/ovn-acl-logging/0.log"
Oct 30 20:38:37.054867 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.054828    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r7lpf_764461d5-e0b2-4856-a97e-0c5d3d34a497/ovn-acl-logging/0.log"
Oct 30 20:38:37.055107 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.055062    5713 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 20:38:37.055107 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.055091    5713 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 20:38:37.055237 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.055148    5713 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 20:38:37.055237 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:38:37.055191    5713 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 20:38:37.071828 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.071770    5713 generic.go:356] "Generic (PLEG): container finished" podID="1c4609ce-58a7-47f3-8eaa-0f19cea5c6db" containerID="d6b9d3c29eb082eb14cbc0dd4664369c791f40d5a75aeeb98689299537f59f75" exitCode=0
Oct 30 20:38:37.080916 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.080879    5713 generic.go:356] "Generic (PLEG): container finished" podID="ccc9dd125588c9f50e9b9a79ea47623e" containerID="12de31f9ba42bfe2aff6d81b3a604a70b24a80ff65ccf3865da3e8cf27ee1868" exitCode=0
Oct 30 20:38:37.086619 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.086594    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r7lpf_764461d5-e0b2-4856-a97e-0c5d3d34a497/ovn-acl-logging/0.log"
Oct 30 20:38:37.087024 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.087001    5713 generic.go:356] "Generic (PLEG): container finished" podID="764461d5-e0b2-4856-a97e-0c5d3d34a497" containerID="0c20ec4888cc8db97227040556ccefd8f74287d75cdc9a38fea5ff5952c6ecd0" exitCode=1
Oct 30 20:38:37.100615 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.100583    5713 generic.go:356] "Generic (PLEG): container finished" podID="08ddd17c-0134-45dc-8a2f-aca6e3668e9f" containerID="8f9e03eae0151b90ef9428446d71ac4fb85fa96451d5819b2299e6d78003c12e" exitCode=0
Oct 30 20:38:37.100615 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.100614    5713 generic.go:356] "Generic (PLEG): container finished" podID="08ddd17c-0134-45dc-8a2f-aca6e3668e9f" containerID="4600938f9a71636c9ee30f66e240f2c3283c1b5f1bcc92ff5ba9bda4b27c9893" exitCode=0
Oct 30 20:38:37.100847 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.100623    5713 generic.go:356] "Generic (PLEG): container finished" podID="08ddd17c-0134-45dc-8a2f-aca6e3668e9f" containerID="1dae6046ddd9de584cc5207ac196d3f359bbf0c4f9fad84e7f06a0f01b5409fa" exitCode=0
Oct 30 20:38:37.100847 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.100631    5713 generic.go:356] "Generic (PLEG): container finished" podID="08ddd17c-0134-45dc-8a2f-aca6e3668e9f" containerID="01133ca7ce0770488f70d4afcf53a63b728580317ef4b11ae2789b598f459d96" exitCode=0
Oct 30 20:38:37.100847 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.100638    5713 generic.go:356] "Generic (PLEG): container finished" podID="08ddd17c-0134-45dc-8a2f-aca6e3668e9f" containerID="6eb18861f50bc8b91cfd55eb493f1a8721d10bf15b7929bd52f60be904cdb9ff" exitCode=0
Oct 30 20:38:37.100847 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.100645    5713 generic.go:356] "Generic (PLEG): container finished" podID="08ddd17c-0134-45dc-8a2f-aca6e3668e9f" containerID="9bb84dcf2cd17cda09fe673ba6010d41f613352eb4fab6ff7921adff4cce0e58" exitCode=0
Oct 30 20:38:37.108200 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.108167    5713 generic.go:356] "Generic (PLEG): container finished" podID="bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" containerID="521dd12d5bf7bf42e12d8d6d5ac01b10c9993cfd8c508bb4c92dea2fc5092310" exitCode=0
Oct 30 20:38:37.112306 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.112277    5713 generic.go:356] "Generic (PLEG): container finished" podID="53aaf696-7925-43b6-86fd-1b536b3095d3" containerID="0c531114f8d8872186e4de8004fefa552bc40dfb17f210071f9c69ed15238a3c" exitCode=0
Oct 30 20:38:37.117217 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:38:37.117194    5713 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-165.us-west-1.compute.internal\" not found"
Oct 30 20:38:37.128626 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.128564    5713 manager.go:324] Recovery completed
Oct 30 20:38:37.149715 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.149689    5713 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:38:37.151545 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.151526    5713 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-165.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:38:37.151606 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.151560    5713 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-165.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:38:37.151606 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.151571    5713 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-165.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:38:37.154548 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.154530    5713 cpu_manager.go:226] "Starting" policy="none"
Oct 30 20:38:37.154548 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.154545    5713 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 20:38:37.154548 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.154547    5713 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:38:37.154694 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.154565    5713 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 20:38:37.154724 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.154691    5713 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 20:38:37.154724 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.154701    5713 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 20:38:37.154724 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.154718    5713 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 20:38:37.154842 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.154724    5713 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 20:38:37.154842 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.154730    5713 policy_none.go:50] "Start"
Oct 30 20:38:37.154842 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.154739    5713 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 20:38:37.154842 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.154747    5713 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 20:38:37.154958 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.154900    5713 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 20:38:37.154958 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.154907    5713 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 20:38:37.154958 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.154913    5713 policy_none.go:44] "Start"
Oct 30 20:38:37.155336 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:38:37.155320    5713 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 30 20:38:37.155488 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.155474    5713 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-165.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:38:37.155547 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.155501    5713 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-165.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:38:37.155547 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.155511    5713 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-165.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:38:37.159837 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.159805    5713 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 20:38:37.162915 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.162898    5713 manager.go:341] "Starting Device Plugin manager"
Oct 30 20:38:37.163016 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:38:37.162931    5713 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 20:38:37.163016 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.162942    5713 server.go:87] "Starting device plugin registration server"
Oct 30 20:38:37.163272 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.163258    5713 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 20:38:37.163336 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.163273    5713 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 20:38:37.163406 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.163385    5713 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 20:38:37.163549 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.163503    5713 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 20:38:37.163659 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.163557    5713 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 20:38:37.163659 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.163564    5713 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 20:38:37.163756 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.163608    5713 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T20:38:37.163544708Z","UUID":"2a075b8c-5d9f-427a-a55f-d21317c5991c","Handler":null,"Name":"","Endpoint":""}
Oct 30 20:38:37.165294 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:38:37.165270    5713 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 20:38:37.165395 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:38:37.165328    5713 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-165.us-west-1.compute.internal\" not found"
Oct 30 20:38:37.166815 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.166781    5713 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 20:38:37.166930 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.166839    5713 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 20:38:37.211429 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.211397    5713 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:38:37.211429 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.211425    5713 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:38:37.211851 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.211836    5713 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 20:38:37.212240 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.212223    5713 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:38:37.263609 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.263575    5713 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 20:38:37.264659 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.264635    5713 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-165.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 20:38:37.264742 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.264669    5713 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-165.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 20:38:37.264742 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.264684    5713 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-165.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 20:38:37.264844 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.264751    5713 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-165.us-west-1.compute.internal"
Oct 30 20:38:37.311316 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.311288    5713 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 20:38:37.355498 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.355443    5713 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-165.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-165.us-west-1.compute.internal"]
Oct 30 20:38:37.356237 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.356191    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-165.us-west-1.compute.internal" event={"ID":"40ad3a6a5fd37a90a3238925c9fca019","Type":"ContainerStarted","Data":"96140bf7b49715a4a17e2b6b944b2a56830caf2610f8b0baa1a7df79e7a09e17"}
Oct 30 20:38:37.356328 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.356244    5713 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-165.us-west-1.compute.internal"
Oct 30 20:38:37.356328 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.356250    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-165.us-west-1.compute.internal" event={"ID":"40ad3a6a5fd37a90a3238925c9fca019","Type":"ContainerStarted","Data":"1ea7e0ed662156243f8ea7572a5da1f21a253dc7697c46faf8bfc5712d171b76"}
Oct 30 20:38:37.356328 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.356268    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-165.us-west-1.compute.internal" event={"ID":"ccc9dd125588c9f50e9b9a79ea47623e","Type":"ContainerStarted","Data":"f048b02d1812d913c41be3ba7447b2f40aa4b28a9b3cd88efef323c8dfa14d70"}
Oct 30 20:38:37.356328 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.356287    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-165.us-west-1.compute.internal" event={"ID":"ccc9dd125588c9f50e9b9a79ea47623e","Type":"ContainerDied","Data":"12de31f9ba42bfe2aff6d81b3a604a70b24a80ff65ccf3865da3e8cf27ee1868"}
Oct 30 20:38:37.356328 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.356304    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-165.us-west-1.compute.internal" event={"ID":"ccc9dd125588c9f50e9b9a79ea47623e","Type":"ContainerStarted","Data":"72b8abeeb9582f7dfef26e301ff0fd3b5f4a2c94b647dbbbac2bf7695ae45efa"}
Oct 30 20:38:37.356560 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.356498    5713 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-165.us-west-1.compute.internal"
Oct 30 20:38:37.372812 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.372757    5713 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-128-165.us-west-1.compute.internal"
Oct 30 20:38:37.372969 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.372867    5713 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-165.us-west-1.compute.internal"
Oct 30 20:38:37.416968 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.416894    5713 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 20:38:37.416968 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:38:37.416945    5713 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-165.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-165.us-west-1.compute.internal"
Oct 30 20:38:37.417634 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.417608    5713 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 20:38:37.417755 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:38:37.417651    5713 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-165.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-165.us-west-1.compute.internal"
Oct 30 20:38:37.514256 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.514214    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/40ad3a6a5fd37a90a3238925c9fca019-config\") pod \"kube-apiserver-proxy-ip-10-0-128-165.us-west-1.compute.internal\" (UID: \"40ad3a6a5fd37a90a3238925c9fca019\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-165.us-west-1.compute.internal"
Oct 30 20:38:37.514256 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.514248    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ccc9dd125588c9f50e9b9a79ea47623e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-165.us-west-1.compute.internal\" (UID: \"ccc9dd125588c9f50e9b9a79ea47623e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-165.us-west-1.compute.internal"
Oct 30 20:38:37.514435 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.514265    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ccc9dd125588c9f50e9b9a79ea47623e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-165.us-west-1.compute.internal\" (UID: \"ccc9dd125588c9f50e9b9a79ea47623e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-165.us-west-1.compute.internal"
Oct 30 20:38:37.615346 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.615309    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ccc9dd125588c9f50e9b9a79ea47623e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-165.us-west-1.compute.internal\" (UID: \"ccc9dd125588c9f50e9b9a79ea47623e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-165.us-west-1.compute.internal"
Oct 30 20:38:37.615501 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.615346    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ccc9dd125588c9f50e9b9a79ea47623e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-165.us-west-1.compute.internal\" (UID: \"ccc9dd125588c9f50e9b9a79ea47623e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-165.us-west-1.compute.internal"
Oct 30 20:38:37.615501 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.615379    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/40ad3a6a5fd37a90a3238925c9fca019-config\") pod \"kube-apiserver-proxy-ip-10-0-128-165.us-west-1.compute.internal\" (UID: \"40ad3a6a5fd37a90a3238925c9fca019\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-165.us-west-1.compute.internal"
Oct 30 20:38:37.615501 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.615424    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/40ad3a6a5fd37a90a3238925c9fca019-config\") pod \"kube-apiserver-proxy-ip-10-0-128-165.us-west-1.compute.internal\" (UID: \"40ad3a6a5fd37a90a3238925c9fca019\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-165.us-west-1.compute.internal"
Oct 30 20:38:37.615501 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.615425    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ccc9dd125588c9f50e9b9a79ea47623e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-165.us-west-1.compute.internal\" (UID: \"ccc9dd125588c9f50e9b9a79ea47623e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-165.us-west-1.compute.internal"
Oct 30 20:38:37.615501 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:37.615430    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ccc9dd125588c9f50e9b9a79ea47623e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-165.us-west-1.compute.internal\" (UID: \"ccc9dd125588c9f50e9b9a79ea47623e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-165.us-west-1.compute.internal"
Oct 30 20:38:38.002236 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.002198    5713 apiserver.go:52] "Watching apiserver"
Oct 30 20:38:38.105612 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.105577    5713 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 20:38:38.106259 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.106235    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvcbb","openshift-insights/insights-runtime-extractor-8khvs","openshift-monitoring/alertmanager-main-0","openshift-multus/multus-additional-cni-plugins-4gl7z","openshift-multus/network-metrics-daemon-hjssq","openshift-network-diagnostics/network-check-target-kbpjj","kube-system/kube-apiserver-proxy-ip-10-0-128-165.us-west-1.compute.internal","openshift-monitoring/metrics-server-bfd9f4b99-p95mj","openshift-cluster-node-tuning-operator/tuned-brmwl","openshift-dns/dns-default-k58xq","openshift-ingress-canary/ingress-canary-jgvrx","kube-system/global-pull-secret-syncer-xj584","openshift-dns/node-resolver-d69bq","openshift-monitoring/openshift-state-metrics-7bd7489957-c4xbh","openshift-monitoring/prometheus-k8s-0","openshift-network-operator/iptables-alerter-bpgsn","openshift-ovn-kubernetes/ovnkube-node-r7lpf","openshift-image-registry/node-ca-5clg5","openshift-monitoring/node-exporter-lwhq8","openshift-multus/multus-h4rs7","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-165.us-west-1.compute.internal","kube-system/konnectivity-agent-lkxfb","openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfcqd"]
Oct 30 20:38:38.121094 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.121064    5713 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 20:38:38.123168 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.123147    5713 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-165.us-west-1.compute.internal"
Oct 30 20:38:38.123324 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.123310    5713 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-165.us-west-1.compute.internal"
Oct 30 20:38:38.158715 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.158688    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:38:38.158924 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.158909    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 20:38:38.159010 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.158966    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 20:38:38.159169 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.159156    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nvzwj\""
Oct 30 20:38:38.160099 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.160086    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jwfqc\""
Oct 30 20:38:38.160152 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.160117    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-c5chh\""
Oct 30 20:38:38.160152 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.160149    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-86vxf\""
Oct 30 20:38:38.161031 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.161021    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 20:38:38.161087 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.161030    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4lpn7\""
Oct 30 20:38:38.162315 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.162296    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-hjmcj\""
Oct 30 20:38:38.162432 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.162349    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bmpds\""
Oct 30 20:38:38.162432 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.162383    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 20:38:38.162432 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.162409    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-g2pwr\""
Oct 30 20:38:38.162432 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.162431    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 20:38:38.162673 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.162441    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-49f2v\""
Oct 30 20:38:38.162673 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.162485    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-sr85n\""
Oct 30 20:38:38.162673 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.162525    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 20:38:38.162673 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.162539    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-rv75s\""
Oct 30 20:38:38.163162 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.163145    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 20:38:38.163516 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.163500    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 20:38:38.163832 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.163816    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:38:38.163902 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.163825    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 20:38:38.163974 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.163961    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 20:38:38.164023 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.164001    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 20:38:38.164071 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.164036    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 20:38:38.164115 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.164076    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 20:38:38.164115 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.164086    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 20:38:38.164115 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.164112    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 20:38:38.164225 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.164143    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 20:38:38.164225 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.164191    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 20:38:38.164225 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.164203    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 20:38:38.164363 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.164229    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 20:38:38.164363 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.164228    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 20:38:38.164363 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.164282    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 20:38:38.164363 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.164310    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 20:38:38.164363 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.164331    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 20:38:38.164363 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.164355    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 20:38:38.164517 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.164414    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 20:38:38.164517 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.164453    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 20:38:38.207072 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.207045    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 20:38:38.207508 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.207491    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 20:38:38.207626 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.207612    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 20:38:38.208146 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.208130    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 20:38:38.208271 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.208255    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 20:38:38.208327 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.208262    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 20:38:38.208327 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.208289    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 20:38:38.208327 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.208309    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 20:38:38.208471 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.208366    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 20:38:38.208471 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.208397    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 30 20:38:38.208471 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.208413    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 20:38:38.208471 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.208423    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 30 20:38:38.208471 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.208402    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 30 20:38:38.208471 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.208425    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 20:38:38.208471 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.208458    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 20:38:38.208471 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.208473    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 20:38:38.211878 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.211861    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 20:38:38.212503 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.212485    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 20:38:38.214458 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.214440    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 20:38:38.214584 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.214570    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 20:38:38.214625 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.214611    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 20:38:38.216764 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.216750    5713 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 20:38:38.216893 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:38:38.216824    5713 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-165.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-165.us-west-1.compute.internal"
Oct 30 20:38:38.216893 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.216867    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:38.216893 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.216890    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:38.217030 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.216908    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:38.217030 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.216935    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0fb51194-9a91-4430-8ded-4844ffb6727a-iptables-alerter-script\") pod \"iptables-alerter-bpgsn\" (UID: \"0fb51194-9a91-4430-8ded-4844ffb6727a\") " pod="openshift-network-operator/iptables-alerter-bpgsn"
Oct 30 20:38:38.217030 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.216959    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgzz6\" (UniqueName: \"kubernetes.io/projected/520d950c-2ec5-457f-b55f-51bd29207757-kube-api-access-xgzz6\") pod \"ingress-canary-jgvrx\" (UID: \"520d950c-2ec5-457f-b55f-51bd29207757\") " pod="openshift-ingress-canary/ingress-canary-jgvrx"
Oct 30 20:38:38.217030 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.216982    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7787041b-34dc-4567-8989-22bfd95456e0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-c4xbh\" (UID: \"7787041b-34dc-4567-8989-22bfd95456e0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c4xbh"
Oct 30 20:38:38.217030 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.217028    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl5v9\" (UniqueName: \"kubernetes.io/projected/08ddd17c-0134-45dc-8a2f-aca6e3668e9f-kube-api-access-sl5v9\") pod \"multus-additional-cni-plugins-4gl7z\" (UID: \"08ddd17c-0134-45dc-8a2f-aca6e3668e9f\") " pod="openshift-multus/multus-additional-cni-plugins-4gl7z"
Oct 30 20:38:38.217265 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.217055    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-var-lib-openvswitch\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:38:38.217265 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.217084    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-run-ovn\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:38:38.217265 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.217126    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:38.217265 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.217179    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:38.217265 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.217200    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:38.217265 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.217219    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0fb51194-9a91-4430-8ded-4844ffb6727a-iptables-alerter-script\") pod \"iptables-alerter-bpgsn\" (UID: \"0fb51194-9a91-4430-8ded-4844ffb6727a\") " pod="openshift-network-operator/iptables-alerter-bpgsn"
Oct 30 20:38:38.217265 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.217247    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:38.217673 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.217441    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b25a9500-23b1-42d8-8a77-ca75dabc6ef5-tmp-dir\") pod \"dns-default-k58xq\" (UID: \"b25a9500-23b1-42d8-8a77-ca75dabc6ef5\") " pod="openshift-dns/dns-default-k58xq"
Oct 30 20:38:38.217673 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.217479    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cjbx\" (UniqueName: \"kubernetes.io/projected/b25a9500-23b1-42d8-8a77-ca75dabc6ef5-kube-api-access-9cjbx\") pod \"dns-default-k58xq\" (UID: \"b25a9500-23b1-42d8-8a77-ca75dabc6ef5\") " pod="openshift-dns/dns-default-k58xq"
Oct 30 20:38:38.217673 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.217504    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:38.217673 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.217494    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:38.217673 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.217556    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a74da60-0284-410b-9d69-0d5192ce0928-original-pull-secret\") pod \"global-pull-secret-syncer-xj584\" (UID: \"1a74da60-0284-410b-9d69-0d5192ce0928\") " pod="kube-system/global-pull-secret-syncer-xj584"
Oct 30 20:38:38.217673 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.217600    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-host-var-lib-cni-bin\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:38:38.217673 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.217637    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-host-run-ovn-kubernetes\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:38:38.218049 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.217694    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b25a9500-23b1-42d8-8a77-ca75dabc6ef5-tmp-dir\") pod \"dns-default-k58xq\" (UID: \"b25a9500-23b1-42d8-8a77-ca75dabc6ef5\") " pod="openshift-dns/dns-default-k58xq"
Oct 30 20:38:38.218049 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.217740    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/764461d5-e0b2-4856-a97e-0c5d3d34a497-ovn-node-metrics-cert\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:38:38.218049 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.217764    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/53aaf696-7925-43b6-86fd-1b536b3095d3-sys\") pod \"node-exporter-lwhq8\" (UID: \"53aaf696-7925-43b6-86fd-1b536b3095d3\") " pod="openshift-monitoring/node-exporter-lwhq8"
Oct 30 20:38:38.218049 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.217920    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08ddd17c-0134-45dc-8a2f-aca6e3668e9f-system-cni-dir\") pod \"multus-additional-cni-plugins-4gl7z\" (UID: \"08ddd17c-0134-45dc-8a2f-aca6e3668e9f\") " pod="openshift-multus/multus-additional-cni-plugins-4gl7z"
Oct 30 20:38:38.218049 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.217953    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/08ddd17c-0134-45dc-8a2f-aca6e3668e9f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4gl7z\" (UID: \"08ddd17c-0134-45dc-8a2f-aca6e3668e9f\") " pod="openshift-multus/multus-additional-cni-plugins-4gl7z"
Oct 30 20:38:38.218049 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.217974    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a74da60-0284-410b-9d69-0d5192ce0928-original-pull-secret\") pod \"global-pull-secret-syncer-xj584\" (UID: \"1a74da60-0284-410b-9d69-0d5192ce0928\") " pod="kube-system/global-pull-secret-syncer-xj584"
Oct 30 20:38:38.218049 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.217993    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-systemd-units\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:38:38.218049 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.218024    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-host-slash\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:38:38.218049 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.218026    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/764461d5-e0b2-4856-a97e-0c5d3d34a497-ovn-node-metrics-cert\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:38:38.218049 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.218033    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:38.218542 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.218079    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/53aaf696-7925-43b6-86fd-1b536b3095d3-node-exporter-wtmp\") pod \"node-exporter-lwhq8\" (UID: \"53aaf696-7925-43b6-86fd-1b536b3095d3\") " pod="openshift-monitoring/node-exporter-lwhq8"
Oct 30 20:38:38.218542 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.218097    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-host-run-netns\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:38:38.218542 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.218125    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx4tv\" (UniqueName: \"kubernetes.io/projected/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-kube-api-access-rx4tv\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:38.218542 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.218167    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzhsg\" (UniqueName: \"kubernetes.io/projected/e54d0918-84e8-40f3-9b18-6e2fafb3d500-kube-api-access-bzhsg\") pod \"network-check-target-kbpjj\" (UID: \"e54d0918-84e8-40f3-9b18-6e2fafb3d500\") " pod="openshift-network-diagnostics/network-check-target-kbpjj"
Oct 30 20:38:38.218542 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.218193    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/08ddd17c-0134-45dc-8a2f-aca6e3668e9f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4gl7z\" (UID: \"08ddd17c-0134-45dc-8a2f-aca6e3668e9f\") " pod="openshift-multus/multus-additional-cni-plugins-4gl7z"
Oct 30 20:38:38.218542 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.218235    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:38.218542 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.218287    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:38.218542 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.218372    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/08ddd17c-0134-45dc-8a2f-aca6e3668e9f-cni-binary-copy\") pod \"multus-additional-cni-plugins-4gl7z\" (UID: \"08ddd17c-0134-45dc-8a2f-aca6e3668e9f\") " pod="openshift-multus/multus-additional-cni-plugins-4gl7z"
Oct 30 20:38:38.218542 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.218401    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:38.218542 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.218417    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frzdn\" (UniqueName: \"kubernetes.io/projected/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-kube-api-access-frzdn\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:38:38.218542 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.218434    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/88a8166b-aa0c-41dc-8bf3-92708c60df57-host\") pod \"node-ca-5clg5\" (UID: \"88a8166b-aa0c-41dc-8bf3-92708c60df57\") " pod="openshift-image-registry/node-ca-5clg5"
Oct 30 20:38:38.218542 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.218451    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5b900a91-15bc-4019-99f5-4619cea03518-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-cfcqd\" (UID: \"5b900a91-15bc-4019-99f5-4619cea03518\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfcqd"
Oct 30 20:38:38.218542 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.218473    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c3704770-6fb6-4086-8a22-9b29468d5a70-konnectivity-ca\") pod \"konnectivity-agent-lkxfb\" (UID: \"c3704770-6fb6-4086-8a22-9b29468d5a70\") " pod="kube-system/konnectivity-agent-lkxfb"
Oct 30 20:38:38.218542 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.218493    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/764461d5-e0b2-4856-a97e-0c5d3d34a497-env-overrides\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:38:38.219214 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.218560    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-etc-kubernetes\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:38:38.219214 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.218570    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/08ddd17c-0134-45dc-8a2f-aca6e3668e9f-cni-binary-copy\") pod \"multus-additional-cni-plugins-4gl7z\" (UID: \"08ddd17c-0134-45dc-8a2f-aca6e3668e9f\") " pod="openshift-multus/multus-additional-cni-plugins-4gl7z"
Oct 30 20:38:38.219214 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.218565    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:38.219214 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.218601    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/764461d5-e0b2-4856-a97e-0c5d3d34a497-ovnkube-config\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:38:38.219214 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.218667    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a7b1237b-e022-4570-9c48-4a8449996c69-registration-dir\") pod \"aws-ebs-csi-driver-node-nvcbb\" (UID: \"a7b1237b-e022-4570-9c48-4a8449996c69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvcbb"
Oct 30 20:38:38.219214 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.218722    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/764461d5-e0b2-4856-a97e-0c5d3d34a497-env-overrides\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:38:38.219214 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.218724    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5b900a91-15bc-4019-99f5-4619cea03518-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-cfcqd\" (UID: \"5b900a91-15bc-4019-99f5-4619cea03518\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfcqd"
Oct 30 20:38:38.219214 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.218717    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c3704770-6fb6-4086-8a22-9b29468d5a70-konnectivity-ca\") pod \"konnectivity-agent-lkxfb\" (UID: \"c3704770-6fb6-4086-8a22-9b29468d5a70\") " pod="kube-system/konnectivity-agent-lkxfb"
Oct 30 20:38:38.219214 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.218730    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b7f60156-3771-486a-a9d2-7acd5ab9f475-run\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:38:38.219214 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.218774    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-cnibin\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:38:38.219214 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.218821    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5b900a91-15bc-4019-99f5-4619cea03518-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-cfcqd\" (UID: \"5b900a91-15bc-4019-99f5-4619cea03518\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfcqd"
Oct 30 20:38:38.219214 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.218850    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/08ddd17c-0134-45dc-8a2f-aca6e3668e9f-cnibin\") pod \"multus-additional-cni-plugins-4gl7z\" (UID: \"08ddd17c-0134-45dc-8a2f-aca6e3668e9f\") " pod="openshift-multus/multus-additional-cni-plugins-4gl7z"
Oct 30 20:38:38.219214 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.218849    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/764461d5-e0b2-4856-a97e-0c5d3d34a497-ovnkube-config\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:38:38.219214 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.218909    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a7b1237b-e022-4570-9c48-4a8449996c69-sys-fs\") pod \"aws-ebs-csi-driver-node-nvcbb\" (UID: \"a7b1237b-e022-4570-9c48-4a8449996c69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvcbb"
Oct 30 20:38:38.219214 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.218927    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:38.219214 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.218950    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7787041b-34dc-4567-8989-22bfd95456e0-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-c4xbh\" (UID: \"7787041b-34dc-4567-8989-22bfd95456e0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c4xbh"
Oct 30 20:38:38.219214 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.218970    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:38.219214 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.219004    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-web-config\") pod \"alertmanager-main-0\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:38.219214 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.219037    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1004eede-9d90-42b0-bd11-42a32056f2ae-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-bfd9f4b99-p95mj\" (UID: \"1004eede-9d90-42b0-bd11-42a32056f2ae\") " pod="openshift-monitoring/metrics-server-bfd9f4b99-p95mj"
Oct 30 20:38:38.219214 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.219097    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b7f60156-3771-486a-a9d2-7acd5ab9f475-etc-tuned\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:38:38.220232 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.219230    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7787041b-34dc-4567-8989-22bfd95456e0-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-c4xbh\" (UID: \"7787041b-34dc-4567-8989-22bfd95456e0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c4xbh"
Oct 30 20:38:38.220232 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.219272    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b7f60156-3771-486a-a9d2-7acd5ab9f475-etc-tuned\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:38:38.220232 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.219286    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1004eede-9d90-42b0-bd11-42a32056f2ae-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-bfd9f4b99-p95mj\" (UID: \"1004eede-9d90-42b0-bd11-42a32056f2ae\") " pod="openshift-monitoring/metrics-server-bfd9f4b99-p95mj"
Oct 30 20:38:38.220232 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.219306    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-cni-binary-copy\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:38:38.220232 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.219363    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1004eede-9d90-42b0-bd11-42a32056f2ae-secret-metrics-server-client-certs\") pod \"metrics-server-bfd9f4b99-p95mj\" (UID: \"1004eede-9d90-42b0-bd11-42a32056f2ae\") " pod="openshift-monitoring/metrics-server-bfd9f4b99-p95mj"
Oct 30 20:38:38.220232 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.219441    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:38.220232 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.219486    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-cni-binary-copy\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:38:38.220232 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.219499    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b7f60156-3771-486a-a9d2-7acd5ab9f475-host\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:38:38.220232 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.219517    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-host-var-lib-kubelet\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:38:38.220232 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.219533    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-multus-socket-dir-parent\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:38:38.220232 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.219547    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-host-run-netns\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:38:38.220232 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.219571    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-run-openvswitch\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:38:38.220232 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.219599    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6b559cf8-14bb-4eaf-9bea-47a956284d20-crio-socket\") pod \"insights-runtime-extractor-8khvs\" (UID: \"6b559cf8-14bb-4eaf-9bea-47a956284d20\") " pod="openshift-insights/insights-runtime-extractor-8khvs"
Oct 30 20:38:38.220232 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.219633    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqm5k\" (UniqueName: \"kubernetes.io/projected/c7a31d20-2bf3-4c38-8816-2181571a1fda-kube-api-access-jqm5k\") pod \"node-resolver-d69bq\" (UID: \"c7a31d20-2bf3-4c38-8816-2181571a1fda\") " pod="openshift-dns/node-resolver-d69bq"
Oct 30 20:38:38.220232 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.219650    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/08ddd17c-0134-45dc-8a2f-aca6e3668e9f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4gl7z\" (UID: \"08ddd17c-0134-45dc-8a2f-aca6e3668e9f\") " pod="openshift-multus/multus-additional-cni-plugins-4gl7z"
Oct 30 20:38:38.220232 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.219774    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/08ddd17c-0134-45dc-8a2f-aca6e3668e9f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4gl7z\" (UID: \"08ddd17c-0134-45dc-8a2f-aca6e3668e9f\") " pod="openshift-multus/multus-additional-cni-plugins-4gl7z"
Oct 30 20:38:38.220232 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.219839    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-host-cni-bin\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:38:38.220232 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.219866    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a7b1237b-e022-4570-9c48-4a8449996c69-socket-dir\") pod \"aws-ebs-csi-driver-node-nvcbb\" (UID: \"a7b1237b-e022-4570-9c48-4a8449996c69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvcbb"
Oct 30 20:38:38.220232 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.219895    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkjkh\" (UniqueName: \"kubernetes.io/projected/a7b1237b-e022-4570-9c48-4a8449996c69-kube-api-access-xkjkh\") pod \"aws-ebs-csi-driver-node-nvcbb\" (UID: \"a7b1237b-e022-4570-9c48-4a8449996c69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvcbb"
Oct 30 20:38:38.220232 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.219923    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-web-config\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:38.220232 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.220014    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6qnn\" (UniqueName: \"kubernetes.io/projected/b7f60156-3771-486a-a9d2-7acd5ab9f475-kube-api-access-k6qnn\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:38:38.220232 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.220095    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/53aaf696-7925-43b6-86fd-1b536b3095d3-node-exporter-textfile\") pod \"node-exporter-lwhq8\" (UID: \"53aaf696-7925-43b6-86fd-1b536b3095d3\") " pod="openshift-monitoring/node-exporter-lwhq8"
Oct 30 20:38:38.220232 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.220117    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/53aaf696-7925-43b6-86fd-1b536b3095d3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lwhq8\" (UID: \"53aaf696-7925-43b6-86fd-1b536b3095d3\") " pod="openshift-monitoring/node-exporter-lwhq8"
Oct 30 20:38:38.220232 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.220135    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b7f60156-3771-486a-a9d2-7acd5ab9f475-etc-sysconfig\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:38:38.220232 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.220158    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b7f60156-3771-486a-a9d2-7acd5ab9f475-tmp\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:38:38.220232 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.220229    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/53aaf696-7925-43b6-86fd-1b536b3095d3-node-exporter-textfile\") pod \"node-exporter-lwhq8\" (UID: \"53aaf696-7925-43b6-86fd-1b536b3095d3\") " pod="openshift-monitoring/node-exporter-lwhq8"
Oct 30 20:38:38.220232 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.220243    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b7f60156-3771-486a-a9d2-7acd5ab9f475-tmp\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:38:38.221599 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.220260    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/53aaf696-7925-43b6-86fd-1b536b3095d3-metrics-client-ca\") pod \"node-exporter-lwhq8\" (UID: \"53aaf696-7925-43b6-86fd-1b536b3095d3\") " pod="openshift-monitoring/node-exporter-lwhq8"
Oct 30 20:38:38.221599 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.220294    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6b559cf8-14bb-4eaf-9bea-47a956284d20-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8khvs\" (UID: \"6b559cf8-14bb-4eaf-9bea-47a956284d20\") " pod="openshift-insights/insights-runtime-extractor-8khvs"
Oct 30 20:38:38.221599 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.220320    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m476q\" (UniqueName: \"kubernetes.io/projected/6b559cf8-14bb-4eaf-9bea-47a956284d20-kube-api-access-m476q\") pod \"insights-runtime-extractor-8khvs\" (UID: \"6b559cf8-14bb-4eaf-9bea-47a956284d20\") " pod="openshift-insights/insights-runtime-extractor-8khvs"
Oct 30 20:38:38.221599 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.220346    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvnz6\" (UniqueName: \"kubernetes.io/projected/7787041b-34dc-4567-8989-22bfd95456e0-kube-api-access-bvnz6\") pod \"openshift-state-metrics-7bd7489957-c4xbh\" (UID: \"7787041b-34dc-4567-8989-22bfd95456e0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c4xbh"
Oct 30 20:38:38.221599 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.220371    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1a74da60-0284-410b-9d69-0d5192ce0928-dbus\") pod \"global-pull-secret-syncer-xj584\" (UID: \"1a74da60-0284-410b-9d69-0d5192ce0928\") " pod="kube-system/global-pull-secret-syncer-xj584"
Oct 30 20:38:38.221599 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.220423    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-multus-daemon-config\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:38:38.221599 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.220519    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:38:38.221599 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.220526    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/53aaf696-7925-43b6-86fd-1b536b3095d3-metrics-client-ca\") pod \"node-exporter-lwhq8\" (UID: \"53aaf696-7925-43b6-86fd-1b536b3095d3\") " pod="openshift-monitoring/node-exporter-lwhq8"
Oct 30 20:38:38.221599 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.220557    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6b559cf8-14bb-4eaf-9bea-47a956284d20-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8khvs\" (UID: \"6b559cf8-14bb-4eaf-9bea-47a956284d20\") " pod="openshift-insights/insights-runtime-extractor-8khvs"
Oct 30 20:38:38.221599 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.220582    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/764461d5-e0b2-4856-a97e-0c5d3d34a497-ovnkube-script-lib\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:38:38.221599 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.220614    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-multus-daemon-config\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:38:38.221599 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.220621    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a7b1237b-e022-4570-9c48-4a8449996c69-device-dir\") pod \"aws-ebs-csi-driver-node-nvcbb\" (UID: \"a7b1237b-e022-4570-9c48-4a8449996c69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvcbb"
Oct 30 20:38:38.221599 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.220661    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:38.221599 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.220682    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7f60156-3771-486a-a9d2-7acd5ab9f475-var-lib-kubelet\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:38:38.221599 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.220699    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/520d950c-2ec5-457f-b55f-51bd29207757-cert\") pod \"ingress-canary-jgvrx\" (UID: \"520d950c-2ec5-457f-b55f-51bd29207757\") " pod="openshift-ingress-canary/ingress-canary-jgvrx"
Oct 30 20:38:38.221599 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.220783    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c7a31d20-2bf3-4c38-8816-2181571a1fda-hosts-file\") pod \"node-resolver-d69bq\" (UID: \"c7a31d20-2bf3-4c38-8816-2181571a1fda\") " pod="openshift-dns/node-resolver-d69bq"
Oct 30 20:38:38.221599 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.220820    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/764461d5-e0b2-4856-a97e-0c5d3d34a497-ovnkube-script-lib\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:38:38.221599 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.220834    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c3704770-6fb6-4086-8a22-9b29468d5a70-agent-certs\") pod \"konnectivity-agent-lkxfb\" (UID: \"c3704770-6fb6-4086-8a22-9b29468d5a70\") " pod="kube-system/konnectivity-agent-lkxfb"
Oct 30 20:38:38.221599 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.220854    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:38.221599 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.220879    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:38.221599 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.220897    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b7f60156-3771-486a-a9d2-7acd5ab9f475-etc-modprobe-d\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:38:38.221599 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.220912    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/53aaf696-7925-43b6-86fd-1b536b3095d3-node-exporter-tls\") pod \"node-exporter-lwhq8\" (UID: \"53aaf696-7925-43b6-86fd-1b536b3095d3\") " pod="openshift-monitoring/node-exporter-lwhq8"
Oct 30 20:38:38.221599 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.220931    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:38.221599 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.220975    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:38.221599 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.221051    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-multus-conf-dir\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:38:38.221599 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.221068    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:38.221599 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.221083    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-node-log\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:38:38.221599 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.221089    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c3704770-6fb6-4086-8a22-9b29468d5a70-agent-certs\") pod \"konnectivity-agent-lkxfb\" (UID: \"c3704770-6fb6-4086-8a22-9b29468d5a70\") " pod="kube-system/konnectivity-agent-lkxfb"
Oct 30 20:38:38.221599 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.221138    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftcfw\" (UniqueName: \"kubernetes.io/projected/0fb51194-9a91-4430-8ded-4844ffb6727a-kube-api-access-ftcfw\") pod \"iptables-alerter-bpgsn\" (UID: \"0fb51194-9a91-4430-8ded-4844ffb6727a\") " pod="openshift-network-operator/iptables-alerter-bpgsn"
Oct 30 20:38:38.221599 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.221181    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/242a67ca-f9ea-431d-adee-ece061468a02-metrics-certs\") pod \"network-metrics-daemon-hjssq\" (UID: \"242a67ca-f9ea-431d-adee-ece061468a02\") " pod="openshift-multus/network-metrics-daemon-hjssq"
Oct 30 20:38:38.221599 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.221207    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-config-volume\") pod \"alertmanager-main-0\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:38.221599 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.221236    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1004eede-9d90-42b0-bd11-42a32056f2ae-secret-metrics-server-tls\") pod \"metrics-server-bfd9f4b99-p95mj\" (UID: \"1004eede-9d90-42b0-bd11-42a32056f2ae\") " pod="openshift-monitoring/metrics-server-bfd9f4b99-p95mj"
Oct 30 20:38:38.221599 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.221242    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:38.221599 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.221265    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/08ddd17c-0134-45dc-8a2f-aca6e3668e9f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4gl7z\" (UID: \"08ddd17c-0134-45dc-8a2f-aca6e3668e9f\") " pod="openshift-multus/multus-additional-cni-plugins-4gl7z"
Oct 30 20:38:38.221599 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.221292    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/53aaf696-7925-43b6-86fd-1b536b3095d3-root\") pod \"node-exporter-lwhq8\" (UID: \"53aaf696-7925-43b6-86fd-1b536b3095d3\") " pod="openshift-monitoring/node-exporter-lwhq8"
Oct 30 20:38:38.221599 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.221320    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7a31d20-2bf3-4c38-8816-2181571a1fda-tmp-dir\") pod \"node-resolver-d69bq\" (UID: \"c7a31d20-2bf3-4c38-8816-2181571a1fda\") " pod="openshift-dns/node-resolver-d69bq"
Oct 30 20:38:38.221599 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.221361    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z96wz\" (UniqueName: \"kubernetes.io/projected/88a8166b-aa0c-41dc-8bf3-92708c60df57-kube-api-access-z96wz\") pod \"node-ca-5clg5\" (UID: \"88a8166b-aa0c-41dc-8bf3-92708c60df57\") " pod="openshift-image-registry/node-ca-5clg5"
Oct 30 20:38:38.221599 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.221376    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:38.221599 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.221363    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/242a67ca-f9ea-431d-adee-ece061468a02-metrics-certs\") pod \"network-metrics-daemon-hjssq\" (UID: \"242a67ca-f9ea-431d-adee-ece061468a02\") " pod="openshift-multus/network-metrics-daemon-hjssq"
Oct 30 20:38:38.221599 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.221392    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7a31d20-2bf3-4c38-8816-2181571a1fda-tmp-dir\") pod \"node-resolver-d69bq\" (UID: \"c7a31d20-2bf3-4c38-8816-2181571a1fda\") " pod="openshift-dns/node-resolver-d69bq"
Oct 30 20:38:38.221599 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.221393    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b25a9500-23b1-42d8-8a77-ca75dabc6ef5-metrics-tls\") pod \"dns-default-k58xq\" (UID: \"b25a9500-23b1-42d8-8a77-ca75dabc6ef5\") " pod="openshift-dns/dns-default-k58xq"
Oct 30 20:38:38.221599 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.221490    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b7f60156-3771-486a-a9d2-7acd5ab9f475-etc-kubernetes\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:38:38.221599 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.221568    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b7f60156-3771-486a-a9d2-7acd5ab9f475-etc-sysctl-conf\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:38:38.223531 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.221627    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:38.223531 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.221667    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5b900a91-15bc-4019-99f5-4619cea03518-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-cfcqd\" (UID: \"5b900a91-15bc-4019-99f5-4619cea03518\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfcqd"
Oct 30 20:38:38.223531 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.221695    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpbnb\" (UniqueName: \"kubernetes.io/projected/53aaf696-7925-43b6-86fd-1b536b3095d3-kube-api-access-kpbnb\") pod \"node-exporter-lwhq8\" (UID: \"53aaf696-7925-43b6-86fd-1b536b3095d3\") " pod="openshift-monitoring/node-exporter-lwhq8"
Oct 30 20:38:38.223531 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.221736    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-system-cni-dir\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:38:38.223531 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.221765    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:38.223531 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.221782    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-run-systemd\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:38:38.223531 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.221835    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a7b1237b-e022-4570-9c48-4a8449996c69-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nvcbb\" (UID: \"a7b1237b-e022-4570-9c48-4a8449996c69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvcbb"
Oct 30 20:38:38.223531 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.221855    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-log-socket\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:38:38.223531 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.221871    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-config-out\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:38.223531 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.221890    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:38.223531 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.221982    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/53aaf696-7925-43b6-86fd-1b536b3095d3-node-exporter-accelerators-collector-config\") pod \"node-exporter-lwhq8\" (UID: \"53aaf696-7925-43b6-86fd-1b536b3095d3\") " pod="openshift-monitoring/node-exporter-lwhq8"
Oct 30 20:38:38.223531 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.221984    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-config-out\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:38.223531 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.221984    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:38.223531 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.222024    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-host-run-multus-certs\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:38:38.223531 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.222070    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6b559cf8-14bb-4eaf-9bea-47a956284d20-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8khvs\" (UID: \"6b559cf8-14bb-4eaf-9bea-47a956284d20\") " pod="openshift-insights/insights-runtime-extractor-8khvs"
Oct 30 20:38:38.223531 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.222112    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:38.223531 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.222147    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1a74da60-0284-410b-9d69-0d5192ce0928-kubelet-config\") pod \"global-pull-secret-syncer-xj584\" (UID: \"1a74da60-0284-410b-9d69-0d5192ce0928\") " pod="kube-system/global-pull-secret-syncer-xj584"
Oct 30 20:38:38.223531 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.222165    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/53aaf696-7925-43b6-86fd-1b536b3095d3-node-exporter-accelerators-collector-config\") pod \"node-exporter-lwhq8\" (UID: \"53aaf696-7925-43b6-86fd-1b536b3095d3\") " pod="openshift-monitoring/node-exporter-lwhq8"
Oct 30 20:38:38.223531 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.222174    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:38.223531 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.222211    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0fb51194-9a91-4430-8ded-4844ffb6727a-host-slash\") pod \"iptables-alerter-bpgsn\" (UID: \"0fb51194-9a91-4430-8ded-4844ffb6727a\") " pod="openshift-network-operator/iptables-alerter-bpgsn"
Oct 30 20:38:38.223531 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.222259    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-multus-cni-dir\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:38:38.223531 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.222304    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5b900a91-15bc-4019-99f5-4619cea03518-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-cfcqd\" (UID: \"5b900a91-15bc-4019-99f5-4619cea03518\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfcqd"
Oct 30 20:38:38.223531 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.222346    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5b900a91-15bc-4019-99f5-4619cea03518-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-cfcqd\" (UID: \"5b900a91-15bc-4019-99f5-4619cea03518\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfcqd"
Oct 30 20:38:38.223531 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.222351    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:38.223531 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.222377    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgsh5\" (UniqueName: \"kubernetes.io/projected/1004eede-9d90-42b0-bd11-42a32056f2ae-kube-api-access-tgsh5\") pod \"metrics-server-bfd9f4b99-p95mj\" (UID: \"1004eede-9d90-42b0-bd11-42a32056f2ae\") " pod="openshift-monitoring/metrics-server-bfd9f4b99-p95mj"
Oct 30 20:38:38.223531 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.222397    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:38.223531 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.222490    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5b900a91-15bc-4019-99f5-4619cea03518-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-cfcqd\" (UID: \"5b900a91-15bc-4019-99f5-4619cea03518\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfcqd"
Oct 30 20:38:38.223531 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.222541    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5b900a91-15bc-4019-99f5-4619cea03518-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-cfcqd\" (UID: \"5b900a91-15bc-4019-99f5-4619cea03518\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfcqd"
Oct 30 20:38:38.223531 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.222545    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a7b1237b-e022-4570-9c48-4a8449996c69-etc-selinux\") pod \"aws-ebs-csi-driver-node-nvcbb\" (UID: \"a7b1237b-e022-4570-9c48-4a8449996c69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvcbb"
Oct 30 20:38:38.223531 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.222584    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/88a8166b-aa0c-41dc-8bf3-92708c60df57-serviceca\") pod \"node-ca-5clg5\" (UID: \"88a8166b-aa0c-41dc-8bf3-92708c60df57\") " pod="openshift-image-registry/node-ca-5clg5"
Oct 30 20:38:38.223531 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.222618    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6b559cf8-14bb-4eaf-9bea-47a956284d20-data-volume\") pod \"insights-runtime-extractor-8khvs\" (UID: \"6b559cf8-14bb-4eaf-9bea-47a956284d20\") " pod="openshift-insights/insights-runtime-extractor-8khvs"
Oct 30 20:38:38.223531 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.222648    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xq44\" (UniqueName: \"kubernetes.io/projected/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-kube-api-access-8xq44\") pod \"alertmanager-main-0\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:38.223531 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.222672    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1004eede-9d90-42b0-bd11-42a32056f2ae-audit-log\") pod \"metrics-server-bfd9f4b99-p95mj\" (UID: \"1004eede-9d90-42b0-bd11-42a32056f2ae\") " pod="openshift-monitoring/metrics-server-bfd9f4b99-p95mj"
Oct 30 20:38:38.223531 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.222678    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6b559cf8-14bb-4eaf-9bea-47a956284d20-data-volume\") pod \"insights-runtime-extractor-8khvs\" (UID: \"6b559cf8-14bb-4eaf-9bea-47a956284d20\") " pod="openshift-insights/insights-runtime-extractor-8khvs"
Oct 30 20:38:38.223531 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.222697    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1004eede-9d90-42b0-bd11-42a32056f2ae-client-ca-bundle\") pod \"metrics-server-bfd9f4b99-p95mj\" (UID: \"1004eede-9d90-42b0-bd11-42a32056f2ae\") " pod="openshift-monitoring/metrics-server-bfd9f4b99-p95mj"
Oct 30 20:38:38.223531 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.222982    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-config\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:38.223531 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.223002    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/88a8166b-aa0c-41dc-8bf3-92708c60df57-serviceca\") pod \"node-ca-5clg5\" (UID: \"88a8166b-aa0c-41dc-8bf3-92708c60df57\") " pod="openshift-image-registry/node-ca-5clg5"
Oct 30 20:38:38.223531 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.223021    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b7f60156-3771-486a-a9d2-7acd5ab9f475-etc-sysctl-d\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:38:38.223531 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.223066    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1004eede-9d90-42b0-bd11-42a32056f2ae-audit-log\") pod \"metrics-server-bfd9f4b99-p95mj\" (UID: \"1004eede-9d90-42b0-bd11-42a32056f2ae\") " pod="openshift-monitoring/metrics-server-bfd9f4b99-p95mj"
Oct 30 20:38:38.223531 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.223101    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-hostroot\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:38:38.223531 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.223126    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:38.223531 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.223151    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jdjz\" (UniqueName: \"kubernetes.io/projected/5b900a91-15bc-4019-99f5-4619cea03518-kube-api-access-4jdjz\") pod \"kube-state-metrics-5dc8d7d89b-cfcqd\" (UID: \"5b900a91-15bc-4019-99f5-4619cea03518\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfcqd"
Oct 30 20:38:38.223531 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.223226    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-host-run-k8s-cni-cncf-io\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:38:38.223531 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.223263    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7787041b-34dc-4567-8989-22bfd95456e0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-c4xbh\" (UID: \"7787041b-34dc-4567-8989-22bfd95456e0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c4xbh"
Oct 30 20:38:38.223531 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.223285    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1004eede-9d90-42b0-bd11-42a32056f2ae-metrics-server-audit-profiles\") pod \"metrics-server-bfd9f4b99-p95mj\" (UID: \"1004eede-9d90-42b0-bd11-42a32056f2ae\") " pod="openshift-monitoring/metrics-server-bfd9f4b99-p95mj"
Oct 30 20:38:38.223531 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.223311    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-host-kubelet\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:38:38.223531 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.223332    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:38.223531 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.223350    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-host-var-lib-cni-multus\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:38:38.223531 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.223414    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-host-cni-netd\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:38:38.223531 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.223443    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b25a9500-23b1-42d8-8a77-ca75dabc6ef5-config-volume\") pod \"dns-default-k58xq\" (UID: \"b25a9500-23b1-42d8-8a77-ca75dabc6ef5\") " pod="openshift-dns/dns-default-k58xq"
Oct 30 20:38:38.223531 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.223477    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b7f60156-3771-486a-a9d2-7acd5ab9f475-etc-systemd\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:38:38.223531 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.223493    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7f60156-3771-486a-a9d2-7acd5ab9f475-sys\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:38:38.223531 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.223509    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b7f60156-3771-486a-a9d2-7acd5ab9f475-lib-modules\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:38:38.223531 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.223511    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:38.223531 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.223535    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4m4z\" (UniqueName: \"kubernetes.io/projected/242a67ca-f9ea-431d-adee-ece061468a02-kube-api-access-k4m4z\") pod \"network-metrics-daemon-hjssq\" (UID: \"242a67ca-f9ea-431d-adee-ece061468a02\") " pod="openshift-multus/network-metrics-daemon-hjssq"
Oct 30 20:38:38.223531 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.223560    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1004eede-9d90-42b0-bd11-42a32056f2ae-metrics-server-audit-profiles\") pod \"metrics-server-bfd9f4b99-p95mj\" (UID: \"1004eede-9d90-42b0-bd11-42a32056f2ae\") " pod="openshift-monitoring/metrics-server-bfd9f4b99-p95mj"
Oct 30 20:38:38.223531 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.223563    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-config-out\") pod \"alertmanager-main-0\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:38.225304 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.223648    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b25a9500-23b1-42d8-8a77-ca75dabc6ef5-config-volume\") pod \"dns-default-k58xq\" (UID: \"b25a9500-23b1-42d8-8a77-ca75dabc6ef5\") " pod="openshift-dns/dns-default-k58xq"
Oct 30 20:38:38.225304 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.223683    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-config-out\") pod \"alertmanager-main-0\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:38.225304 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.223698    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-etc-openvswitch\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:38:38.225304 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.223729    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl4qd\" (UniqueName: \"kubernetes.io/projected/764461d5-e0b2-4856-a97e-0c5d3d34a497-kube-api-access-nl4qd\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:38:38.256874 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.256823    5713 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 20:38:38.256957 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:38:38.256873    5713 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-165.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-165.us-west-1.compute.internal"
Oct 30 20:38:38.259665 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.259647    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 20:38:38.259749 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.259701    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 20:38:38.259749 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.259728    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 30 20:38:38.259835 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.259811    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 20:38:38.260161 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.260147    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 20:38:38.260377 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.260352    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 20:38:38.260444 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.260420    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 20:38:38.260509 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.260479    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 20:38:38.260555 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.260504    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-55jjhefcr4ig6\""
Oct 30 20:38:38.260555 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.260534    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 20:38:38.260645 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.260585    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 20:38:38.260990 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.260971    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 20:38:38.261099 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.260992    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Oct 30 20:38:38.261099 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.261049    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/520d950c-2ec5-457f-b55f-51bd29207757-cert\") pod \"ingress-canary-jgvrx\" (UID: \"520d950c-2ec5-457f-b55f-51bd29207757\") " pod="openshift-ingress-canary/ingress-canary-jgvrx"
Oct 30 20:38:38.261099 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.261065    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 20:38:38.261099 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.261053    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 30 20:38:38.261099 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.261083    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 20:38:38.261367 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.261111    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 20:38:38.261367 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.261136    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 20:38:38.261367 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.261161    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-pqmk2\""
Oct 30 20:38:38.261367 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.261208    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 20:38:38.261367 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.261236    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 20:38:38.261367 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.261281    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-nl57c\""
Oct 30 20:38:38.261367 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.261288    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-szlhz\""
Oct 30 20:38:38.261367 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.261307    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-jt52d\""
Oct 30 20:38:38.261367 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.261298    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 20:38:38.261782 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.261394    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 30 20:38:38.261782 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.261418    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:38.261782 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.261468    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 30 20:38:38.261782 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.261471    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-h8n42\""
Oct 30 20:38:38.261782 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.261498    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 20:38:38.261782 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.261512    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-config-volume\") pod \"alertmanager-main-0\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:38.261782 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.261569    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-kr8d5\""
Oct 30 20:38:38.261782 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.261647    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-envd3r50lfh67\""
Oct 30 20:38:38.261782 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.261668    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b25a9500-23b1-42d8-8a77-ca75dabc6ef5-metrics-tls\") pod \"dns-default-k58xq\" (UID: \"b25a9500-23b1-42d8-8a77-ca75dabc6ef5\") " pod="openshift-dns/dns-default-k58xq"
Oct 30 20:38:38.261782 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.261777    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 20:38:38.262285 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.261831    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 20:38:38.262285 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.261953    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:38.262285 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.261974    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5b900a91-15bc-4019-99f5-4619cea03518-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-cfcqd\" (UID: \"5b900a91-15bc-4019-99f5-4619cea03518\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfcqd"
Oct 30 20:38:38.262285 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.261974    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dmr5l\""
Oct 30 20:38:38.262285 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.262017    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2srwc\""
Oct 30 20:38:38.262285 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.262089    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-94s75\""
Oct 30 20:38:38.262285 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.262097    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 20:38:38.262285 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.262136    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wsjv8\""
Oct 30 20:38:38.262285 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.262196    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 20:38:38.262285 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.262233    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:38.262285 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.262260    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 20:38:38.262285 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.262280    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6b559cf8-14bb-4eaf-9bea-47a956284d20-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8khvs\" (UID: \"6b559cf8-14bb-4eaf-9bea-47a956284d20\") " pod="openshift-insights/insights-runtime-extractor-8khvs"
Oct 30 20:38:38.262630 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.262309    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Oct 30 20:38:38.262630 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.262403    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:38.263247 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.263233    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1004eede-9d90-42b0-bd11-42a32056f2ae-client-ca-bundle\") pod \"metrics-server-bfd9f4b99-p95mj\" (UID: \"1004eede-9d90-42b0-bd11-42a32056f2ae\") " pod="openshift-monitoring/metrics-server-bfd9f4b99-p95mj"
Oct 30 20:38:38.263565 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.263552    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-config\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:38.263671 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.263659    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:38.263704 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.263677    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7787041b-34dc-4567-8989-22bfd95456e0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-c4xbh\" (UID: \"7787041b-34dc-4567-8989-22bfd95456e0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c4xbh"
Oct 30 20:38:38.267479 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.267461    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:38.267840 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.267819    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7787041b-34dc-4567-8989-22bfd95456e0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-c4xbh\" (UID: \"7787041b-34dc-4567-8989-22bfd95456e0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c4xbh"
Oct 30 20:38:38.267905 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.267872    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:38.268008 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.267989    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:38.271564 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.268921    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:38.271564 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.269046    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:38.271564 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.269200    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:38.271564 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.269239    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5b900a91-15bc-4019-99f5-4619cea03518-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-cfcqd\" (UID: \"5b900a91-15bc-4019-99f5-4619cea03518\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfcqd"
Oct 30 20:38:38.271564 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.269403    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:38.271564 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.269494    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-web-config\") pod \"alertmanager-main-0\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:38.271564 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.269583    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1004eede-9d90-42b0-bd11-42a32056f2ae-secret-metrics-server-client-certs\") pod \"metrics-server-bfd9f4b99-p95mj\" (UID: \"1004eede-9d90-42b0-bd11-42a32056f2ae\") " pod="openshift-monitoring/metrics-server-bfd9f4b99-p95mj"
Oct 30 20:38:38.271564 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.269639    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:38.271564 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.271557    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-web-config\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:38.271924 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.271674    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1004eede-9d90-42b0-bd11-42a32056f2ae-secret-metrics-server-tls\") pod \"metrics-server-bfd9f4b99-p95mj\" (UID: \"1004eede-9d90-42b0-bd11-42a32056f2ae\") " pod="openshift-monitoring/metrics-server-bfd9f4b99-p95mj"
Oct 30 20:38:38.272108 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.272080    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/53aaf696-7925-43b6-86fd-1b536b3095d3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lwhq8\" (UID: \"53aaf696-7925-43b6-86fd-1b536b3095d3\") " pod="openshift-monitoring/node-exporter-lwhq8"
Oct 30 20:38:38.272214 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.272077    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:38.272214 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.272126    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/53aaf696-7925-43b6-86fd-1b536b3095d3-node-exporter-tls\") pod \"node-exporter-lwhq8\" (UID: \"53aaf696-7925-43b6-86fd-1b536b3095d3\") " pod="openshift-monitoring/node-exporter-lwhq8"
Oct 30 20:38:38.272532 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.272509    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cjbx\" (UniqueName: \"kubernetes.io/projected/b25a9500-23b1-42d8-8a77-ca75dabc6ef5-kube-api-access-9cjbx\") pod \"dns-default-k58xq\" (UID: \"b25a9500-23b1-42d8-8a77-ca75dabc6ef5\") " pod="openshift-dns/dns-default-k58xq"
Oct 30 20:38:38.272654 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.272632    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl5v9\" (UniqueName: \"kubernetes.io/projected/08ddd17c-0134-45dc-8a2f-aca6e3668e9f-kube-api-access-sl5v9\") pod \"multus-additional-cni-plugins-4gl7z\" (UID: \"08ddd17c-0134-45dc-8a2f-aca6e3668e9f\") " pod="openshift-multus/multus-additional-cni-plugins-4gl7z"
Oct 30 20:38:38.272770 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.272749    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgzz6\" (UniqueName: \"kubernetes.io/projected/520d950c-2ec5-457f-b55f-51bd29207757-kube-api-access-xgzz6\") pod \"ingress-canary-jgvrx\" (UID: \"520d950c-2ec5-457f-b55f-51bd29207757\") " pod="openshift-ingress-canary/ingress-canary-jgvrx"
Oct 30 20:38:38.272909 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.272893    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzhsg\" (UniqueName: \"kubernetes.io/projected/e54d0918-84e8-40f3-9b18-6e2fafb3d500-kube-api-access-bzhsg\") pod \"network-check-target-kbpjj\" (UID: \"e54d0918-84e8-40f3-9b18-6e2fafb3d500\") " pod="openshift-network-diagnostics/network-check-target-kbpjj"
Oct 30 20:38:38.273086 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.273072    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frzdn\" (UniqueName: \"kubernetes.io/projected/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-kube-api-access-frzdn\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:38:38.273180 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.273165    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx4tv\" (UniqueName: \"kubernetes.io/projected/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-kube-api-access-rx4tv\") pod \"prometheus-k8s-0\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:38.274522 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.274505    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkjkh\" (UniqueName: \"kubernetes.io/projected/a7b1237b-e022-4570-9c48-4a8449996c69-kube-api-access-xkjkh\") pod \"aws-ebs-csi-driver-node-nvcbb\" (UID: \"a7b1237b-e022-4570-9c48-4a8449996c69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvcbb"
Oct 30 20:38:38.274587 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.274557    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqm5k\" (UniqueName: \"kubernetes.io/projected/c7a31d20-2bf3-4c38-8816-2181571a1fda-kube-api-access-jqm5k\") pod \"node-resolver-d69bq\" (UID: \"c7a31d20-2bf3-4c38-8816-2181571a1fda\") " pod="openshift-dns/node-resolver-d69bq"
Oct 30 20:38:38.274676 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.274662    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6qnn\" (UniqueName: \"kubernetes.io/projected/b7f60156-3771-486a-a9d2-7acd5ab9f475-kube-api-access-k6qnn\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:38:38.276024 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.276009    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m476q\" (UniqueName: \"kubernetes.io/projected/6b559cf8-14bb-4eaf-9bea-47a956284d20-kube-api-access-m476q\") pod \"insights-runtime-extractor-8khvs\" (UID: \"6b559cf8-14bb-4eaf-9bea-47a956284d20\") " pod="openshift-insights/insights-runtime-extractor-8khvs"
Oct 30 20:38:38.276162 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.276148    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z96wz\" (UniqueName: \"kubernetes.io/projected/88a8166b-aa0c-41dc-8bf3-92708c60df57-kube-api-access-z96wz\") pod \"node-ca-5clg5\" (UID: \"88a8166b-aa0c-41dc-8bf3-92708c60df57\") " pod="openshift-image-registry/node-ca-5clg5"
Oct 30 20:38:38.276960 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.276946    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftcfw\" (UniqueName: \"kubernetes.io/projected/0fb51194-9a91-4430-8ded-4844ffb6727a-kube-api-access-ftcfw\") pod \"iptables-alerter-bpgsn\" (UID: \"0fb51194-9a91-4430-8ded-4844ffb6727a\") " pod="openshift-network-operator/iptables-alerter-bpgsn"
Oct 30 20:38:38.277656 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.277640    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpbnb\" (UniqueName: \"kubernetes.io/projected/53aaf696-7925-43b6-86fd-1b536b3095d3-kube-api-access-kpbnb\") pod \"node-exporter-lwhq8\" (UID: \"53aaf696-7925-43b6-86fd-1b536b3095d3\") " pod="openshift-monitoring/node-exporter-lwhq8"
Oct 30 20:38:38.278020 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.278007    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgsh5\" (UniqueName: \"kubernetes.io/projected/1004eede-9d90-42b0-bd11-42a32056f2ae-kube-api-access-tgsh5\") pod \"metrics-server-bfd9f4b99-p95mj\" (UID: \"1004eede-9d90-42b0-bd11-42a32056f2ae\") " pod="openshift-monitoring/metrics-server-bfd9f4b99-p95mj"
Oct 30 20:38:38.278308 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.278293    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvnz6\" (UniqueName: \"kubernetes.io/projected/7787041b-34dc-4567-8989-22bfd95456e0-kube-api-access-bvnz6\") pod \"openshift-state-metrics-7bd7489957-c4xbh\" (UID: \"7787041b-34dc-4567-8989-22bfd95456e0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c4xbh"
Oct 30 20:38:38.278771 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.278754    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xq44\" (UniqueName: \"kubernetes.io/projected/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-kube-api-access-8xq44\") pod \"alertmanager-main-0\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:38:38.280888 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.280868    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4m4z\" (UniqueName: \"kubernetes.io/projected/242a67ca-f9ea-431d-adee-ece061468a02-kube-api-access-k4m4z\") pod \"network-metrics-daemon-hjssq\" (UID: \"242a67ca-f9ea-431d-adee-ece061468a02\") " pod="openshift-multus/network-metrics-daemon-hjssq"
Oct 30 20:38:38.280976 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.280956    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jdjz\" (UniqueName: \"kubernetes.io/projected/5b900a91-15bc-4019-99f5-4619cea03518-kube-api-access-4jdjz\") pod \"kube-state-metrics-5dc8d7d89b-cfcqd\" (UID: \"5b900a91-15bc-4019-99f5-4619cea03518\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-cfcqd"
Oct 30 20:38:38.281087 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.281071    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl4qd\" (UniqueName: \"kubernetes.io/projected/764461d5-e0b2-4856-a97e-0c5d3d34a497-kube-api-access-nl4qd\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:38:38.324201 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.324179    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08ddd17c-0134-45dc-8a2f-aca6e3668e9f-system-cni-dir\") pod \"multus-additional-cni-plugins-4gl7z\" (UID: \"08ddd17c-0134-45dc-8a2f-aca6e3668e9f\") " pod="openshift-multus/multus-additional-cni-plugins-4gl7z"
Oct 30 20:38:38.324289 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.324204    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-systemd-units\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:38:38.324289 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.324219    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-host-slash\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:38:38.324289 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.324233    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/53aaf696-7925-43b6-86fd-1b536b3095d3-node-exporter-wtmp\") pod \"node-exporter-lwhq8\" (UID: \"53aaf696-7925-43b6-86fd-1b536b3095d3\") " pod="openshift-monitoring/node-exporter-lwhq8"
Oct 30 20:38:38.324289 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.324247    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-host-run-netns\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:38:38.324289 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.324271    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/88a8166b-aa0c-41dc-8bf3-92708c60df57-host\") pod \"node-ca-5clg5\" (UID: \"88a8166b-aa0c-41dc-8bf3-92708c60df57\") " pod="openshift-image-registry/node-ca-5clg5"
Oct 30 20:38:38.324491 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.324289    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-systemd-units\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:38:38.324491 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.324302    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-host-slash\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:38:38.324491 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.324314    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/88a8166b-aa0c-41dc-8bf3-92708c60df57-host\") pod \"node-ca-5clg5\" (UID: \"88a8166b-aa0c-41dc-8bf3-92708c60df57\") " pod="openshift-image-registry/node-ca-5clg5"
Oct 30 20:38:38.324491 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.324325    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-host-run-netns\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:38:38.324491 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.324339    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08ddd17c-0134-45dc-8a2f-aca6e3668e9f-system-cni-dir\") pod \"multus-additional-cni-plugins-4gl7z\" (UID: \"08ddd17c-0134-45dc-8a2f-aca6e3668e9f\") " pod="openshift-multus/multus-additional-cni-plugins-4gl7z"
Oct 30 20:38:38.324491 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.324341    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/53aaf696-7925-43b6-86fd-1b536b3095d3-node-exporter-wtmp\") pod \"node-exporter-lwhq8\" (UID: \"53aaf696-7925-43b6-86fd-1b536b3095d3\") " pod="openshift-monitoring/node-exporter-lwhq8"
Oct 30 20:38:38.324491 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.324383    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-etc-kubernetes\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:38:38.324491 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.324406    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a7b1237b-e022-4570-9c48-4a8449996c69-registration-dir\") pod \"aws-ebs-csi-driver-node-nvcbb\" (UID: \"a7b1237b-e022-4570-9c48-4a8449996c69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvcbb"
Oct 30 20:38:38.324491 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.324421    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b7f60156-3771-486a-a9d2-7acd5ab9f475-run\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:38:38.324491 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.324438    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-cnibin\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:38:38.324491 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.324462    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/08ddd17c-0134-45dc-8a2f-aca6e3668e9f-cnibin\") pod \"multus-additional-cni-plugins-4gl7z\" (UID: \"08ddd17c-0134-45dc-8a2f-aca6e3668e9f\") " pod="openshift-multus/multus-additional-cni-plugins-4gl7z"
Oct 30 20:38:38.324491 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.324476    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a7b1237b-e022-4570-9c48-4a8449996c69-registration-dir\") pod \"aws-ebs-csi-driver-node-nvcbb\" (UID: \"a7b1237b-e022-4570-9c48-4a8449996c69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvcbb"
Oct 30 20:38:38.324491 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.324487    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b7f60156-3771-486a-a9d2-7acd5ab9f475-run\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:38:38.324888 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.324502    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-cnibin\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:38:38.324888 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.324506    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-etc-kubernetes\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:38:38.324888 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.324484    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a7b1237b-e022-4570-9c48-4a8449996c69-sys-fs\") pod \"aws-ebs-csi-driver-node-nvcbb\" (UID: \"a7b1237b-e022-4570-9c48-4a8449996c69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvcbb"
Oct 30 20:38:38.324888 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.324524    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/08ddd17c-0134-45dc-8a2f-aca6e3668e9f-cnibin\") pod \"multus-additional-cni-plugins-4gl7z\" (UID: \"08ddd17c-0134-45dc-8a2f-aca6e3668e9f\") " pod="openshift-multus/multus-additional-cni-plugins-4gl7z"
Oct 30 20:38:38.324888 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.324541    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a7b1237b-e022-4570-9c48-4a8449996c69-sys-fs\") pod \"aws-ebs-csi-driver-node-nvcbb\" (UID: \"a7b1237b-e022-4570-9c48-4a8449996c69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvcbb"
Oct 30 20:38:38.324888 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.324545    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b7f60156-3771-486a-a9d2-7acd5ab9f475-host\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:38:38.324888 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.324569    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b7f60156-3771-486a-a9d2-7acd5ab9f475-host\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:38:38.324888 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.324572    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-host-var-lib-kubelet\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:38:38.324888 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.324595    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-multus-socket-dir-parent\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:38:38.324888 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.324631    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-host-var-lib-kubelet\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:38:38.324888 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.324643    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-multus-socket-dir-parent\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:38:38.324888 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.324656    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-host-run-netns\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:38:38.324888 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.324687    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-run-openvswitch\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:38:38.324888 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.324703    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6b559cf8-14bb-4eaf-9bea-47a956284d20-crio-socket\") pod \"insights-runtime-extractor-8khvs\" (UID: \"6b559cf8-14bb-4eaf-9bea-47a956284d20\") " pod="openshift-insights/insights-runtime-extractor-8khvs"
Oct 30 20:38:38.324888 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.324715    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-host-run-netns\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:38:38.324888 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.324726    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-host-cni-bin\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:38:38.324888 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.324744    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6b559cf8-14bb-4eaf-9bea-47a956284d20-crio-socket\") pod \"insights-runtime-extractor-8khvs\" (UID: \"6b559cf8-14bb-4eaf-9bea-47a956284d20\") " pod="openshift-insights/insights-runtime-extractor-8khvs"
Oct 30 20:38:38.324888 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.324748    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a7b1237b-e022-4570-9c48-4a8449996c69-socket-dir\") pod \"aws-ebs-csi-driver-node-nvcbb\" (UID: \"a7b1237b-e022-4570-9c48-4a8449996c69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvcbb"
Oct 30 20:38:38.324888 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.324766    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-host-cni-bin\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:38:38.324888 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.324750    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-run-openvswitch\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:38:38.324888 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.324773    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b7f60156-3771-486a-a9d2-7acd5ab9f475-etc-sysconfig\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:38:38.324888 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.324780    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a7b1237b-e022-4570-9c48-4a8449996c69-socket-dir\") pod \"aws-ebs-csi-driver-node-nvcbb\" (UID: \"a7b1237b-e022-4570-9c48-4a8449996c69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvcbb"
Oct 30 20:38:38.324888 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.324823    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1a74da60-0284-410b-9d69-0d5192ce0928-dbus\") pod \"global-pull-secret-syncer-xj584\" (UID: \"1a74da60-0284-410b-9d69-0d5192ce0928\") " pod="kube-system/global-pull-secret-syncer-xj584"
Oct 30 20:38:38.324888 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.324847    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:38:38.324888 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.324857    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1a74da60-0284-410b-9d69-0d5192ce0928-dbus\") pod \"global-pull-secret-syncer-xj584\" (UID: \"1a74da60-0284-410b-9d69-0d5192ce0928\") " pod="kube-system/global-pull-secret-syncer-xj584"
Oct 30 20:38:38.324888 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.324827    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b7f60156-3771-486a-a9d2-7acd5ab9f475-etc-sysconfig\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:38:38.324888 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.324871    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a7b1237b-e022-4570-9c48-4a8449996c69-device-dir\") pod \"aws-ebs-csi-driver-node-nvcbb\" (UID: \"a7b1237b-e022-4570-9c48-4a8449996c69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvcbb"
Oct 30 20:38:38.324888 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.324889    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:38:38.324888 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.324893    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7f60156-3771-486a-a9d2-7acd5ab9f475-var-lib-kubelet\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:38:38.325970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.324924    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c7a31d20-2bf3-4c38-8816-2181571a1fda-hosts-file\") pod \"node-resolver-d69bq\" (UID: \"c7a31d20-2bf3-4c38-8816-2181571a1fda\") " pod="openshift-dns/node-resolver-d69bq"
Oct 30 20:38:38.325970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.324933    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a7b1237b-e022-4570-9c48-4a8449996c69-device-dir\") pod \"aws-ebs-csi-driver-node-nvcbb\" (UID: \"a7b1237b-e022-4570-9c48-4a8449996c69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvcbb"
Oct 30 20:38:38.325970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.324941    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b7f60156-3771-486a-a9d2-7acd5ab9f475-etc-modprobe-d\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:38:38.325970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.324974    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7f60156-3771-486a-a9d2-7acd5ab9f475-var-lib-kubelet\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:38:38.325970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.324979    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c7a31d20-2bf3-4c38-8816-2181571a1fda-hosts-file\") pod \"node-resolver-d69bq\" (UID: \"c7a31d20-2bf3-4c38-8816-2181571a1fda\") " pod="openshift-dns/node-resolver-d69bq"
Oct 30 20:38:38.325970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.324993    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-multus-conf-dir\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:38:38.325970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.325007    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-node-log\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:38:38.325970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.325024    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/08ddd17c-0134-45dc-8a2f-aca6e3668e9f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4gl7z\" (UID: \"08ddd17c-0134-45dc-8a2f-aca6e3668e9f\") " pod="openshift-multus/multus-additional-cni-plugins-4gl7z"
Oct 30 20:38:38.325970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.325028    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b7f60156-3771-486a-a9d2-7acd5ab9f475-etc-modprobe-d\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:38:38.325970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.325045    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-multus-conf-dir\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:38:38.325970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.325057    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/53aaf696-7925-43b6-86fd-1b536b3095d3-root\") pod \"node-exporter-lwhq8\" (UID: \"53aaf696-7925-43b6-86fd-1b536b3095d3\") " pod="openshift-monitoring/node-exporter-lwhq8"
Oct 30 20:38:38.325970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.325067    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/08ddd17c-0134-45dc-8a2f-aca6e3668e9f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4gl7z\" (UID: \"08ddd17c-0134-45dc-8a2f-aca6e3668e9f\") " pod="openshift-multus/multus-additional-cni-plugins-4gl7z"
Oct 30 20:38:38.325970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.325077    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/53aaf696-7925-43b6-86fd-1b536b3095d3-root\") pod \"node-exporter-lwhq8\" (UID: \"53aaf696-7925-43b6-86fd-1b536b3095d3\") " pod="openshift-monitoring/node-exporter-lwhq8"
Oct 30 20:38:38.325970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.325086    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b7f60156-3771-486a-a9d2-7acd5ab9f475-etc-kubernetes\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:38:38.325970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.325103    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-node-log\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:38:38.325970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.325104    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b7f60156-3771-486a-a9d2-7acd5ab9f475-etc-sysctl-conf\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:38:38.325970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.325111    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b7f60156-3771-486a-a9d2-7acd5ab9f475-etc-kubernetes\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:38:38.325970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.325131    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-system-cni-dir\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:38:38.325970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.325157    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-run-systemd\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:38:38.325970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.325179    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a7b1237b-e022-4570-9c48-4a8449996c69-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nvcbb\" (UID: \"a7b1237b-e022-4570-9c48-4a8449996c69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvcbb"
Oct 30 20:38:38.325970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.325200    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-log-socket\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:38:38.325970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.325211    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b7f60156-3771-486a-a9d2-7acd5ab9f475-etc-sysctl-conf\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:38:38.325970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.325225    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-host-run-multus-certs\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:38:38.325970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.325228    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-system-cni-dir\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:38:38.325970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.325226    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-run-systemd\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:38:38.325970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.325249    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1a74da60-0284-410b-9d69-0d5192ce0928-kubelet-config\") pod \"global-pull-secret-syncer-xj584\" (UID: \"1a74da60-0284-410b-9d69-0d5192ce0928\") " pod="kube-system/global-pull-secret-syncer-xj584"
Oct 30 20:38:38.325970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.325256    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-host-run-multus-certs\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:38:38.325970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.325251    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-log-socket\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:38:38.325970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.325274    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0fb51194-9a91-4430-8ded-4844ffb6727a-host-slash\") pod \"iptables-alerter-bpgsn\" (UID: \"0fb51194-9a91-4430-8ded-4844ffb6727a\") " pod="openshift-network-operator/iptables-alerter-bpgsn"
Oct 30 20:38:38.325970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.325284    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1a74da60-0284-410b-9d69-0d5192ce0928-kubelet-config\") pod \"global-pull-secret-syncer-xj584\" (UID: \"1a74da60-0284-410b-9d69-0d5192ce0928\") " pod="kube-system/global-pull-secret-syncer-xj584"
Oct 30 20:38:38.325970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.325258    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a7b1237b-e022-4570-9c48-4a8449996c69-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nvcbb\" (UID: \"a7b1237b-e022-4570-9c48-4a8449996c69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvcbb"
Oct 30 20:38:38.325970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.325298    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-multus-cni-dir\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:38:38.325970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.325321    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a7b1237b-e022-4570-9c48-4a8449996c69-etc-selinux\") pod \"aws-ebs-csi-driver-node-nvcbb\" (UID: \"a7b1237b-e022-4570-9c48-4a8449996c69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvcbb"
Oct 30 20:38:38.325970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.325323    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0fb51194-9a91-4430-8ded-4844ffb6727a-host-slash\") pod \"iptables-alerter-bpgsn\" (UID: \"0fb51194-9a91-4430-8ded-4844ffb6727a\") " pod="openshift-network-operator/iptables-alerter-bpgsn"
Oct 30 20:38:38.325970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.325350    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b7f60156-3771-486a-a9d2-7acd5ab9f475-etc-sysctl-d\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:38:38.325970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.325371    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-hostroot\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:38:38.325970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.325381    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-multus-cni-dir\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:38:38.325970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.325382    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a7b1237b-e022-4570-9c48-4a8449996c69-etc-selinux\") pod \"aws-ebs-csi-driver-node-nvcbb\" (UID: \"a7b1237b-e022-4570-9c48-4a8449996c69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nvcbb"
Oct 30 20:38:38.325970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.325392    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-host-run-k8s-cni-cncf-io\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:38:38.325970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.325408    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-host-kubelet\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:38:38.325970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.325421    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-hostroot\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:38:38.325970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.325424    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-host-var-lib-cni-multus\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:38:38.325970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.325457    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-host-cni-netd\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:38:38.325970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.325464    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-host-var-lib-cni-multus\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:38:38.325970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.325475    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b7f60156-3771-486a-a9d2-7acd5ab9f475-etc-sysctl-d\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:38:38.325970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.325486    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-host-run-k8s-cni-cncf-io\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:38:38.325970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.325483    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b7f60156-3771-486a-a9d2-7acd5ab9f475-etc-systemd\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:38:38.325970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.325514    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-host-kubelet\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:38:38.325970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.325524    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-host-cni-netd\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:38:38.325970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.325532    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7f60156-3771-486a-a9d2-7acd5ab9f475-sys\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:38:38.325970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.325538    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b7f60156-3771-486a-a9d2-7acd5ab9f475-etc-systemd\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:38:38.325970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.325569    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7f60156-3771-486a-a9d2-7acd5ab9f475-sys\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:38:38.325970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.325594    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b7f60156-3771-486a-a9d2-7acd5ab9f475-lib-modules\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:38:38.325970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.325623    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-etc-openvswitch\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:38:38.325970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.325647    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-var-lib-openvswitch\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:38:38.325970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.325661    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-run-ovn\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:38:38.325970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.325685    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-host-var-lib-cni-bin\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:38:38.325970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.325690    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-etc-openvswitch\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:38:38.325970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.325687    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b7f60156-3771-486a-a9d2-7acd5ab9f475-lib-modules\") pod \"tuned-brmwl\" (UID: \"b7f60156-3771-486a-a9d2-7acd5ab9f475\") " pod="openshift-cluster-node-tuning-operator/tuned-brmwl"
Oct 30 20:38:38.325970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.325707    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-var-lib-openvswitch\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:38:38.325970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.325711    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-host-run-ovn-kubernetes\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:38:38.325970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.325718    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-run-ovn\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:38:38.325970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.325744    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/53aaf696-7925-43b6-86fd-1b536b3095d3-sys\") pod \"node-exporter-lwhq8\" (UID: \"53aaf696-7925-43b6-86fd-1b536b3095d3\") " pod="openshift-monitoring/node-exporter-lwhq8"
Oct 30 20:38:38.325970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.325749    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/764461d5-e0b2-4856-a97e-0c5d3d34a497-host-run-ovn-kubernetes\") pod \"ovnkube-node-r7lpf\" (UID: \"764461d5-e0b2-4856-a97e-0c5d3d34a497\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:38:38.325970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.325742    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/af5cc97c-3f88-40e2-b6e8-e5f7b279bf85-host-var-lib-cni-bin\") pod \"multus-h4rs7\" (UID: \"af5cc97c-3f88-40e2-b6e8-e5f7b279bf85\") " pod="openshift-multus/multus-h4rs7"
Oct 30 20:38:38.325970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.325813    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/53aaf696-7925-43b6-86fd-1b536b3095d3-sys\") pod \"node-exporter-lwhq8\" (UID: \"53aaf696-7925-43b6-86fd-1b536b3095d3\") " pod="openshift-monitoring/node-exporter-lwhq8"
Oct 30 20:38:38.784212 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:38.784178    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-k58xq"
Oct 30 20:38:39.036998 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:39.036918    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:39.052452 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:39.052423    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:41.270286 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:41.270250    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:38:41.420278 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:41.415954    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kbpjj"
Oct 30 20:38:44.309472 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:44.309436    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:38:44.611060 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:44.610989    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-lkxfb"
Oct 30 20:38:46.134211 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:46.134176    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:38:46.134571 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:46.134443    5713 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 20:38:54.457062 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:38:54.456982    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 20:38:58.773847 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:38:58.773819    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-bfd9f4b99-p95mj"
Oct 30 20:39:02.525967 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:02.525939    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r7lpf"
Oct 30 20:39:11.419492 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:11.419466    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-kbpjj"
Oct 30 20:39:24.457765 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:39:24.457728    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 20:39:41.286639 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:41.286609    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:39:51.100748 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:51.100714    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 20:39:51.101603 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:51.101548    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1c4609ce-58a7-47f3-8eaa-0f19cea5c6db" containerName="alertmanager" containerID="cri-o://d38e224686999e89588a3fa2b45d1c29d2c94553d17e7e25e2e3251c0281ab4a" gracePeriod=120
Oct 30 20:39:51.101913 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:51.101861    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1c4609ce-58a7-47f3-8eaa-0f19cea5c6db" containerName="kube-rbac-proxy" containerID="cri-o://a70e118fb15f6bb5edca24ed3dfeb614808b8b6100704b9dbdfb9be04e21f3a7" gracePeriod=120
Oct 30 20:39:51.102004 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:51.101940    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1c4609ce-58a7-47f3-8eaa-0f19cea5c6db" containerName="kube-rbac-proxy-web" containerID="cri-o://f27e0161f269a31de0d92e4b6943ef3c333705a2895885f8d95a3aec9b5a248a" gracePeriod=120
Oct 30 20:39:51.102058 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:51.101877    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1c4609ce-58a7-47f3-8eaa-0f19cea5c6db" containerName="config-reloader" containerID="cri-o://c7296d272b2a7cdb3a06dff55dcf6811d03f31db5bfc6882918389bfb6e3b6cd" gracePeriod=120
Oct 30 20:39:51.102108 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:51.102056    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1c4609ce-58a7-47f3-8eaa-0f19cea5c6db" containerName="kube-rbac-proxy-metric" containerID="cri-o://bd8e64fb9dc49d675e305dfe67dd2ac11c07e5014c92311e685bc19270a55b8b" gracePeriod=120
Oct 30 20:39:51.102483 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:51.102038    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1c4609ce-58a7-47f3-8eaa-0f19cea5c6db" containerName="prom-label-proxy" containerID="cri-o://d140b08b6210f2b3667f398b74f0086088403d96b90c4b000f43ea224ac6b458" gracePeriod=120
Oct 30 20:39:51.413849 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:51.413746    5713 generic.go:356] "Generic (PLEG): container finished" podID="1c4609ce-58a7-47f3-8eaa-0f19cea5c6db" containerID="d140b08b6210f2b3667f398b74f0086088403d96b90c4b000f43ea224ac6b458" exitCode=0
Oct 30 20:39:51.413849 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:51.413770    5713 generic.go:356] "Generic (PLEG): container finished" podID="1c4609ce-58a7-47f3-8eaa-0f19cea5c6db" containerID="a70e118fb15f6bb5edca24ed3dfeb614808b8b6100704b9dbdfb9be04e21f3a7" exitCode=0
Oct 30 20:39:51.413849 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:51.413779    5713 generic.go:356] "Generic (PLEG): container finished" podID="1c4609ce-58a7-47f3-8eaa-0f19cea5c6db" containerID="c7296d272b2a7cdb3a06dff55dcf6811d03f31db5bfc6882918389bfb6e3b6cd" exitCode=0
Oct 30 20:39:51.413849 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:51.413813    5713 generic.go:356] "Generic (PLEG): container finished" podID="1c4609ce-58a7-47f3-8eaa-0f19cea5c6db" containerID="d38e224686999e89588a3fa2b45d1c29d2c94553d17e7e25e2e3251c0281ab4a" exitCode=0
Oct 30 20:39:51.413849 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:51.413825    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db","Type":"ContainerDied","Data":"d140b08b6210f2b3667f398b74f0086088403d96b90c4b000f43ea224ac6b458"}
Oct 30 20:39:51.414158 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:51.413936    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db","Type":"ContainerDied","Data":"a70e118fb15f6bb5edca24ed3dfeb614808b8b6100704b9dbdfb9be04e21f3a7"}
Oct 30 20:39:51.414158 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:51.413978    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db","Type":"ContainerDied","Data":"c7296d272b2a7cdb3a06dff55dcf6811d03f31db5bfc6882918389bfb6e3b6cd"}
Oct 30 20:39:51.414158 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:51.414013    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db","Type":"ContainerDied","Data":"d38e224686999e89588a3fa2b45d1c29d2c94553d17e7e25e2e3251c0281ab4a"}
Oct 30 20:39:52.358173 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.358145    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:39:52.421464 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.421393    5713 generic.go:356] "Generic (PLEG): container finished" podID="1c4609ce-58a7-47f3-8eaa-0f19cea5c6db" containerID="bd8e64fb9dc49d675e305dfe67dd2ac11c07e5014c92311e685bc19270a55b8b" exitCode=0
Oct 30 20:39:52.421464 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.421419    5713 generic.go:356] "Generic (PLEG): container finished" podID="1c4609ce-58a7-47f3-8eaa-0f19cea5c6db" containerID="f27e0161f269a31de0d92e4b6943ef3c333705a2895885f8d95a3aec9b5a248a" exitCode=0
Oct 30 20:39:52.421464 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.421462    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:39:52.421671 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.421480    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db","Type":"ContainerDied","Data":"bd8e64fb9dc49d675e305dfe67dd2ac11c07e5014c92311e685bc19270a55b8b"}
Oct 30 20:39:52.421671 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.421536    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db","Type":"ContainerDied","Data":"f27e0161f269a31de0d92e4b6943ef3c333705a2895885f8d95a3aec9b5a248a"}
Oct 30 20:39:52.421671 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.421565    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db","Type":"ContainerDied","Data":"7ff1c0abadbf0f85df09a48cb328c77e7e85b622024858da9d8c89ef20a49824"}
Oct 30 20:39:52.421671 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.421593    5713 scope.go:122] "RemoveContainer" containerID="d140b08b6210f2b3667f398b74f0086088403d96b90c4b000f43ea224ac6b458"
Oct 30 20:39:52.426843 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.426819    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-secret-alertmanager-kube-rbac-proxy-web\") pod \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") "
Oct 30 20:39:52.426956 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.426874    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-cluster-tls-config\") pod \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") "
Oct 30 20:39:52.426956 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.426904    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-kube-api-access-8xq44\" (UniqueName: \"kubernetes.io/projected/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-kube-api-access-8xq44\") pod \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") "
Oct 30 20:39:52.426956 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.426946    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-config-out\" (UniqueName: \"kubernetes.io/empty-dir/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-config-out\") pod \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") "
Oct 30 20:39:52.427119 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.427069    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-tls-assets\" (UniqueName: \"kubernetes.io/projected/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-tls-assets\") pod \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") "
Oct 30 20:39:52.427186 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.427121    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-alertmanager-main-db\") pod \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") "
Oct 30 20:39:52.427186 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.427181    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-secret-alertmanager-kube-rbac-proxy\") pod \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") "
Oct 30 20:39:52.427286 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.427232    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-web-config\" (UniqueName: \"kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-web-config\") pod \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") "
Oct 30 20:39:52.427286 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.427267    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-secret-alertmanager-kube-rbac-proxy-metric\") pod \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") "
Oct 30 20:39:52.427393 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.427305    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-alertmanager-trusted-ca-bundle\") pod \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") "
Oct 30 20:39:52.427393 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.427345    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-config-volume\" (UniqueName: \"kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-config-volume\") pod \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") "
Oct 30 20:39:52.427393 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.427385    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-metrics-client-ca\") pod \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") "
Oct 30 20:39:52.427714 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.427417    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-secret-alertmanager-main-tls\") pod \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\" (UID: \"1c4609ce-58a7-47f3-8eaa-0f19cea5c6db\") "
Oct 30 20:39:52.427833 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.427806    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-alertmanager-trusted-ca-bundle" pod "1c4609ce-58a7-47f3-8eaa-0f19cea5c6db" (UID: "1c4609ce-58a7-47f3-8eaa-0f19cea5c6db"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:39:52.428280 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.428253    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-alertmanager-main-db" pod "1c4609ce-58a7-47f3-8eaa-0f19cea5c6db" (UID: "1c4609ce-58a7-47f3-8eaa-0f19cea5c6db"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:39:52.428749 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.428726    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-metrics-client-ca" pod "1c4609ce-58a7-47f3-8eaa-0f19cea5c6db" (UID: "1c4609ce-58a7-47f3-8eaa-0f19cea5c6db"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:39:52.430640 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.430589    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-kube-api-access-8xq44" pod "1c4609ce-58a7-47f3-8eaa-0f19cea5c6db" (UID: "1c4609ce-58a7-47f3-8eaa-0f19cea5c6db"). InnerVolumeSpecName "kube-api-access-8xq44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:39:52.431423 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.431396    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-config-out" pod "1c4609ce-58a7-47f3-8eaa-0f19cea5c6db" (UID: "1c4609ce-58a7-47f3-8eaa-0f19cea5c6db"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:39:52.431531 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.431469    5713 scope.go:122] "RemoveContainer" containerID="bd8e64fb9dc49d675e305dfe67dd2ac11c07e5014c92311e685bc19270a55b8b"
Oct 30 20:39:52.432182 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.432142    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-secret-alertmanager-kube-rbac-proxy-metric" pod "1c4609ce-58a7-47f3-8eaa-0f19cea5c6db" (UID: "1c4609ce-58a7-47f3-8eaa-0f19cea5c6db"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:39:52.432607 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.432489    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-secret-alertmanager-kube-rbac-proxy-web" pod "1c4609ce-58a7-47f3-8eaa-0f19cea5c6db" (UID: "1c4609ce-58a7-47f3-8eaa-0f19cea5c6db"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:39:52.432693 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.432657    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-config-volume" pod "1c4609ce-58a7-47f3-8eaa-0f19cea5c6db" (UID: "1c4609ce-58a7-47f3-8eaa-0f19cea5c6db"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:39:52.432755 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.432723    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-tls-assets" pod "1c4609ce-58a7-47f3-8eaa-0f19cea5c6db" (UID: "1c4609ce-58a7-47f3-8eaa-0f19cea5c6db"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:39:52.432850 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.432826    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-secret-alertmanager-main-tls" pod "1c4609ce-58a7-47f3-8eaa-0f19cea5c6db" (UID: "1c4609ce-58a7-47f3-8eaa-0f19cea5c6db"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:39:52.432953 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.432933    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-secret-alertmanager-kube-rbac-proxy" pod "1c4609ce-58a7-47f3-8eaa-0f19cea5c6db" (UID: "1c4609ce-58a7-47f3-8eaa-0f19cea5c6db"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:39:52.435557 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.435533    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-cluster-tls-config" pod "1c4609ce-58a7-47f3-8eaa-0f19cea5c6db" (UID: "1c4609ce-58a7-47f3-8eaa-0f19cea5c6db"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:39:52.441554 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.441534    5713 scope.go:122] "RemoveContainer" containerID="a70e118fb15f6bb5edca24ed3dfeb614808b8b6100704b9dbdfb9be04e21f3a7"
Oct 30 20:39:52.443421 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.443400    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-web-config" pod "1c4609ce-58a7-47f3-8eaa-0f19cea5c6db" (UID: "1c4609ce-58a7-47f3-8eaa-0f19cea5c6db"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:39:52.448148 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.448126    5713 scope.go:122] "RemoveContainer" containerID="f27e0161f269a31de0d92e4b6943ef3c333705a2895885f8d95a3aec9b5a248a"
Oct 30 20:39:52.454429 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.454416    5713 scope.go:122] "RemoveContainer" containerID="c7296d272b2a7cdb3a06dff55dcf6811d03f31db5bfc6882918389bfb6e3b6cd"
Oct 30 20:39:52.460877 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.460853    5713 scope.go:122] "RemoveContainer" containerID="d38e224686999e89588a3fa2b45d1c29d2c94553d17e7e25e2e3251c0281ab4a"
Oct 30 20:39:52.467541 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.467526    5713 scope.go:122] "RemoveContainer" containerID="d6b9d3c29eb082eb14cbc0dd4664369c791f40d5a75aeeb98689299537f59f75"
Oct 30 20:39:52.474158 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.474142    5713 scope.go:122] "RemoveContainer" containerID="d140b08b6210f2b3667f398b74f0086088403d96b90c4b000f43ea224ac6b458"
Oct 30 20:39:52.474415 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:39:52.474397    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d140b08b6210f2b3667f398b74f0086088403d96b90c4b000f43ea224ac6b458\": container with ID starting with d140b08b6210f2b3667f398b74f0086088403d96b90c4b000f43ea224ac6b458 not found: ID does not exist" containerID="d140b08b6210f2b3667f398b74f0086088403d96b90c4b000f43ea224ac6b458"
Oct 30 20:39:52.474458 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.474428    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d140b08b6210f2b3667f398b74f0086088403d96b90c4b000f43ea224ac6b458"} err="failed to get container status \"d140b08b6210f2b3667f398b74f0086088403d96b90c4b000f43ea224ac6b458\": rpc error: code = NotFound desc = could not find container \"d140b08b6210f2b3667f398b74f0086088403d96b90c4b000f43ea224ac6b458\": container with ID starting with d140b08b6210f2b3667f398b74f0086088403d96b90c4b000f43ea224ac6b458 not found: ID does not exist"
Oct 30 20:39:52.474498 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.474458    5713 scope.go:122] "RemoveContainer" containerID="bd8e64fb9dc49d675e305dfe67dd2ac11c07e5014c92311e685bc19270a55b8b"
Oct 30 20:39:52.474706 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:39:52.474689    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd8e64fb9dc49d675e305dfe67dd2ac11c07e5014c92311e685bc19270a55b8b\": container with ID starting with bd8e64fb9dc49d675e305dfe67dd2ac11c07e5014c92311e685bc19270a55b8b not found: ID does not exist" containerID="bd8e64fb9dc49d675e305dfe67dd2ac11c07e5014c92311e685bc19270a55b8b"
Oct 30 20:39:52.474748 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.474715    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd8e64fb9dc49d675e305dfe67dd2ac11c07e5014c92311e685bc19270a55b8b"} err="failed to get container status \"bd8e64fb9dc49d675e305dfe67dd2ac11c07e5014c92311e685bc19270a55b8b\": rpc error: code = NotFound desc = could not find container \"bd8e64fb9dc49d675e305dfe67dd2ac11c07e5014c92311e685bc19270a55b8b\": container with ID starting with bd8e64fb9dc49d675e305dfe67dd2ac11c07e5014c92311e685bc19270a55b8b not found: ID does not exist"
Oct 30 20:39:52.474748 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.474730    5713 scope.go:122] "RemoveContainer" containerID="a70e118fb15f6bb5edca24ed3dfeb614808b8b6100704b9dbdfb9be04e21f3a7"
Oct 30 20:39:52.474977 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:39:52.474962    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a70e118fb15f6bb5edca24ed3dfeb614808b8b6100704b9dbdfb9be04e21f3a7\": container with ID starting with a70e118fb15f6bb5edca24ed3dfeb614808b8b6100704b9dbdfb9be04e21f3a7 not found: ID does not exist" containerID="a70e118fb15f6bb5edca24ed3dfeb614808b8b6100704b9dbdfb9be04e21f3a7"
Oct 30 20:39:52.475017 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.474981    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a70e118fb15f6bb5edca24ed3dfeb614808b8b6100704b9dbdfb9be04e21f3a7"} err="failed to get container status \"a70e118fb15f6bb5edca24ed3dfeb614808b8b6100704b9dbdfb9be04e21f3a7\": rpc error: code = NotFound desc = could not find container \"a70e118fb15f6bb5edca24ed3dfeb614808b8b6100704b9dbdfb9be04e21f3a7\": container with ID starting with a70e118fb15f6bb5edca24ed3dfeb614808b8b6100704b9dbdfb9be04e21f3a7 not found: ID does not exist"
Oct 30 20:39:52.475017 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.474993    5713 scope.go:122] "RemoveContainer" containerID="f27e0161f269a31de0d92e4b6943ef3c333705a2895885f8d95a3aec9b5a248a"
Oct 30 20:39:52.475209 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:39:52.475190    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f27e0161f269a31de0d92e4b6943ef3c333705a2895885f8d95a3aec9b5a248a\": container with ID starting with f27e0161f269a31de0d92e4b6943ef3c333705a2895885f8d95a3aec9b5a248a not found: ID does not exist" containerID="f27e0161f269a31de0d92e4b6943ef3c333705a2895885f8d95a3aec9b5a248a"
Oct 30 20:39:52.475248 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.475215    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f27e0161f269a31de0d92e4b6943ef3c333705a2895885f8d95a3aec9b5a248a"} err="failed to get container status \"f27e0161f269a31de0d92e4b6943ef3c333705a2895885f8d95a3aec9b5a248a\": rpc error: code = NotFound desc = could not find container \"f27e0161f269a31de0d92e4b6943ef3c333705a2895885f8d95a3aec9b5a248a\": container with ID starting with f27e0161f269a31de0d92e4b6943ef3c333705a2895885f8d95a3aec9b5a248a not found: ID does not exist"
Oct 30 20:39:52.475248 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.475230    5713 scope.go:122] "RemoveContainer" containerID="c7296d272b2a7cdb3a06dff55dcf6811d03f31db5bfc6882918389bfb6e3b6cd"
Oct 30 20:39:52.475437 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:39:52.475423    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7296d272b2a7cdb3a06dff55dcf6811d03f31db5bfc6882918389bfb6e3b6cd\": container with ID starting with c7296d272b2a7cdb3a06dff55dcf6811d03f31db5bfc6882918389bfb6e3b6cd not found: ID does not exist" containerID="c7296d272b2a7cdb3a06dff55dcf6811d03f31db5bfc6882918389bfb6e3b6cd"
Oct 30 20:39:52.475472 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.475440    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7296d272b2a7cdb3a06dff55dcf6811d03f31db5bfc6882918389bfb6e3b6cd"} err="failed to get container status \"c7296d272b2a7cdb3a06dff55dcf6811d03f31db5bfc6882918389bfb6e3b6cd\": rpc error: code = NotFound desc = could not find container \"c7296d272b2a7cdb3a06dff55dcf6811d03f31db5bfc6882918389bfb6e3b6cd\": container with ID starting with c7296d272b2a7cdb3a06dff55dcf6811d03f31db5bfc6882918389bfb6e3b6cd not found: ID does not exist"
Oct 30 20:39:52.475472 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.475450    5713 scope.go:122] "RemoveContainer" containerID="d38e224686999e89588a3fa2b45d1c29d2c94553d17e7e25e2e3251c0281ab4a"
Oct 30 20:39:52.475648 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:39:52.475633    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d38e224686999e89588a3fa2b45d1c29d2c94553d17e7e25e2e3251c0281ab4a\": container with ID starting with d38e224686999e89588a3fa2b45d1c29d2c94553d17e7e25e2e3251c0281ab4a not found: ID does not exist" containerID="d38e224686999e89588a3fa2b45d1c29d2c94553d17e7e25e2e3251c0281ab4a"
Oct 30 20:39:52.475685 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.475652    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d38e224686999e89588a3fa2b45d1c29d2c94553d17e7e25e2e3251c0281ab4a"} err="failed to get container status \"d38e224686999e89588a3fa2b45d1c29d2c94553d17e7e25e2e3251c0281ab4a\": rpc error: code = NotFound desc = could not find container \"d38e224686999e89588a3fa2b45d1c29d2c94553d17e7e25e2e3251c0281ab4a\": container with ID starting with d38e224686999e89588a3fa2b45d1c29d2c94553d17e7e25e2e3251c0281ab4a not found: ID does not exist"
Oct 30 20:39:52.475685 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.475666    5713 scope.go:122] "RemoveContainer" containerID="d6b9d3c29eb082eb14cbc0dd4664369c791f40d5a75aeeb98689299537f59f75"
Oct 30 20:39:52.475918 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:39:52.475903    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6b9d3c29eb082eb14cbc0dd4664369c791f40d5a75aeeb98689299537f59f75\": container with ID starting with d6b9d3c29eb082eb14cbc0dd4664369c791f40d5a75aeeb98689299537f59f75 not found: ID does not exist" containerID="d6b9d3c29eb082eb14cbc0dd4664369c791f40d5a75aeeb98689299537f59f75"
Oct 30 20:39:52.475979 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.475921    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6b9d3c29eb082eb14cbc0dd4664369c791f40d5a75aeeb98689299537f59f75"} err="failed to get container status \"d6b9d3c29eb082eb14cbc0dd4664369c791f40d5a75aeeb98689299537f59f75\": rpc error: code = NotFound desc = could not find container \"d6b9d3c29eb082eb14cbc0dd4664369c791f40d5a75aeeb98689299537f59f75\": container with ID starting with d6b9d3c29eb082eb14cbc0dd4664369c791f40d5a75aeeb98689299537f59f75 not found: ID does not exist"
Oct 30 20:39:52.475979 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.475933    5713 scope.go:122] "RemoveContainer" containerID="d140b08b6210f2b3667f398b74f0086088403d96b90c4b000f43ea224ac6b458"
Oct 30 20:39:52.476180 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.476162    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d140b08b6210f2b3667f398b74f0086088403d96b90c4b000f43ea224ac6b458"} err="failed to get container status \"d140b08b6210f2b3667f398b74f0086088403d96b90c4b000f43ea224ac6b458\": rpc error: code = NotFound desc = could not find container \"d140b08b6210f2b3667f398b74f0086088403d96b90c4b000f43ea224ac6b458\": container with ID starting with d140b08b6210f2b3667f398b74f0086088403d96b90c4b000f43ea224ac6b458 not found: ID does not exist"
Oct 30 20:39:52.476227 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.476181    5713 scope.go:122] "RemoveContainer" containerID="bd8e64fb9dc49d675e305dfe67dd2ac11c07e5014c92311e685bc19270a55b8b"
Oct 30 20:39:52.476400 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.476385    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd8e64fb9dc49d675e305dfe67dd2ac11c07e5014c92311e685bc19270a55b8b"} err="failed to get container status \"bd8e64fb9dc49d675e305dfe67dd2ac11c07e5014c92311e685bc19270a55b8b\": rpc error: code = NotFound desc = could not find container \"bd8e64fb9dc49d675e305dfe67dd2ac11c07e5014c92311e685bc19270a55b8b\": container with ID starting with bd8e64fb9dc49d675e305dfe67dd2ac11c07e5014c92311e685bc19270a55b8b not found: ID does not exist"
Oct 30 20:39:52.476400 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.476399    5713 scope.go:122] "RemoveContainer" containerID="a70e118fb15f6bb5edca24ed3dfeb614808b8b6100704b9dbdfb9be04e21f3a7"
Oct 30 20:39:52.476611 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.476595    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a70e118fb15f6bb5edca24ed3dfeb614808b8b6100704b9dbdfb9be04e21f3a7"} err="failed to get container status \"a70e118fb15f6bb5edca24ed3dfeb614808b8b6100704b9dbdfb9be04e21f3a7\": rpc error: code = NotFound desc = could not find container \"a70e118fb15f6bb5edca24ed3dfeb614808b8b6100704b9dbdfb9be04e21f3a7\": container with ID starting with a70e118fb15f6bb5edca24ed3dfeb614808b8b6100704b9dbdfb9be04e21f3a7 not found: ID does not exist"
Oct 30 20:39:52.476662 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.476611    5713 scope.go:122] "RemoveContainer" containerID="f27e0161f269a31de0d92e4b6943ef3c333705a2895885f8d95a3aec9b5a248a"
Oct 30 20:39:52.476836 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.476819    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f27e0161f269a31de0d92e4b6943ef3c333705a2895885f8d95a3aec9b5a248a"} err="failed to get container status \"f27e0161f269a31de0d92e4b6943ef3c333705a2895885f8d95a3aec9b5a248a\": rpc error: code = NotFound desc = could not find container \"f27e0161f269a31de0d92e4b6943ef3c333705a2895885f8d95a3aec9b5a248a\": container with ID starting with f27e0161f269a31de0d92e4b6943ef3c333705a2895885f8d95a3aec9b5a248a not found: ID does not exist"
Oct 30 20:39:52.476898 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.476836    5713 scope.go:122] "RemoveContainer" containerID="c7296d272b2a7cdb3a06dff55dcf6811d03f31db5bfc6882918389bfb6e3b6cd"
Oct 30 20:39:52.477055 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.477039    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7296d272b2a7cdb3a06dff55dcf6811d03f31db5bfc6882918389bfb6e3b6cd"} err="failed to get container status \"c7296d272b2a7cdb3a06dff55dcf6811d03f31db5bfc6882918389bfb6e3b6cd\": rpc error: code = NotFound desc = could not find container \"c7296d272b2a7cdb3a06dff55dcf6811d03f31db5bfc6882918389bfb6e3b6cd\": container with ID starting with c7296d272b2a7cdb3a06dff55dcf6811d03f31db5bfc6882918389bfb6e3b6cd not found: ID does not exist"
Oct 30 20:39:52.477099 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.477055    5713 scope.go:122] "RemoveContainer" containerID="d38e224686999e89588a3fa2b45d1c29d2c94553d17e7e25e2e3251c0281ab4a"
Oct 30 20:39:52.477256 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.477240    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d38e224686999e89588a3fa2b45d1c29d2c94553d17e7e25e2e3251c0281ab4a"} err="failed to get container status \"d38e224686999e89588a3fa2b45d1c29d2c94553d17e7e25e2e3251c0281ab4a\": rpc error: code = NotFound desc = could not find container \"d38e224686999e89588a3fa2b45d1c29d2c94553d17e7e25e2e3251c0281ab4a\": container with ID starting with d38e224686999e89588a3fa2b45d1c29d2c94553d17e7e25e2e3251c0281ab4a not found: ID does not exist"
Oct 30 20:39:52.477303 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.477256    5713 scope.go:122] "RemoveContainer" containerID="d6b9d3c29eb082eb14cbc0dd4664369c791f40d5a75aeeb98689299537f59f75"
Oct 30 20:39:52.477459 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.477441    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6b9d3c29eb082eb14cbc0dd4664369c791f40d5a75aeeb98689299537f59f75"} err="failed to get container status \"d6b9d3c29eb082eb14cbc0dd4664369c791f40d5a75aeeb98689299537f59f75\": rpc error: code = NotFound desc = could not find container \"d6b9d3c29eb082eb14cbc0dd4664369c791f40d5a75aeeb98689299537f59f75\": container with ID starting with d6b9d3c29eb082eb14cbc0dd4664369c791f40d5a75aeeb98689299537f59f75 not found: ID does not exist"
Oct 30 20:39:52.528154 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.528126    5713 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:39:52.528154 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.528155    5713 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-web-config\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:39:52.528266 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.528165    5713 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:39:52.528266 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.528176    5713 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:39:52.528266 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.528186    5713 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-config-volume\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:39:52.528266 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.528195    5713 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-metrics-client-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:39:52.528266 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.528204    5713 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-secret-alertmanager-main-tls\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:39:52.528266 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.528213    5713 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:39:52.528266 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.528222    5713 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-cluster-tls-config\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:39:52.528266 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.528231    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xq44\" (UniqueName: \"kubernetes.io/projected/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-kube-api-access-8xq44\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:39:52.528266 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.528240    5713 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-config-out\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:39:52.528266 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.528248    5713 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-tls-assets\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:39:52.528266 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.528256    5713 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db-alertmanager-main-db\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:39:52.845159 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.845119    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 20:39:52.851119 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.851092    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 20:39:52.882248 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.882218    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 20:39:52.882498 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.882484    5713 cpu_manager.go:414] "containerMap: removing container" podUID="1c4609ce-58a7-47f3-8eaa-0f19cea5c6db" containerName="kube-rbac-proxy-metric"
Oct 30 20:39:52.882565 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.882499    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c4609ce-58a7-47f3-8eaa-0f19cea5c6db" containerName="kube-rbac-proxy-metric"
Oct 30 20:39:52.882565 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.882509    5713 cpu_manager.go:414] "containerMap: removing container" podUID="1c4609ce-58a7-47f3-8eaa-0f19cea5c6db" containerName="kube-rbac-proxy"
Oct 30 20:39:52.882565 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.882514    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c4609ce-58a7-47f3-8eaa-0f19cea5c6db" containerName="kube-rbac-proxy"
Oct 30 20:39:52.882565 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.882521    5713 cpu_manager.go:414] "containerMap: removing container" podUID="1c4609ce-58a7-47f3-8eaa-0f19cea5c6db" containerName="config-reloader"
Oct 30 20:39:52.882565 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.882527    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c4609ce-58a7-47f3-8eaa-0f19cea5c6db" containerName="config-reloader"
Oct 30 20:39:52.882565 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.882545    5713 cpu_manager.go:414] "containerMap: removing container" podUID="1c4609ce-58a7-47f3-8eaa-0f19cea5c6db" containerName="prom-label-proxy"
Oct 30 20:39:52.882565 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.882550    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c4609ce-58a7-47f3-8eaa-0f19cea5c6db" containerName="prom-label-proxy"
Oct 30 20:39:52.882565 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.882557    5713 cpu_manager.go:414] "containerMap: removing container" podUID="1c4609ce-58a7-47f3-8eaa-0f19cea5c6db" containerName="init-config-reloader"
Oct 30 20:39:52.882565 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.882563    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c4609ce-58a7-47f3-8eaa-0f19cea5c6db" containerName="init-config-reloader"
Oct 30 20:39:52.882565 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.882572    5713 cpu_manager.go:414] "containerMap: removing container" podUID="1c4609ce-58a7-47f3-8eaa-0f19cea5c6db" containerName="kube-rbac-proxy-web"
Oct 30 20:39:52.883099 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.882576    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c4609ce-58a7-47f3-8eaa-0f19cea5c6db" containerName="kube-rbac-proxy-web"
Oct 30 20:39:52.883099 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.882590    5713 cpu_manager.go:414] "containerMap: removing container" podUID="1c4609ce-58a7-47f3-8eaa-0f19cea5c6db" containerName="alertmanager"
Oct 30 20:39:52.883099 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.882594    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c4609ce-58a7-47f3-8eaa-0f19cea5c6db" containerName="alertmanager"
Oct 30 20:39:52.883099 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.882654    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c4609ce-58a7-47f3-8eaa-0f19cea5c6db" containerName="kube-rbac-proxy-metric"
Oct 30 20:39:52.883099 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.882661    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c4609ce-58a7-47f3-8eaa-0f19cea5c6db" containerName="kube-rbac-proxy"
Oct 30 20:39:52.883099 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.882668    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c4609ce-58a7-47f3-8eaa-0f19cea5c6db" containerName="config-reloader"
Oct 30 20:39:52.883099 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.882674    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c4609ce-58a7-47f3-8eaa-0f19cea5c6db" containerName="prom-label-proxy"
Oct 30 20:39:52.883099 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.882679    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c4609ce-58a7-47f3-8eaa-0f19cea5c6db" containerName="init-config-reloader"
Oct 30 20:39:52.883099 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.882685    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c4609ce-58a7-47f3-8eaa-0f19cea5c6db" containerName="kube-rbac-proxy-web"
Oct 30 20:39:52.883099 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.882694    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c4609ce-58a7-47f3-8eaa-0f19cea5c6db" containerName="alertmanager"
Oct 30 20:39:52.886186 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.886168    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:39:52.942943 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.942919    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 20:39:52.943195 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.943181    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 20:39:52.943274 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.943208    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 20:39:52.943335 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.943320    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 20:39:52.943387 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.943334    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-pqmk2\""
Oct 30 20:39:52.943435 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.943392    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 20:39:52.943435 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.943401    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 20:39:52.943505 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.943447    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 20:39:52.943560 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.943532    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 20:39:52.992979 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.992909    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 20:39:52.993619 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:52.993601    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 20:39:53.032237 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:53.032208    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/80d8f079-dd81-4840-9fdc-3a3e056db0a4-config-volume\") pod \"alertmanager-main-0\" (UID: \"80d8f079-dd81-4840-9fdc-3a3e056db0a4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:39:53.032237 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:53.032236    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/80d8f079-dd81-4840-9fdc-3a3e056db0a4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"80d8f079-dd81-4840-9fdc-3a3e056db0a4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:39:53.032415 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:53.032257    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/80d8f079-dd81-4840-9fdc-3a3e056db0a4-config-out\") pod \"alertmanager-main-0\" (UID: \"80d8f079-dd81-4840-9fdc-3a3e056db0a4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:39:53.032415 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:53.032279    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/80d8f079-dd81-4840-9fdc-3a3e056db0a4-web-config\") pod \"alertmanager-main-0\" (UID: \"80d8f079-dd81-4840-9fdc-3a3e056db0a4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:39:53.032415 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:53.032396    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtlvn\" (UniqueName: \"kubernetes.io/projected/80d8f079-dd81-4840-9fdc-3a3e056db0a4-kube-api-access-mtlvn\") pod \"alertmanager-main-0\" (UID: \"80d8f079-dd81-4840-9fdc-3a3e056db0a4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:39:53.032567 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:53.032438    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/80d8f079-dd81-4840-9fdc-3a3e056db0a4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"80d8f079-dd81-4840-9fdc-3a3e056db0a4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:39:53.032567 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:53.032502    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/80d8f079-dd81-4840-9fdc-3a3e056db0a4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"80d8f079-dd81-4840-9fdc-3a3e056db0a4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:39:53.032567 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:53.032527    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/80d8f079-dd81-4840-9fdc-3a3e056db0a4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"80d8f079-dd81-4840-9fdc-3a3e056db0a4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:39:53.032567 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:53.032547    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/80d8f079-dd81-4840-9fdc-3a3e056db0a4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"80d8f079-dd81-4840-9fdc-3a3e056db0a4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:39:53.032704 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:53.032598    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/80d8f079-dd81-4840-9fdc-3a3e056db0a4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"80d8f079-dd81-4840-9fdc-3a3e056db0a4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:39:53.032704 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:53.032622    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/80d8f079-dd81-4840-9fdc-3a3e056db0a4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"80d8f079-dd81-4840-9fdc-3a3e056db0a4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:39:53.032704 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:53.032661    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/80d8f079-dd81-4840-9fdc-3a3e056db0a4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"80d8f079-dd81-4840-9fdc-3a3e056db0a4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:39:53.032704 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:53.032686    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80d8f079-dd81-4840-9fdc-3a3e056db0a4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"80d8f079-dd81-4840-9fdc-3a3e056db0a4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:39:53.060525 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:53.060500    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c4609ce-58a7-47f3-8eaa-0f19cea5c6db" path="/var/lib/kubelet/pods/1c4609ce-58a7-47f3-8eaa-0f19cea5c6db/volumes"
Oct 30 20:39:53.133674 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:53.133641    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/80d8f079-dd81-4840-9fdc-3a3e056db0a4-config-volume\") pod \"alertmanager-main-0\" (UID: \"80d8f079-dd81-4840-9fdc-3a3e056db0a4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:39:53.133853 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:53.133678    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/80d8f079-dd81-4840-9fdc-3a3e056db0a4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"80d8f079-dd81-4840-9fdc-3a3e056db0a4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:39:53.133853 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:53.133708    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/80d8f079-dd81-4840-9fdc-3a3e056db0a4-config-out\") pod \"alertmanager-main-0\" (UID: \"80d8f079-dd81-4840-9fdc-3a3e056db0a4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:39:53.133853 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:53.133741    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/80d8f079-dd81-4840-9fdc-3a3e056db0a4-web-config\") pod \"alertmanager-main-0\" (UID: \"80d8f079-dd81-4840-9fdc-3a3e056db0a4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:39:53.133853 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:53.133764    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtlvn\" (UniqueName: \"kubernetes.io/projected/80d8f079-dd81-4840-9fdc-3a3e056db0a4-kube-api-access-mtlvn\") pod \"alertmanager-main-0\" (UID: \"80d8f079-dd81-4840-9fdc-3a3e056db0a4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:39:53.133853 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:53.133808    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/80d8f079-dd81-4840-9fdc-3a3e056db0a4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"80d8f079-dd81-4840-9fdc-3a3e056db0a4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:39:53.133853 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:53.133845    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/80d8f079-dd81-4840-9fdc-3a3e056db0a4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"80d8f079-dd81-4840-9fdc-3a3e056db0a4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:39:53.134173 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:53.133867    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/80d8f079-dd81-4840-9fdc-3a3e056db0a4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"80d8f079-dd81-4840-9fdc-3a3e056db0a4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:39:53.134173 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:53.133892    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/80d8f079-dd81-4840-9fdc-3a3e056db0a4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"80d8f079-dd81-4840-9fdc-3a3e056db0a4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:39:53.134173 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:53.133945    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/80d8f079-dd81-4840-9fdc-3a3e056db0a4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"80d8f079-dd81-4840-9fdc-3a3e056db0a4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:39:53.134173 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:53.133972    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/80d8f079-dd81-4840-9fdc-3a3e056db0a4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"80d8f079-dd81-4840-9fdc-3a3e056db0a4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:39:53.134173 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:53.133998    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/80d8f079-dd81-4840-9fdc-3a3e056db0a4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"80d8f079-dd81-4840-9fdc-3a3e056db0a4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:39:53.134173 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:53.134023    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80d8f079-dd81-4840-9fdc-3a3e056db0a4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"80d8f079-dd81-4840-9fdc-3a3e056db0a4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:39:53.134173 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:53.134118    5713 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 20:39:53.134509 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:53.134460    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/80d8f079-dd81-4840-9fdc-3a3e056db0a4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"80d8f079-dd81-4840-9fdc-3a3e056db0a4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:39:53.135238 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:53.135207    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/80d8f079-dd81-4840-9fdc-3a3e056db0a4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"80d8f079-dd81-4840-9fdc-3a3e056db0a4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:39:53.135366 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:53.135260    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80d8f079-dd81-4840-9fdc-3a3e056db0a4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"80d8f079-dd81-4840-9fdc-3a3e056db0a4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:39:53.137250 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:53.137199    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/80d8f079-dd81-4840-9fdc-3a3e056db0a4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"80d8f079-dd81-4840-9fdc-3a3e056db0a4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:39:53.137250 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:53.137213    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/80d8f079-dd81-4840-9fdc-3a3e056db0a4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"80d8f079-dd81-4840-9fdc-3a3e056db0a4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:39:53.137421 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:53.137403    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/80d8f079-dd81-4840-9fdc-3a3e056db0a4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"80d8f079-dd81-4840-9fdc-3a3e056db0a4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:39:53.137486 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:53.137408    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/80d8f079-dd81-4840-9fdc-3a3e056db0a4-config-volume\") pod \"alertmanager-main-0\" (UID: \"80d8f079-dd81-4840-9fdc-3a3e056db0a4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:39:53.137601 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:53.137581    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/80d8f079-dd81-4840-9fdc-3a3e056db0a4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"80d8f079-dd81-4840-9fdc-3a3e056db0a4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:39:53.137761 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:53.137741    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/80d8f079-dd81-4840-9fdc-3a3e056db0a4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"80d8f079-dd81-4840-9fdc-3a3e056db0a4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:39:53.137842 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:53.137828    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/80d8f079-dd81-4840-9fdc-3a3e056db0a4-web-config\") pod \"alertmanager-main-0\" (UID: \"80d8f079-dd81-4840-9fdc-3a3e056db0a4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:39:53.138133 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:53.138120    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/80d8f079-dd81-4840-9fdc-3a3e056db0a4-config-out\") pod \"alertmanager-main-0\" (UID: \"80d8f079-dd81-4840-9fdc-3a3e056db0a4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:39:53.138779 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:53.138764    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/80d8f079-dd81-4840-9fdc-3a3e056db0a4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"80d8f079-dd81-4840-9fdc-3a3e056db0a4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:39:53.192156 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:53.192123    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtlvn\" (UniqueName: \"kubernetes.io/projected/80d8f079-dd81-4840-9fdc-3a3e056db0a4-kube-api-access-mtlvn\") pod \"alertmanager-main-0\" (UID: \"80d8f079-dd81-4840-9fdc-3a3e056db0a4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:39:53.207044 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:53.207024    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 20:39:53.631305 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:53.631223    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 20:39:53.635182 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:39:53.635151    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80d8f079_dd81_4840_9fdc_3a3e056db0a4.slice/crio-1f21146b1ec3f65ce4d1dc2c67d66d75c33e0eea54c9318784b5dd725d586921 WatchSource:0}: Error finding container 1f21146b1ec3f65ce4d1dc2c67d66d75c33e0eea54c9318784b5dd725d586921: Status 404 returned error can't find the container with id 1f21146b1ec3f65ce4d1dc2c67d66d75c33e0eea54c9318784b5dd725d586921
Oct 30 20:39:54.431871 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:54.431840    5713 generic.go:356] "Generic (PLEG): container finished" podID="80d8f079-dd81-4840-9fdc-3a3e056db0a4" containerID="76e850761bc55bb9fb565e379788f5ee4b4ee2581ed0d33af825af2de48bf129" exitCode=0
Oct 30 20:39:54.432036 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:54.431887    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"80d8f079-dd81-4840-9fdc-3a3e056db0a4","Type":"ContainerDied","Data":"76e850761bc55bb9fb565e379788f5ee4b4ee2581ed0d33af825af2de48bf129"}
Oct 30 20:39:54.432036 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:54.431937    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"80d8f079-dd81-4840-9fdc-3a3e056db0a4","Type":"ContainerStarted","Data":"1f21146b1ec3f65ce4d1dc2c67d66d75c33e0eea54c9318784b5dd725d586921"}
Oct 30 20:39:54.459591 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:39:54.459561    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 20:39:54.824896 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:54.824861    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-744b77f955-tzvzl"]
Oct 30 20:39:54.827102 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:54.827086    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-744b77f955-tzvzl"
Oct 30 20:39:54.879301 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:54.879272    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Oct 30 20:39:54.880115 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:54.880089    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Oct 30 20:39:54.880250 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:54.880189    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-gkps8\""
Oct 30 20:39:54.880250 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:54.880202    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Oct 30 20:39:54.881106 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:54.881093    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Oct 30 20:39:54.881147 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:54.881105    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Oct 30 20:39:54.931107 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:54.931076    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Oct 30 20:39:54.932337 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:54.932313    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-744b77f955-tzvzl"]
Oct 30 20:39:54.948630 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:54.948556    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f94cda1e-ed8b-4b22-9620-a0c3cf1af271-secret-telemeter-client\") pod \"telemeter-client-744b77f955-tzvzl\" (UID: \"f94cda1e-ed8b-4b22-9620-a0c3cf1af271\") " pod="openshift-monitoring/telemeter-client-744b77f955-tzvzl"
Oct 30 20:39:54.948630 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:54.948601    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2mtg\" (UniqueName: \"kubernetes.io/projected/f94cda1e-ed8b-4b22-9620-a0c3cf1af271-kube-api-access-d2mtg\") pod \"telemeter-client-744b77f955-tzvzl\" (UID: \"f94cda1e-ed8b-4b22-9620-a0c3cf1af271\") " pod="openshift-monitoring/telemeter-client-744b77f955-tzvzl"
Oct 30 20:39:54.948826 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:54.948629    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f94cda1e-ed8b-4b22-9620-a0c3cf1af271-serving-certs-ca-bundle\") pod \"telemeter-client-744b77f955-tzvzl\" (UID: \"f94cda1e-ed8b-4b22-9620-a0c3cf1af271\") " pod="openshift-monitoring/telemeter-client-744b77f955-tzvzl"
Oct 30 20:39:54.948826 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:54.948682    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f94cda1e-ed8b-4b22-9620-a0c3cf1af271-metrics-client-ca\") pod \"telemeter-client-744b77f955-tzvzl\" (UID: \"f94cda1e-ed8b-4b22-9620-a0c3cf1af271\") " pod="openshift-monitoring/telemeter-client-744b77f955-tzvzl"
Oct 30 20:39:54.948826 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:54.948774    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f94cda1e-ed8b-4b22-9620-a0c3cf1af271-telemeter-client-tls\") pod \"telemeter-client-744b77f955-tzvzl\" (UID: \"f94cda1e-ed8b-4b22-9620-a0c3cf1af271\") " pod="openshift-monitoring/telemeter-client-744b77f955-tzvzl"
Oct 30 20:39:54.948977 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:54.948872    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f94cda1e-ed8b-4b22-9620-a0c3cf1af271-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-744b77f955-tzvzl\" (UID: \"f94cda1e-ed8b-4b22-9620-a0c3cf1af271\") " pod="openshift-monitoring/telemeter-client-744b77f955-tzvzl"
Oct 30 20:39:54.948977 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:54.948933    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f94cda1e-ed8b-4b22-9620-a0c3cf1af271-federate-client-tls\") pod \"telemeter-client-744b77f955-tzvzl\" (UID: \"f94cda1e-ed8b-4b22-9620-a0c3cf1af271\") " pod="openshift-monitoring/telemeter-client-744b77f955-tzvzl"
Oct 30 20:39:54.949050 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:54.948978    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f94cda1e-ed8b-4b22-9620-a0c3cf1af271-telemeter-trusted-ca-bundle\") pod \"telemeter-client-744b77f955-tzvzl\" (UID: \"f94cda1e-ed8b-4b22-9620-a0c3cf1af271\") " pod="openshift-monitoring/telemeter-client-744b77f955-tzvzl"
Oct 30 20:39:55.049522 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:55.049489    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2mtg\" (UniqueName: \"kubernetes.io/projected/f94cda1e-ed8b-4b22-9620-a0c3cf1af271-kube-api-access-d2mtg\") pod \"telemeter-client-744b77f955-tzvzl\" (UID: \"f94cda1e-ed8b-4b22-9620-a0c3cf1af271\") " pod="openshift-monitoring/telemeter-client-744b77f955-tzvzl"
Oct 30 20:39:55.049522 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:55.049524    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f94cda1e-ed8b-4b22-9620-a0c3cf1af271-serving-certs-ca-bundle\") pod \"telemeter-client-744b77f955-tzvzl\" (UID: \"f94cda1e-ed8b-4b22-9620-a0c3cf1af271\") " pod="openshift-monitoring/telemeter-client-744b77f955-tzvzl"
Oct 30 20:39:55.049778 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:55.049542    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f94cda1e-ed8b-4b22-9620-a0c3cf1af271-metrics-client-ca\") pod \"telemeter-client-744b77f955-tzvzl\" (UID: \"f94cda1e-ed8b-4b22-9620-a0c3cf1af271\") " pod="openshift-monitoring/telemeter-client-744b77f955-tzvzl"
Oct 30 20:39:55.049778 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:55.049568    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f94cda1e-ed8b-4b22-9620-a0c3cf1af271-telemeter-client-tls\") pod \"telemeter-client-744b77f955-tzvzl\" (UID: \"f94cda1e-ed8b-4b22-9620-a0c3cf1af271\") " pod="openshift-monitoring/telemeter-client-744b77f955-tzvzl"
Oct 30 20:39:55.049778 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:55.049612    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f94cda1e-ed8b-4b22-9620-a0c3cf1af271-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-744b77f955-tzvzl\" (UID: \"f94cda1e-ed8b-4b22-9620-a0c3cf1af271\") " pod="openshift-monitoring/telemeter-client-744b77f955-tzvzl"
Oct 30 20:39:55.049778 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:55.049640    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f94cda1e-ed8b-4b22-9620-a0c3cf1af271-federate-client-tls\") pod \"telemeter-client-744b77f955-tzvzl\" (UID: \"f94cda1e-ed8b-4b22-9620-a0c3cf1af271\") " pod="openshift-monitoring/telemeter-client-744b77f955-tzvzl"
Oct 30 20:39:55.049778 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:55.049685    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f94cda1e-ed8b-4b22-9620-a0c3cf1af271-telemeter-trusted-ca-bundle\") pod \"telemeter-client-744b77f955-tzvzl\" (UID: \"f94cda1e-ed8b-4b22-9620-a0c3cf1af271\") " pod="openshift-monitoring/telemeter-client-744b77f955-tzvzl"
Oct 30 20:39:55.049778 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:55.049727    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f94cda1e-ed8b-4b22-9620-a0c3cf1af271-secret-telemeter-client\") pod \"telemeter-client-744b77f955-tzvzl\" (UID: \"f94cda1e-ed8b-4b22-9620-a0c3cf1af271\") " pod="openshift-monitoring/telemeter-client-744b77f955-tzvzl"
Oct 30 20:39:55.050569 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:55.050476    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f94cda1e-ed8b-4b22-9620-a0c3cf1af271-metrics-client-ca\") pod \"telemeter-client-744b77f955-tzvzl\" (UID: \"f94cda1e-ed8b-4b22-9620-a0c3cf1af271\") " pod="openshift-monitoring/telemeter-client-744b77f955-tzvzl"
Oct 30 20:39:55.050569 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:55.050456    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f94cda1e-ed8b-4b22-9620-a0c3cf1af271-serving-certs-ca-bundle\") pod \"telemeter-client-744b77f955-tzvzl\" (UID: \"f94cda1e-ed8b-4b22-9620-a0c3cf1af271\") " pod="openshift-monitoring/telemeter-client-744b77f955-tzvzl"
Oct 30 20:39:55.050808 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:55.050765    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f94cda1e-ed8b-4b22-9620-a0c3cf1af271-telemeter-trusted-ca-bundle\") pod \"telemeter-client-744b77f955-tzvzl\" (UID: \"f94cda1e-ed8b-4b22-9620-a0c3cf1af271\") " pod="openshift-monitoring/telemeter-client-744b77f955-tzvzl"
Oct 30 20:39:55.052484 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:55.052452    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f94cda1e-ed8b-4b22-9620-a0c3cf1af271-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-744b77f955-tzvzl\" (UID: \"f94cda1e-ed8b-4b22-9620-a0c3cf1af271\") " pod="openshift-monitoring/telemeter-client-744b77f955-tzvzl"
Oct 30 20:39:55.052565 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:55.052516    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f94cda1e-ed8b-4b22-9620-a0c3cf1af271-secret-telemeter-client\") pod \"telemeter-client-744b77f955-tzvzl\" (UID: \"f94cda1e-ed8b-4b22-9620-a0c3cf1af271\") " pod="openshift-monitoring/telemeter-client-744b77f955-tzvzl"
Oct 30 20:39:55.052601 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:55.052569    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f94cda1e-ed8b-4b22-9620-a0c3cf1af271-telemeter-client-tls\") pod \"telemeter-client-744b77f955-tzvzl\" (UID: \"f94cda1e-ed8b-4b22-9620-a0c3cf1af271\") " pod="openshift-monitoring/telemeter-client-744b77f955-tzvzl"
Oct 30 20:39:55.052718 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:55.052699    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f94cda1e-ed8b-4b22-9620-a0c3cf1af271-federate-client-tls\") pod \"telemeter-client-744b77f955-tzvzl\" (UID: \"f94cda1e-ed8b-4b22-9620-a0c3cf1af271\") " pod="openshift-monitoring/telemeter-client-744b77f955-tzvzl"
Oct 30 20:39:55.106716 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:55.106677    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2mtg\" (UniqueName: \"kubernetes.io/projected/f94cda1e-ed8b-4b22-9620-a0c3cf1af271-kube-api-access-d2mtg\") pod \"telemeter-client-744b77f955-tzvzl\" (UID: \"f94cda1e-ed8b-4b22-9620-a0c3cf1af271\") " pod="openshift-monitoring/telemeter-client-744b77f955-tzvzl"
Oct 30 20:39:55.138647 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:55.138622    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-744b77f955-tzvzl"
Oct 30 20:39:55.438800 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:55.438753    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"80d8f079-dd81-4840-9fdc-3a3e056db0a4","Type":"ContainerStarted","Data":"6043dea112fd22b643b1d792ff3d65848da3b09410200226c3606d655395709a"}
Oct 30 20:39:55.438995 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:55.438849    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"80d8f079-dd81-4840-9fdc-3a3e056db0a4","Type":"ContainerStarted","Data":"93243008da3678c73c3eb27c0f89d927395fcddb2a7678fdbe05b6a998b5e6c9"}
Oct 30 20:39:55.438995 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:55.438889    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"80d8f079-dd81-4840-9fdc-3a3e056db0a4","Type":"ContainerStarted","Data":"5d706ce90d5b3e6c702ca8b076119bf3e8b49dde3caf3c9aac5379e5ce44a73c"}
Oct 30 20:39:55.438995 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:55.438932    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"80d8f079-dd81-4840-9fdc-3a3e056db0a4","Type":"ContainerStarted","Data":"515c3d838fb990d93e2d6dbfad4cc968afa4c9c8daa794086bcce1faa6affbb2"}
Oct 30 20:39:55.438995 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:55.438965    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"80d8f079-dd81-4840-9fdc-3a3e056db0a4","Type":"ContainerStarted","Data":"3d254549eb54565c838a969b2f6cc734eb4bb344ec626c23f601afa5d39a5a13"}
Oct 30 20:39:55.439210 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:55.439006    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"80d8f079-dd81-4840-9fdc-3a3e056db0a4","Type":"ContainerStarted","Data":"3f213a7ac0187912406ce88638e2f56ac822a5abaaa44fd7b03b74f9a38f9fe8"}
Oct 30 20:39:55.560165 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:55.560105    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.546486689 podStartE2EDuration="3.56008954s" totalImagesPullingTime="0s" totalInitContainerRuntime="13.602851ms" isStatefulPod=false podCreationTimestamp="2026-10-30 20:39:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:39:55.559094674 +0000 UTC m=+78.623871094" watchObservedRunningTime="2026-10-30 20:39:55.56008954 +0000 UTC m=+78.624865932"
Oct 30 20:39:55.630837 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:55.630804    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-744b77f955-tzvzl"]
Oct 30 20:39:55.635327 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:39:55.634729    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf94cda1e_ed8b_4b22_9620_a0c3cf1af271.slice/crio-a72fdd511982122c959333d5abbb8964eadbfb0fb69d3e026657c9c9670f6f2c WatchSource:0}: Error finding container a72fdd511982122c959333d5abbb8964eadbfb0fb69d3e026657c9c9670f6f2c: Status 404 returned error can't find the container with id a72fdd511982122c959333d5abbb8964eadbfb0fb69d3e026657c9c9670f6f2c
Oct 30 20:39:55.637081 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:55.637061    5713 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:39:56.446580 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:56.446543    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-744b77f955-tzvzl" event={"ID":"f94cda1e-ed8b-4b22-9620-a0c3cf1af271","Type":"ContainerStarted","Data":"a72fdd511982122c959333d5abbb8964eadbfb0fb69d3e026657c9c9670f6f2c"}
Oct 30 20:39:56.796519 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:56.796449    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 20:39:56.796774 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:56.796742    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" containerName="prometheus" containerID="cri-o://cea18e6abdfde296bb779d83752561bccff0c9547b9829446ca54e81b9c71bad" gracePeriod=600
Oct 30 20:39:56.796912 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:56.796767    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" containerName="thanos-sidecar" containerID="cri-o://b2305237333dd77b38d54b1bf6d4e64a02247bcb4e8d0baf57949b5b2a0b8893" gracePeriod=600
Oct 30 20:39:56.796912 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:56.796821    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" containerName="kube-rbac-proxy-thanos" containerID="cri-o://bada155176449de4f95a9134cc2bcf9d9a6e49153aa34d6b19d69a007ec22a66" gracePeriod=600
Oct 30 20:39:56.796912 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:56.796805    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" containerName="kube-rbac-proxy-web" containerID="cri-o://9d4a60f970f9c790fb7a15f66d0c939c5cfe097422fc7390b667543e90ec02d7" gracePeriod=600
Oct 30 20:39:56.796912 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:56.796839    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" containerName="config-reloader" containerID="cri-o://5d9bc8b61136c18c540be963353fc2c21014d8179a9d7d563d8b939d136c8ce5" gracePeriod=600
Oct 30 20:39:56.796912 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:56.796760    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" containerName="kube-rbac-proxy" containerID="cri-o://ac90672f57cdaaed8c17ddf63fec27de5e7299ed8d999f071519583ed0c0229b" gracePeriod=600
Oct 30 20:39:57.069181 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.069154    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:39:57.169037 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.169000    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-configmap-serving-certs-ca-bundle\") pod \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") "
Oct 30 20:39:57.169226 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.169060    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-secret-kube-rbac-proxy\") pod \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") "
Oct 30 20:39:57.169226 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.169094    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-secret-prometheus-k8s-tls\") pod \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") "
Oct 30 20:39:57.169226 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.169132    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-prometheus-k8s-rulefiles-1\") pod \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") "
Oct 30 20:39:57.169226 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.169162    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-secret-grpc-tls\") pod \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") "
Oct 30 20:39:57.169226 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.169196    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-secret-metrics-client-certs\") pod \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") "
Oct 30 20:39:57.169434 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.169240    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-web-config\" (UniqueName: \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-web-config\") pod \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") "
Oct 30 20:39:57.169434 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.169286    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-prometheus-trusted-ca-bundle\") pod \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") "
Oct 30 20:39:57.169434 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.169340    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-kube-api-access-rx4tv\" (UniqueName: \"kubernetes.io/projected/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-kube-api-access-rx4tv\") pod \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") "
Oct 30 20:39:57.169434 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.169371    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-config\" (UniqueName: \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-config\") pod \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") "
Oct 30 20:39:57.169434 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.169410    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-prometheus-k8s-rulefiles-0\") pod \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") "
Oct 30 20:39:57.169612 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.169438    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-tls-assets\" (UniqueName: \"kubernetes.io/projected/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-tls-assets\") pod \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") "
Oct 30 20:39:57.169612 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.169445    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-configmap-serving-certs-ca-bundle" pod "bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" (UID: "bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:39:57.169612 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.169468    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") "
Oct 30 20:39:57.169612 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.169529    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-configmap-metrics-client-ca\") pod \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") "
Oct 30 20:39:57.169612 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.169571    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-prometheus-k8s-db\") pod \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") "
Oct 30 20:39:57.169612 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.169603    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-thanos-prometheus-http-client-file\") pod \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") "
Oct 30 20:39:57.169927 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.169640    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-config-out\" (UniqueName: \"kubernetes.io/empty-dir/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-config-out\") pod \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") "
Oct 30 20:39:57.169927 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.169826    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") "
Oct 30 20:39:57.169927 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.169871    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-prometheus-k8s-rulefiles-2\") pod \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") "
Oct 30 20:39:57.169927 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.169902    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-configmap-kubelet-serving-ca-bundle\") pod \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\" (UID: \"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df\") "
Oct 30 20:39:57.171213 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.170166    5713 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:39:57.171213 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.169897    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-prometheus-k8s-rulefiles-1" pod "bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" (UID: "bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:39:57.171367 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.171211    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-prometheus-trusted-ca-bundle" pod "bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" (UID: "bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:39:57.171367 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.171342    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-configmap-metrics-client-ca" pod "bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" (UID: "bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:39:57.172867 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.172622    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-secret-prometheus-k8s-tls" pod "bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" (UID: "bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:39:57.173402 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.173293    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-prometheus-k8s-rulefiles-2" pod "bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" (UID: "bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:39:57.173691 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.173491    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-prometheus-k8s-db" pod "bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" (UID: "bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:39:57.174063 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.173807    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-configmap-kubelet-serving-ca-bundle" pod "bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" (UID: "bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:39:57.174063 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.174022    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-prometheus-k8s-rulefiles-0" pod "bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" (UID: "bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:39:57.174211 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.174123    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-thanos-prometheus-http-client-file" pod "bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" (UID: "bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:39:57.174300 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.174280    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-kube-api-access-rx4tv" pod "bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" (UID: "bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df"). InnerVolumeSpecName "kube-api-access-rx4tv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:39:57.175003 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.174979    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-secret-kube-rbac-proxy" pod "bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" (UID: "bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:39:57.175146 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.175111    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-config" pod "bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" (UID: "bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:39:57.175251 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.175186    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-secret-grpc-tls" pod "bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" (UID: "bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:39:57.175303 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.175275    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-secret-prometheus-k8s-kube-rbac-proxy-web" pod "bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" (UID: "bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:39:57.175361 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.175345    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-secret-metrics-client-certs" pod "bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" (UID: "bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:39:57.175719 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.175697    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-tls-assets" pod "bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" (UID: "bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:39:57.175773 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.175758    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-secret-prometheus-k8s-thanos-sidecar-tls" pod "bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" (UID: "bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:39:57.176036 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.176021    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-config-out" pod "bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" (UID: "bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:39:57.184266 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.184245    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-web-config" pod "bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" (UID: "bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:39:57.271069 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.271033    5713 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-secret-metrics-client-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:39:57.271069 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.271070    5713 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-web-config\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:39:57.271241 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.271087    5713 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-prometheus-trusted-ca-bundle\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:39:57.271241 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.271102    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rx4tv\" (UniqueName: \"kubernetes.io/projected/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-kube-api-access-rx4tv\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:39:57.271241 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.271116    5713 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-config\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:39:57.271241 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.271131    5713 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:39:57.271241 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.271146    5713 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-tls-assets\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:39:57.271241 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.271161    5713 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:39:57.271241 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.271176    5713 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-configmap-metrics-client-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:39:57.271241 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.271191    5713 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-prometheus-k8s-db\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:39:57.271241 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.271207    5713 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-thanos-prometheus-http-client-file\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:39:57.271241 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.271222    5713 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-config-out\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:39:57.271241 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.271236    5713 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:39:57.271616 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.271251    5713 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:39:57.271616 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.271265    5713 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:39:57.271616 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.271280    5713 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-secret-kube-rbac-proxy\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:39:57.271616 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.271294    5713 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-secret-prometheus-k8s-tls\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:39:57.271616 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.271309    5713 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:39:57.271616 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.271323    5713 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df-secret-grpc-tls\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:39:57.454025 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.453993    5713 generic.go:356] "Generic (PLEG): container finished" podID="bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" containerID="bada155176449de4f95a9134cc2bcf9d9a6e49153aa34d6b19d69a007ec22a66" exitCode=0
Oct 30 20:39:57.454025 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.454018    5713 generic.go:356] "Generic (PLEG): container finished" podID="bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" containerID="ac90672f57cdaaed8c17ddf63fec27de5e7299ed8d999f071519583ed0c0229b" exitCode=0
Oct 30 20:39:57.454025 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.454024    5713 generic.go:356] "Generic (PLEG): container finished" podID="bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" containerID="9d4a60f970f9c790fb7a15f66d0c939c5cfe097422fc7390b667543e90ec02d7" exitCode=0
Oct 30 20:39:57.454025 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.454030    5713 generic.go:356] "Generic (PLEG): container finished" podID="bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" containerID="b2305237333dd77b38d54b1bf6d4e64a02247bcb4e8d0baf57949b5b2a0b8893" exitCode=0
Oct 30 20:39:57.454025 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.454037    5713 generic.go:356] "Generic (PLEG): container finished" podID="bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" containerID="5d9bc8b61136c18c540be963353fc2c21014d8179a9d7d563d8b939d136c8ce5" exitCode=0
Oct 30 20:39:57.454584 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.454044    5713 generic.go:356] "Generic (PLEG): container finished" podID="bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" containerID="cea18e6abdfde296bb779d83752561bccff0c9547b9829446ca54e81b9c71bad" exitCode=0
Oct 30 20:39:57.454584 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.454068    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:39:57.454584 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.454085    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df","Type":"ContainerDied","Data":"bada155176449de4f95a9134cc2bcf9d9a6e49153aa34d6b19d69a007ec22a66"}
Oct 30 20:39:57.454584 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.454147    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df","Type":"ContainerDied","Data":"ac90672f57cdaaed8c17ddf63fec27de5e7299ed8d999f071519583ed0c0229b"}
Oct 30 20:39:57.454584 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.454175    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df","Type":"ContainerDied","Data":"9d4a60f970f9c790fb7a15f66d0c939c5cfe097422fc7390b667543e90ec02d7"}
Oct 30 20:39:57.454584 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.454200    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df","Type":"ContainerDied","Data":"b2305237333dd77b38d54b1bf6d4e64a02247bcb4e8d0baf57949b5b2a0b8893"}
Oct 30 20:39:57.454584 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.454243    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df","Type":"ContainerDied","Data":"5d9bc8b61136c18c540be963353fc2c21014d8179a9d7d563d8b939d136c8ce5"}
Oct 30 20:39:57.454584 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.454267    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df","Type":"ContainerDied","Data":"cea18e6abdfde296bb779d83752561bccff0c9547b9829446ca54e81b9c71bad"}
Oct 30 20:39:57.454584 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.454214    5713 scope.go:122] "RemoveContainer" containerID="bada155176449de4f95a9134cc2bcf9d9a6e49153aa34d6b19d69a007ec22a66"
Oct 30 20:39:57.454584 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.454289    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df","Type":"ContainerDied","Data":"ef111d4318a24f1b18f799bb083297e58132a20086bb28aedf2679e9b3dc1913"}
Oct 30 20:39:57.462736 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.462717    5713 scope.go:122] "RemoveContainer" containerID="ac90672f57cdaaed8c17ddf63fec27de5e7299ed8d999f071519583ed0c0229b"
Oct 30 20:39:57.470084 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.470063    5713 scope.go:122] "RemoveContainer" containerID="9d4a60f970f9c790fb7a15f66d0c939c5cfe097422fc7390b667543e90ec02d7"
Oct 30 20:39:57.477987 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.477967    5713 scope.go:122] "RemoveContainer" containerID="b2305237333dd77b38d54b1bf6d4e64a02247bcb4e8d0baf57949b5b2a0b8893"
Oct 30 20:39:57.485757 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.485734    5713 scope.go:122] "RemoveContainer" containerID="5d9bc8b61136c18c540be963353fc2c21014d8179a9d7d563d8b939d136c8ce5"
Oct 30 20:39:57.494476 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.494458    5713 scope.go:122] "RemoveContainer" containerID="cea18e6abdfde296bb779d83752561bccff0c9547b9829446ca54e81b9c71bad"
Oct 30 20:39:57.502236 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.502212    5713 scope.go:122] "RemoveContainer" containerID="521dd12d5bf7bf42e12d8d6d5ac01b10c9993cfd8c508bb4c92dea2fc5092310"
Oct 30 20:39:57.510990 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.510972    5713 scope.go:122] "RemoveContainer" containerID="bada155176449de4f95a9134cc2bcf9d9a6e49153aa34d6b19d69a007ec22a66"
Oct 30 20:39:57.511276 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:39:57.511258    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bada155176449de4f95a9134cc2bcf9d9a6e49153aa34d6b19d69a007ec22a66\": container with ID starting with bada155176449de4f95a9134cc2bcf9d9a6e49153aa34d6b19d69a007ec22a66 not found: ID does not exist" containerID="bada155176449de4f95a9134cc2bcf9d9a6e49153aa34d6b19d69a007ec22a66"
Oct 30 20:39:57.511329 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.511287    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bada155176449de4f95a9134cc2bcf9d9a6e49153aa34d6b19d69a007ec22a66"} err="failed to get container status \"bada155176449de4f95a9134cc2bcf9d9a6e49153aa34d6b19d69a007ec22a66\": rpc error: code = NotFound desc = could not find container \"bada155176449de4f95a9134cc2bcf9d9a6e49153aa34d6b19d69a007ec22a66\": container with ID starting with bada155176449de4f95a9134cc2bcf9d9a6e49153aa34d6b19d69a007ec22a66 not found: ID does not exist"
Oct 30 20:39:57.511329 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.511309    5713 scope.go:122] "RemoveContainer" containerID="ac90672f57cdaaed8c17ddf63fec27de5e7299ed8d999f071519583ed0c0229b"
Oct 30 20:39:57.511530 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:39:57.511513    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac90672f57cdaaed8c17ddf63fec27de5e7299ed8d999f071519583ed0c0229b\": container with ID starting with ac90672f57cdaaed8c17ddf63fec27de5e7299ed8d999f071519583ed0c0229b not found: ID does not exist" containerID="ac90672f57cdaaed8c17ddf63fec27de5e7299ed8d999f071519583ed0c0229b"
Oct 30 20:39:57.511568 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.511534    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac90672f57cdaaed8c17ddf63fec27de5e7299ed8d999f071519583ed0c0229b"} err="failed to get container status \"ac90672f57cdaaed8c17ddf63fec27de5e7299ed8d999f071519583ed0c0229b\": rpc error: code = NotFound desc = could not find container \"ac90672f57cdaaed8c17ddf63fec27de5e7299ed8d999f071519583ed0c0229b\": container with ID starting with ac90672f57cdaaed8c17ddf63fec27de5e7299ed8d999f071519583ed0c0229b not found: ID does not exist"
Oct 30 20:39:57.511568 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.511547    5713 scope.go:122] "RemoveContainer" containerID="9d4a60f970f9c790fb7a15f66d0c939c5cfe097422fc7390b667543e90ec02d7"
Oct 30 20:39:57.511763 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:39:57.511748    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d4a60f970f9c790fb7a15f66d0c939c5cfe097422fc7390b667543e90ec02d7\": container with ID starting with 9d4a60f970f9c790fb7a15f66d0c939c5cfe097422fc7390b667543e90ec02d7 not found: ID does not exist" containerID="9d4a60f970f9c790fb7a15f66d0c939c5cfe097422fc7390b667543e90ec02d7"
Oct 30 20:39:57.511826 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.511765    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d4a60f970f9c790fb7a15f66d0c939c5cfe097422fc7390b667543e90ec02d7"} err="failed to get container status \"9d4a60f970f9c790fb7a15f66d0c939c5cfe097422fc7390b667543e90ec02d7\": rpc error: code = NotFound desc = could not find container \"9d4a60f970f9c790fb7a15f66d0c939c5cfe097422fc7390b667543e90ec02d7\": container with ID starting with 9d4a60f970f9c790fb7a15f66d0c939c5cfe097422fc7390b667543e90ec02d7 not found: ID does not exist"
Oct 30 20:39:57.511826 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.511777    5713 scope.go:122] "RemoveContainer" containerID="b2305237333dd77b38d54b1bf6d4e64a02247bcb4e8d0baf57949b5b2a0b8893"
Oct 30 20:39:57.512029 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:39:57.512010    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2305237333dd77b38d54b1bf6d4e64a02247bcb4e8d0baf57949b5b2a0b8893\": container with ID starting with b2305237333dd77b38d54b1bf6d4e64a02247bcb4e8d0baf57949b5b2a0b8893 not found: ID does not exist" containerID="b2305237333dd77b38d54b1bf6d4e64a02247bcb4e8d0baf57949b5b2a0b8893"
Oct 30 20:39:57.512074 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.512038    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2305237333dd77b38d54b1bf6d4e64a02247bcb4e8d0baf57949b5b2a0b8893"} err="failed to get container status \"b2305237333dd77b38d54b1bf6d4e64a02247bcb4e8d0baf57949b5b2a0b8893\": rpc error: code = NotFound desc = could not find container \"b2305237333dd77b38d54b1bf6d4e64a02247bcb4e8d0baf57949b5b2a0b8893\": container with ID starting with b2305237333dd77b38d54b1bf6d4e64a02247bcb4e8d0baf57949b5b2a0b8893 not found: ID does not exist"
Oct 30 20:39:57.512074 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.512051    5713 scope.go:122] "RemoveContainer" containerID="5d9bc8b61136c18c540be963353fc2c21014d8179a9d7d563d8b939d136c8ce5"
Oct 30 20:39:57.512272 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:39:57.512251    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d9bc8b61136c18c540be963353fc2c21014d8179a9d7d563d8b939d136c8ce5\": container with ID starting with 5d9bc8b61136c18c540be963353fc2c21014d8179a9d7d563d8b939d136c8ce5 not found: ID does not exist" containerID="5d9bc8b61136c18c540be963353fc2c21014d8179a9d7d563d8b939d136c8ce5"
Oct 30 20:39:57.512350 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.512281    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d9bc8b61136c18c540be963353fc2c21014d8179a9d7d563d8b939d136c8ce5"} err="failed to get container status \"5d9bc8b61136c18c540be963353fc2c21014d8179a9d7d563d8b939d136c8ce5\": rpc error: code = NotFound desc = could not find container \"5d9bc8b61136c18c540be963353fc2c21014d8179a9d7d563d8b939d136c8ce5\": container with ID starting with 5d9bc8b61136c18c540be963353fc2c21014d8179a9d7d563d8b939d136c8ce5 not found: ID does not exist"
Oct 30 20:39:57.512350 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.512301    5713 scope.go:122] "RemoveContainer" containerID="cea18e6abdfde296bb779d83752561bccff0c9547b9829446ca54e81b9c71bad"
Oct 30 20:39:57.512621 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:39:57.512597    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cea18e6abdfde296bb779d83752561bccff0c9547b9829446ca54e81b9c71bad\": container with ID starting with cea18e6abdfde296bb779d83752561bccff0c9547b9829446ca54e81b9c71bad not found: ID does not exist" containerID="cea18e6abdfde296bb779d83752561bccff0c9547b9829446ca54e81b9c71bad"
Oct 30 20:39:57.512744 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.512626    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cea18e6abdfde296bb779d83752561bccff0c9547b9829446ca54e81b9c71bad"} err="failed to get container status \"cea18e6abdfde296bb779d83752561bccff0c9547b9829446ca54e81b9c71bad\": rpc error: code = NotFound desc = could not find container \"cea18e6abdfde296bb779d83752561bccff0c9547b9829446ca54e81b9c71bad\": container with ID starting with cea18e6abdfde296bb779d83752561bccff0c9547b9829446ca54e81b9c71bad not found: ID does not exist"
Oct 30 20:39:57.512744 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.512644    5713 scope.go:122] "RemoveContainer" containerID="521dd12d5bf7bf42e12d8d6d5ac01b10c9993cfd8c508bb4c92dea2fc5092310"
Oct 30 20:39:57.513120 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:39:57.513091    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"521dd12d5bf7bf42e12d8d6d5ac01b10c9993cfd8c508bb4c92dea2fc5092310\": container with ID starting with 521dd12d5bf7bf42e12d8d6d5ac01b10c9993cfd8c508bb4c92dea2fc5092310 not found: ID does not exist" containerID="521dd12d5bf7bf42e12d8d6d5ac01b10c9993cfd8c508bb4c92dea2fc5092310"
Oct 30 20:39:57.513184 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.513117    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"521dd12d5bf7bf42e12d8d6d5ac01b10c9993cfd8c508bb4c92dea2fc5092310"} err="failed to get container status \"521dd12d5bf7bf42e12d8d6d5ac01b10c9993cfd8c508bb4c92dea2fc5092310\": rpc error: code = NotFound desc = could not find container \"521dd12d5bf7bf42e12d8d6d5ac01b10c9993cfd8c508bb4c92dea2fc5092310\": container with ID starting with 521dd12d5bf7bf42e12d8d6d5ac01b10c9993cfd8c508bb4c92dea2fc5092310 not found: ID does not exist"
Oct 30 20:39:57.513184 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.513136    5713 scope.go:122] "RemoveContainer" containerID="bada155176449de4f95a9134cc2bcf9d9a6e49153aa34d6b19d69a007ec22a66"
Oct 30 20:39:57.513406 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.513386    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bada155176449de4f95a9134cc2bcf9d9a6e49153aa34d6b19d69a007ec22a66"} err="failed to get container status \"bada155176449de4f95a9134cc2bcf9d9a6e49153aa34d6b19d69a007ec22a66\": rpc error: code = NotFound desc = could not find container \"bada155176449de4f95a9134cc2bcf9d9a6e49153aa34d6b19d69a007ec22a66\": container with ID starting with bada155176449de4f95a9134cc2bcf9d9a6e49153aa34d6b19d69a007ec22a66 not found: ID does not exist"
Oct 30 20:39:57.513451 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.513408    5713 scope.go:122] "RemoveContainer" containerID="ac90672f57cdaaed8c17ddf63fec27de5e7299ed8d999f071519583ed0c0229b"
Oct 30 20:39:57.513707 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.513680    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac90672f57cdaaed8c17ddf63fec27de5e7299ed8d999f071519583ed0c0229b"} err="failed to get container status \"ac90672f57cdaaed8c17ddf63fec27de5e7299ed8d999f071519583ed0c0229b\": rpc error: code = NotFound desc = could not find container \"ac90672f57cdaaed8c17ddf63fec27de5e7299ed8d999f071519583ed0c0229b\": container with ID starting with ac90672f57cdaaed8c17ddf63fec27de5e7299ed8d999f071519583ed0c0229b not found: ID does not exist"
Oct 30 20:39:57.513707 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.513706    5713 scope.go:122] "RemoveContainer" containerID="9d4a60f970f9c790fb7a15f66d0c939c5cfe097422fc7390b667543e90ec02d7"
Oct 30 20:39:57.514117 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.514094    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d4a60f970f9c790fb7a15f66d0c939c5cfe097422fc7390b667543e90ec02d7"} err="failed to get container status \"9d4a60f970f9c790fb7a15f66d0c939c5cfe097422fc7390b667543e90ec02d7\": rpc error: code = NotFound desc = could not find container \"9d4a60f970f9c790fb7a15f66d0c939c5cfe097422fc7390b667543e90ec02d7\": container with ID starting with 9d4a60f970f9c790fb7a15f66d0c939c5cfe097422fc7390b667543e90ec02d7 not found: ID does not exist"
Oct 30 20:39:57.514180 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.514120    5713 scope.go:122] "RemoveContainer" containerID="b2305237333dd77b38d54b1bf6d4e64a02247bcb4e8d0baf57949b5b2a0b8893"
Oct 30 20:39:57.514346 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.514327    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2305237333dd77b38d54b1bf6d4e64a02247bcb4e8d0baf57949b5b2a0b8893"} err="failed to get container status \"b2305237333dd77b38d54b1bf6d4e64a02247bcb4e8d0baf57949b5b2a0b8893\": rpc error: code = NotFound desc = could not find container \"b2305237333dd77b38d54b1bf6d4e64a02247bcb4e8d0baf57949b5b2a0b8893\": container with ID starting with b2305237333dd77b38d54b1bf6d4e64a02247bcb4e8d0baf57949b5b2a0b8893 not found: ID does not exist"
Oct 30 20:39:57.514413 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.514349    5713 scope.go:122] "RemoveContainer" containerID="5d9bc8b61136c18c540be963353fc2c21014d8179a9d7d563d8b939d136c8ce5"
Oct 30 20:39:57.514641 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.514622    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d9bc8b61136c18c540be963353fc2c21014d8179a9d7d563d8b939d136c8ce5"} err="failed to get container status \"5d9bc8b61136c18c540be963353fc2c21014d8179a9d7d563d8b939d136c8ce5\": rpc error: code = NotFound desc = could not find container \"5d9bc8b61136c18c540be963353fc2c21014d8179a9d7d563d8b939d136c8ce5\": container with ID starting with 5d9bc8b61136c18c540be963353fc2c21014d8179a9d7d563d8b939d136c8ce5 not found: ID does not exist"
Oct 30 20:39:57.514703 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.514641    5713 scope.go:122] "RemoveContainer" containerID="cea18e6abdfde296bb779d83752561bccff0c9547b9829446ca54e81b9c71bad"
Oct 30 20:39:57.514939 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.514922    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cea18e6abdfde296bb779d83752561bccff0c9547b9829446ca54e81b9c71bad"} err="failed to get container status \"cea18e6abdfde296bb779d83752561bccff0c9547b9829446ca54e81b9c71bad\": rpc error: code = NotFound desc = could not find container \"cea18e6abdfde296bb779d83752561bccff0c9547b9829446ca54e81b9c71bad\": container with ID starting with cea18e6abdfde296bb779d83752561bccff0c9547b9829446ca54e81b9c71bad not found: ID does not exist"
Oct 30 20:39:57.515004 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.514940    5713 scope.go:122] "RemoveContainer" containerID="521dd12d5bf7bf42e12d8d6d5ac01b10c9993cfd8c508bb4c92dea2fc5092310"
Oct 30 20:39:57.515195 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.515172    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"521dd12d5bf7bf42e12d8d6d5ac01b10c9993cfd8c508bb4c92dea2fc5092310"} err="failed to get container status \"521dd12d5bf7bf42e12d8d6d5ac01b10c9993cfd8c508bb4c92dea2fc5092310\": rpc error: code = NotFound desc = could not find container \"521dd12d5bf7bf42e12d8d6d5ac01b10c9993cfd8c508bb4c92dea2fc5092310\": container with ID starting with 521dd12d5bf7bf42e12d8d6d5ac01b10c9993cfd8c508bb4c92dea2fc5092310 not found: ID does not exist"
Oct 30 20:39:57.515195 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.515193    5713 scope.go:122] "RemoveContainer" containerID="bada155176449de4f95a9134cc2bcf9d9a6e49153aa34d6b19d69a007ec22a66"
Oct 30 20:39:57.515471 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.515452    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bada155176449de4f95a9134cc2bcf9d9a6e49153aa34d6b19d69a007ec22a66"} err="failed to get container status \"bada155176449de4f95a9134cc2bcf9d9a6e49153aa34d6b19d69a007ec22a66\": rpc error: code = NotFound desc = could not find container \"bada155176449de4f95a9134cc2bcf9d9a6e49153aa34d6b19d69a007ec22a66\": container with ID starting with bada155176449de4f95a9134cc2bcf9d9a6e49153aa34d6b19d69a007ec22a66 not found: ID does not exist"
Oct 30 20:39:57.515471 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.515471    5713 scope.go:122] "RemoveContainer" containerID="ac90672f57cdaaed8c17ddf63fec27de5e7299ed8d999f071519583ed0c0229b"
Oct 30 20:39:57.515721 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.515704    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac90672f57cdaaed8c17ddf63fec27de5e7299ed8d999f071519583ed0c0229b"} err="failed to get container status \"ac90672f57cdaaed8c17ddf63fec27de5e7299ed8d999f071519583ed0c0229b\": rpc error: code = NotFound desc = could not find container \"ac90672f57cdaaed8c17ddf63fec27de5e7299ed8d999f071519583ed0c0229b\": container with ID starting with ac90672f57cdaaed8c17ddf63fec27de5e7299ed8d999f071519583ed0c0229b not found: ID does not exist"
Oct 30 20:39:57.515810 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.515721    5713 scope.go:122] "RemoveContainer" containerID="9d4a60f970f9c790fb7a15f66d0c939c5cfe097422fc7390b667543e90ec02d7"
Oct 30 20:39:57.515981 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.515963    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d4a60f970f9c790fb7a15f66d0c939c5cfe097422fc7390b667543e90ec02d7"} err="failed to get container status \"9d4a60f970f9c790fb7a15f66d0c939c5cfe097422fc7390b667543e90ec02d7\": rpc error: code = NotFound desc = could not find container \"9d4a60f970f9c790fb7a15f66d0c939c5cfe097422fc7390b667543e90ec02d7\": container with ID starting with 9d4a60f970f9c790fb7a15f66d0c939c5cfe097422fc7390b667543e90ec02d7 not found: ID does not exist"
Oct 30 20:39:57.516026 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.515982    5713 scope.go:122] "RemoveContainer" containerID="b2305237333dd77b38d54b1bf6d4e64a02247bcb4e8d0baf57949b5b2a0b8893"
Oct 30 20:39:57.516227 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.516207    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2305237333dd77b38d54b1bf6d4e64a02247bcb4e8d0baf57949b5b2a0b8893"} err="failed to get container status \"b2305237333dd77b38d54b1bf6d4e64a02247bcb4e8d0baf57949b5b2a0b8893\": rpc error: code = NotFound desc = could not find container \"b2305237333dd77b38d54b1bf6d4e64a02247bcb4e8d0baf57949b5b2a0b8893\": container with ID starting with b2305237333dd77b38d54b1bf6d4e64a02247bcb4e8d0baf57949b5b2a0b8893 not found: ID does not exist"
Oct 30 20:39:57.516283 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.516228    5713 scope.go:122] "RemoveContainer" containerID="5d9bc8b61136c18c540be963353fc2c21014d8179a9d7d563d8b939d136c8ce5"
Oct 30 20:39:57.516475 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.516458    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d9bc8b61136c18c540be963353fc2c21014d8179a9d7d563d8b939d136c8ce5"} err="failed to get container status \"5d9bc8b61136c18c540be963353fc2c21014d8179a9d7d563d8b939d136c8ce5\": rpc error: code = NotFound desc = could not find container \"5d9bc8b61136c18c540be963353fc2c21014d8179a9d7d563d8b939d136c8ce5\": container with ID starting with 5d9bc8b61136c18c540be963353fc2c21014d8179a9d7d563d8b939d136c8ce5 not found: ID does not exist"
Oct 30 20:39:57.516475 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.516475    5713 scope.go:122] "RemoveContainer" containerID="cea18e6abdfde296bb779d83752561bccff0c9547b9829446ca54e81b9c71bad"
Oct 30 20:39:57.516706 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.516690    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cea18e6abdfde296bb779d83752561bccff0c9547b9829446ca54e81b9c71bad"} err="failed to get container status \"cea18e6abdfde296bb779d83752561bccff0c9547b9829446ca54e81b9c71bad\": rpc error: code = NotFound desc = could not find container \"cea18e6abdfde296bb779d83752561bccff0c9547b9829446ca54e81b9c71bad\": container with ID starting with cea18e6abdfde296bb779d83752561bccff0c9547b9829446ca54e81b9c71bad not found: ID does not exist"
Oct 30 20:39:57.516706 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.516705    5713 scope.go:122] "RemoveContainer" containerID="521dd12d5bf7bf42e12d8d6d5ac01b10c9993cfd8c508bb4c92dea2fc5092310"
Oct 30 20:39:57.516935 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.516916    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"521dd12d5bf7bf42e12d8d6d5ac01b10c9993cfd8c508bb4c92dea2fc5092310"} err="failed to get container status \"521dd12d5bf7bf42e12d8d6d5ac01b10c9993cfd8c508bb4c92dea2fc5092310\": rpc error: code = NotFound desc = could not find container \"521dd12d5bf7bf42e12d8d6d5ac01b10c9993cfd8c508bb4c92dea2fc5092310\": container with ID starting with 521dd12d5bf7bf42e12d8d6d5ac01b10c9993cfd8c508bb4c92dea2fc5092310 not found: ID does not exist"
Oct 30 20:39:57.517002 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.516938    5713 scope.go:122] "RemoveContainer" containerID="bada155176449de4f95a9134cc2bcf9d9a6e49153aa34d6b19d69a007ec22a66"
Oct 30 20:39:57.517237 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.517213    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bada155176449de4f95a9134cc2bcf9d9a6e49153aa34d6b19d69a007ec22a66"} err="failed to get container status \"bada155176449de4f95a9134cc2bcf9d9a6e49153aa34d6b19d69a007ec22a66\": rpc error: code = NotFound desc = could not find container \"bada155176449de4f95a9134cc2bcf9d9a6e49153aa34d6b19d69a007ec22a66\": container with ID starting with bada155176449de4f95a9134cc2bcf9d9a6e49153aa34d6b19d69a007ec22a66 not found: ID does not exist"
Oct 30 20:39:57.517237 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.517230    5713 scope.go:122] "RemoveContainer" containerID="ac90672f57cdaaed8c17ddf63fec27de5e7299ed8d999f071519583ed0c0229b"
Oct 30 20:39:57.517502 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.517483    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac90672f57cdaaed8c17ddf63fec27de5e7299ed8d999f071519583ed0c0229b"} err="failed to get container status \"ac90672f57cdaaed8c17ddf63fec27de5e7299ed8d999f071519583ed0c0229b\": rpc error: code = NotFound desc = could not find container \"ac90672f57cdaaed8c17ddf63fec27de5e7299ed8d999f071519583ed0c0229b\": container with ID starting with ac90672f57cdaaed8c17ddf63fec27de5e7299ed8d999f071519583ed0c0229b not found: ID does not exist"
Oct 30 20:39:57.517582 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.517504    5713 scope.go:122] "RemoveContainer" containerID="9d4a60f970f9c790fb7a15f66d0c939c5cfe097422fc7390b667543e90ec02d7"
Oct 30 20:39:57.517776 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.517759    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d4a60f970f9c790fb7a15f66d0c939c5cfe097422fc7390b667543e90ec02d7"} err="failed to get container status \"9d4a60f970f9c790fb7a15f66d0c939c5cfe097422fc7390b667543e90ec02d7\": rpc error: code = NotFound desc = could not find container \"9d4a60f970f9c790fb7a15f66d0c939c5cfe097422fc7390b667543e90ec02d7\": container with ID starting with 9d4a60f970f9c790fb7a15f66d0c939c5cfe097422fc7390b667543e90ec02d7 not found: ID does not exist"
Oct 30 20:39:57.517840 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.517778    5713 scope.go:122] "RemoveContainer" containerID="b2305237333dd77b38d54b1bf6d4e64a02247bcb4e8d0baf57949b5b2a0b8893"
Oct 30 20:39:57.518066 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.518048    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2305237333dd77b38d54b1bf6d4e64a02247bcb4e8d0baf57949b5b2a0b8893"} err="failed to get container status \"b2305237333dd77b38d54b1bf6d4e64a02247bcb4e8d0baf57949b5b2a0b8893\": rpc error: code = NotFound desc = could not find container \"b2305237333dd77b38d54b1bf6d4e64a02247bcb4e8d0baf57949b5b2a0b8893\": container with ID starting with b2305237333dd77b38d54b1bf6d4e64a02247bcb4e8d0baf57949b5b2a0b8893 not found: ID does not exist"
Oct 30 20:39:57.518110 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.518066    5713 scope.go:122] "RemoveContainer" containerID="5d9bc8b61136c18c540be963353fc2c21014d8179a9d7d563d8b939d136c8ce5"
Oct 30 20:39:57.518301 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.518284    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d9bc8b61136c18c540be963353fc2c21014d8179a9d7d563d8b939d136c8ce5"} err="failed to get container status \"5d9bc8b61136c18c540be963353fc2c21014d8179a9d7d563d8b939d136c8ce5\": rpc error: code = NotFound desc = could not find container \"5d9bc8b61136c18c540be963353fc2c21014d8179a9d7d563d8b939d136c8ce5\": container with ID starting with 5d9bc8b61136c18c540be963353fc2c21014d8179a9d7d563d8b939d136c8ce5 not found: ID does not exist"
Oct 30 20:39:57.518365 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.518302    5713 scope.go:122] "RemoveContainer" containerID="cea18e6abdfde296bb779d83752561bccff0c9547b9829446ca54e81b9c71bad"
Oct 30 20:39:57.518569 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.518553    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cea18e6abdfde296bb779d83752561bccff0c9547b9829446ca54e81b9c71bad"} err="failed to get container status \"cea18e6abdfde296bb779d83752561bccff0c9547b9829446ca54e81b9c71bad\": rpc error: code = NotFound desc = could not find container \"cea18e6abdfde296bb779d83752561bccff0c9547b9829446ca54e81b9c71bad\": container with ID starting with cea18e6abdfde296bb779d83752561bccff0c9547b9829446ca54e81b9c71bad not found: ID does not exist"
Oct 30 20:39:57.518631 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.518569    5713 scope.go:122] "RemoveContainer" containerID="521dd12d5bf7bf42e12d8d6d5ac01b10c9993cfd8c508bb4c92dea2fc5092310"
Oct 30 20:39:57.518827 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.518810    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"521dd12d5bf7bf42e12d8d6d5ac01b10c9993cfd8c508bb4c92dea2fc5092310"} err="failed to get container status \"521dd12d5bf7bf42e12d8d6d5ac01b10c9993cfd8c508bb4c92dea2fc5092310\": rpc error: code = NotFound desc = could not find container \"521dd12d5bf7bf42e12d8d6d5ac01b10c9993cfd8c508bb4c92dea2fc5092310\": container with ID starting with 521dd12d5bf7bf42e12d8d6d5ac01b10c9993cfd8c508bb4c92dea2fc5092310 not found: ID does not exist"
Oct 30 20:39:57.518889 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.518828    5713 scope.go:122] "RemoveContainer" containerID="bada155176449de4f95a9134cc2bcf9d9a6e49153aa34d6b19d69a007ec22a66"
Oct 30 20:39:57.519081 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.519064    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bada155176449de4f95a9134cc2bcf9d9a6e49153aa34d6b19d69a007ec22a66"} err="failed to get container status \"bada155176449de4f95a9134cc2bcf9d9a6e49153aa34d6b19d69a007ec22a66\": rpc error: code = NotFound desc = could not find container \"bada155176449de4f95a9134cc2bcf9d9a6e49153aa34d6b19d69a007ec22a66\": container with ID starting with bada155176449de4f95a9134cc2bcf9d9a6e49153aa34d6b19d69a007ec22a66 not found: ID does not exist"
Oct 30 20:39:57.519160 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.519083    5713 scope.go:122] "RemoveContainer" containerID="ac90672f57cdaaed8c17ddf63fec27de5e7299ed8d999f071519583ed0c0229b"
Oct 30 20:39:57.519337 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.519321    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac90672f57cdaaed8c17ddf63fec27de5e7299ed8d999f071519583ed0c0229b"} err="failed to get container status \"ac90672f57cdaaed8c17ddf63fec27de5e7299ed8d999f071519583ed0c0229b\": rpc error: code = NotFound desc = could not find container \"ac90672f57cdaaed8c17ddf63fec27de5e7299ed8d999f071519583ed0c0229b\": container with ID starting with ac90672f57cdaaed8c17ddf63fec27de5e7299ed8d999f071519583ed0c0229b not found: ID does not exist"
Oct 30 20:39:57.519392 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.519336    5713 scope.go:122] "RemoveContainer" containerID="9d4a60f970f9c790fb7a15f66d0c939c5cfe097422fc7390b667543e90ec02d7"
Oct 30 20:39:57.519546 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.519531    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d4a60f970f9c790fb7a15f66d0c939c5cfe097422fc7390b667543e90ec02d7"} err="failed to get container status \"9d4a60f970f9c790fb7a15f66d0c939c5cfe097422fc7390b667543e90ec02d7\": rpc error: code = NotFound desc = could not find container \"9d4a60f970f9c790fb7a15f66d0c939c5cfe097422fc7390b667543e90ec02d7\": container with ID starting with 9d4a60f970f9c790fb7a15f66d0c939c5cfe097422fc7390b667543e90ec02d7 not found: ID does not exist"
Oct 30 20:39:57.519593 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.519546    5713 scope.go:122] "RemoveContainer" containerID="b2305237333dd77b38d54b1bf6d4e64a02247bcb4e8d0baf57949b5b2a0b8893"
Oct 30 20:39:57.519778 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.519761    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2305237333dd77b38d54b1bf6d4e64a02247bcb4e8d0baf57949b5b2a0b8893"} err="failed to get container status \"b2305237333dd77b38d54b1bf6d4e64a02247bcb4e8d0baf57949b5b2a0b8893\": rpc error: code = NotFound desc = could not find container \"b2305237333dd77b38d54b1bf6d4e64a02247bcb4e8d0baf57949b5b2a0b8893\": container with ID starting with b2305237333dd77b38d54b1bf6d4e64a02247bcb4e8d0baf57949b5b2a0b8893 not found: ID does not exist"
Oct 30 20:39:57.519778 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.519778    5713 scope.go:122] "RemoveContainer" containerID="5d9bc8b61136c18c540be963353fc2c21014d8179a9d7d563d8b939d136c8ce5"
Oct 30 20:39:57.520043 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.520023    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d9bc8b61136c18c540be963353fc2c21014d8179a9d7d563d8b939d136c8ce5"} err="failed to get container status \"5d9bc8b61136c18c540be963353fc2c21014d8179a9d7d563d8b939d136c8ce5\": rpc error: code = NotFound desc = could not find container \"5d9bc8b61136c18c540be963353fc2c21014d8179a9d7d563d8b939d136c8ce5\": container with ID starting with 5d9bc8b61136c18c540be963353fc2c21014d8179a9d7d563d8b939d136c8ce5 not found: ID does not exist"
Oct 30 20:39:57.520043 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.520043    5713 scope.go:122] "RemoveContainer" containerID="cea18e6abdfde296bb779d83752561bccff0c9547b9829446ca54e81b9c71bad"
Oct 30 20:39:57.520313 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.520296    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cea18e6abdfde296bb779d83752561bccff0c9547b9829446ca54e81b9c71bad"} err="failed to get container status \"cea18e6abdfde296bb779d83752561bccff0c9547b9829446ca54e81b9c71bad\": rpc error: code = NotFound desc = could not find container \"cea18e6abdfde296bb779d83752561bccff0c9547b9829446ca54e81b9c71bad\": container with ID starting with cea18e6abdfde296bb779d83752561bccff0c9547b9829446ca54e81b9c71bad not found: ID does not exist"
Oct 30 20:39:57.520373 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.520313    5713 scope.go:122] "RemoveContainer" containerID="521dd12d5bf7bf42e12d8d6d5ac01b10c9993cfd8c508bb4c92dea2fc5092310"
Oct 30 20:39:57.520554 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.520532    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"521dd12d5bf7bf42e12d8d6d5ac01b10c9993cfd8c508bb4c92dea2fc5092310"} err="failed to get container status \"521dd12d5bf7bf42e12d8d6d5ac01b10c9993cfd8c508bb4c92dea2fc5092310\": rpc error: code = NotFound desc = could not find container \"521dd12d5bf7bf42e12d8d6d5ac01b10c9993cfd8c508bb4c92dea2fc5092310\": container with ID starting with 521dd12d5bf7bf42e12d8d6d5ac01b10c9993cfd8c508bb4c92dea2fc5092310 not found: ID does not exist"
Oct 30 20:39:57.520554 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.520552    5713 scope.go:122] "RemoveContainer" containerID="bada155176449de4f95a9134cc2bcf9d9a6e49153aa34d6b19d69a007ec22a66"
Oct 30 20:39:57.520815 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.520780    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bada155176449de4f95a9134cc2bcf9d9a6e49153aa34d6b19d69a007ec22a66"} err="failed to get container status \"bada155176449de4f95a9134cc2bcf9d9a6e49153aa34d6b19d69a007ec22a66\": rpc error: code = NotFound desc = could not find container \"bada155176449de4f95a9134cc2bcf9d9a6e49153aa34d6b19d69a007ec22a66\": container with ID starting with bada155176449de4f95a9134cc2bcf9d9a6e49153aa34d6b19d69a007ec22a66 not found: ID does not exist"
Oct 30 20:39:57.520815 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.520815    5713 scope.go:122] "RemoveContainer" containerID="ac90672f57cdaaed8c17ddf63fec27de5e7299ed8d999f071519583ed0c0229b"
Oct 30 20:39:57.521074 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.521054    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac90672f57cdaaed8c17ddf63fec27de5e7299ed8d999f071519583ed0c0229b"} err="failed to get container status \"ac90672f57cdaaed8c17ddf63fec27de5e7299ed8d999f071519583ed0c0229b\": rpc error: code = NotFound desc = could not find container \"ac90672f57cdaaed8c17ddf63fec27de5e7299ed8d999f071519583ed0c0229b\": container with ID starting with ac90672f57cdaaed8c17ddf63fec27de5e7299ed8d999f071519583ed0c0229b not found: ID does not exist"
Oct 30 20:39:57.521157 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.521077    5713 scope.go:122] "RemoveContainer" containerID="9d4a60f970f9c790fb7a15f66d0c939c5cfe097422fc7390b667543e90ec02d7"
Oct 30 20:39:57.521363 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.521344    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d4a60f970f9c790fb7a15f66d0c939c5cfe097422fc7390b667543e90ec02d7"} err="failed to get container status \"9d4a60f970f9c790fb7a15f66d0c939c5cfe097422fc7390b667543e90ec02d7\": rpc error: code = NotFound desc = could not find container \"9d4a60f970f9c790fb7a15f66d0c939c5cfe097422fc7390b667543e90ec02d7\": container with ID starting with 9d4a60f970f9c790fb7a15f66d0c939c5cfe097422fc7390b667543e90ec02d7 not found: ID does not exist"
Oct 30 20:39:57.521452 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.521363    5713 scope.go:122] "RemoveContainer" containerID="b2305237333dd77b38d54b1bf6d4e64a02247bcb4e8d0baf57949b5b2a0b8893"
Oct 30 20:39:57.521610 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.521589    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2305237333dd77b38d54b1bf6d4e64a02247bcb4e8d0baf57949b5b2a0b8893"} err="failed to get container status \"b2305237333dd77b38d54b1bf6d4e64a02247bcb4e8d0baf57949b5b2a0b8893\": rpc error: code = NotFound desc = could not find container \"b2305237333dd77b38d54b1bf6d4e64a02247bcb4e8d0baf57949b5b2a0b8893\": container with ID starting with b2305237333dd77b38d54b1bf6d4e64a02247bcb4e8d0baf57949b5b2a0b8893 not found: ID does not exist"
Oct 30 20:39:57.521694 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.521611    5713 scope.go:122] "RemoveContainer" containerID="5d9bc8b61136c18c540be963353fc2c21014d8179a9d7d563d8b939d136c8ce5"
Oct 30 20:39:57.521849 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.521830    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d9bc8b61136c18c540be963353fc2c21014d8179a9d7d563d8b939d136c8ce5"} err="failed to get container status \"5d9bc8b61136c18c540be963353fc2c21014d8179a9d7d563d8b939d136c8ce5\": rpc error: code = NotFound desc = could not find container \"5d9bc8b61136c18c540be963353fc2c21014d8179a9d7d563d8b939d136c8ce5\": container with ID starting with 5d9bc8b61136c18c540be963353fc2c21014d8179a9d7d563d8b939d136c8ce5 not found: ID does not exist"
Oct 30 20:39:57.521923 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.521851    5713 scope.go:122] "RemoveContainer" containerID="cea18e6abdfde296bb779d83752561bccff0c9547b9829446ca54e81b9c71bad"
Oct 30 20:39:57.522113 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.522091    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cea18e6abdfde296bb779d83752561bccff0c9547b9829446ca54e81b9c71bad"} err="failed to get container status \"cea18e6abdfde296bb779d83752561bccff0c9547b9829446ca54e81b9c71bad\": rpc error: code = NotFound desc = could not find container \"cea18e6abdfde296bb779d83752561bccff0c9547b9829446ca54e81b9c71bad\": container with ID starting with cea18e6abdfde296bb779d83752561bccff0c9547b9829446ca54e81b9c71bad not found: ID does not exist"
Oct 30 20:39:57.522183 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.522114    5713 scope.go:122] "RemoveContainer" containerID="521dd12d5bf7bf42e12d8d6d5ac01b10c9993cfd8c508bb4c92dea2fc5092310"
Oct 30 20:39:57.522379 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.522360    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"521dd12d5bf7bf42e12d8d6d5ac01b10c9993cfd8c508bb4c92dea2fc5092310"} err="failed to get container status \"521dd12d5bf7bf42e12d8d6d5ac01b10c9993cfd8c508bb4c92dea2fc5092310\": rpc error: code = NotFound desc = could not find container \"521dd12d5bf7bf42e12d8d6d5ac01b10c9993cfd8c508bb4c92dea2fc5092310\": container with ID starting with 521dd12d5bf7bf42e12d8d6d5ac01b10c9993cfd8c508bb4c92dea2fc5092310 not found: ID does not exist"
Oct 30 20:39:57.620995 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.620966    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 20:39:57.670776 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.670738    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 20:39:57.671253 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.671225    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 20:39:57.671628 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.671598    5713 cpu_manager.go:414] "containerMap: removing container" podUID="bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" containerName="kube-rbac-proxy"
Oct 30 20:39:57.671628 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.671623    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" containerName="kube-rbac-proxy"
Oct 30 20:39:57.671783 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.671646    5713 cpu_manager.go:414] "containerMap: removing container" podUID="bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" containerName="kube-rbac-proxy-thanos"
Oct 30 20:39:57.671783 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.671655    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" containerName="kube-rbac-proxy-thanos"
Oct 30 20:39:57.671783 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.671671    5713 cpu_manager.go:414] "containerMap: removing container" podUID="bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" containerName="kube-rbac-proxy-web"
Oct 30 20:39:57.671783 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.671679    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" containerName="kube-rbac-proxy-web"
Oct 30 20:39:57.671783 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.671690    5713 cpu_manager.go:414] "containerMap: removing container" podUID="bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" containerName="init-config-reloader"
Oct 30 20:39:57.671783 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.671697    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" containerName="init-config-reloader"
Oct 30 20:39:57.671783 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.671709    5713 cpu_manager.go:414] "containerMap: removing container" podUID="bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" containerName="thanos-sidecar"
Oct 30 20:39:57.671783 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.671715    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" containerName="thanos-sidecar"
Oct 30 20:39:57.671783 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.671729    5713 cpu_manager.go:414] "containerMap: removing container" podUID="bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" containerName="config-reloader"
Oct 30 20:39:57.671783 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.671736    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" containerName="config-reloader"
Oct 30 20:39:57.671783 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.671748    5713 cpu_manager.go:414] "containerMap: removing container" podUID="bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" containerName="prometheus"
Oct 30 20:39:57.671783 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.671755    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" containerName="prometheus"
Oct 30 20:39:57.672322 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.671849    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" containerName="thanos-sidecar"
Oct 30 20:39:57.672322 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.671862    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" containerName="config-reloader"
Oct 30 20:39:57.672322 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.671871    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" containerName="prometheus"
Oct 30 20:39:57.672322 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.671881    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" containerName="kube-rbac-proxy"
Oct 30 20:39:57.672322 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.671892    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" containerName="kube-rbac-proxy-thanos"
Oct 30 20:39:57.672322 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.671901    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" containerName="kube-rbac-proxy-web"
Oct 30 20:39:57.672322 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.671909    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" containerName="init-config-reloader"
Oct 30 20:39:57.675590 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.675570    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:39:57.728600 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.728523    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 20:39:57.728600 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.728523    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 20:39:57.728600 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.728591    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 20:39:57.728922 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.728623    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 20:39:57.728922 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.728640    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 20:39:57.729221 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.729195    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 20:39:57.729360 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.729248    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 20:39:57.729360 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.729263    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-szlhz\""
Oct 30 20:39:57.729360 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.729329    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-55jjhefcr4ig6\""
Oct 30 20:39:57.729360 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.729351    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 20:39:57.729646 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.729625    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 20:39:57.729646 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.729627    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 20:39:57.729890 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.729648    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 20:39:57.729890 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.729695    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 20:39:57.776473 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.776442    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4597073-8ff0-4a4e-933c-c64e934fe977-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d4597073-8ff0-4a4e-933c-c64e934fe977\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:39:57.776473 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.776477    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d4597073-8ff0-4a4e-933c-c64e934fe977-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d4597073-8ff0-4a4e-933c-c64e934fe977\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:39:57.776691 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.776496    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d4597073-8ff0-4a4e-933c-c64e934fe977-config\") pod \"prometheus-k8s-0\" (UID: \"d4597073-8ff0-4a4e-933c-c64e934fe977\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:39:57.776691 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.776518    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d4597073-8ff0-4a4e-933c-c64e934fe977-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d4597073-8ff0-4a4e-933c-c64e934fe977\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:39:57.776691 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.776606    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d4597073-8ff0-4a4e-933c-c64e934fe977-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d4597073-8ff0-4a4e-933c-c64e934fe977\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:39:57.776836 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.776813    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d4597073-8ff0-4a4e-933c-c64e934fe977-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d4597073-8ff0-4a4e-933c-c64e934fe977\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:39:57.776883 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.776860    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d4597073-8ff0-4a4e-933c-c64e934fe977-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d4597073-8ff0-4a4e-933c-c64e934fe977\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:39:57.776999 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.776973    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d4597073-8ff0-4a4e-933c-c64e934fe977-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d4597073-8ff0-4a4e-933c-c64e934fe977\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:39:57.777057 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.777019    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d4597073-8ff0-4a4e-933c-c64e934fe977-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d4597073-8ff0-4a4e-933c-c64e934fe977\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:39:57.777057 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.777049    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d4597073-8ff0-4a4e-933c-c64e934fe977-config-out\") pod \"prometheus-k8s-0\" (UID: \"d4597073-8ff0-4a4e-933c-c64e934fe977\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:39:57.777139 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.777077    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d4597073-8ff0-4a4e-933c-c64e934fe977-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d4597073-8ff0-4a4e-933c-c64e934fe977\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:39:57.777139 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.777122    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d4597073-8ff0-4a4e-933c-c64e934fe977-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d4597073-8ff0-4a4e-933c-c64e934fe977\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:39:57.777211 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.777173    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4597073-8ff0-4a4e-933c-c64e934fe977-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d4597073-8ff0-4a4e-933c-c64e934fe977\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:39:57.777252 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.777209    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d4597073-8ff0-4a4e-933c-c64e934fe977-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d4597073-8ff0-4a4e-933c-c64e934fe977\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:39:57.777252 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.777232    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t9zq\" (UniqueName: \"kubernetes.io/projected/d4597073-8ff0-4a4e-933c-c64e934fe977-kube-api-access-8t9zq\") pod \"prometheus-k8s-0\" (UID: \"d4597073-8ff0-4a4e-933c-c64e934fe977\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:39:57.777385 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.777353    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4597073-8ff0-4a4e-933c-c64e934fe977-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d4597073-8ff0-4a4e-933c-c64e934fe977\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:39:57.777493 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.777396    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d4597073-8ff0-4a4e-933c-c64e934fe977-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d4597073-8ff0-4a4e-933c-c64e934fe977\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:39:57.777493 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.777418    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d4597073-8ff0-4a4e-933c-c64e934fe977-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d4597073-8ff0-4a4e-933c-c64e934fe977\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:39:57.777493 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.777437    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d4597073-8ff0-4a4e-933c-c64e934fe977-web-config\") pod \"prometheus-k8s-0\" (UID: \"d4597073-8ff0-4a4e-933c-c64e934fe977\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:39:57.777493 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.777466    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d4597073-8ff0-4a4e-933c-c64e934fe977-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d4597073-8ff0-4a4e-933c-c64e934fe977\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:39:57.779209 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.779189    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 20:39:57.779303 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.779212    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 20:39:57.789421 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.789392    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 20:39:57.878403 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.878364    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4597073-8ff0-4a4e-933c-c64e934fe977-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d4597073-8ff0-4a4e-933c-c64e934fe977\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:39:57.878403 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.878401    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d4597073-8ff0-4a4e-933c-c64e934fe977-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d4597073-8ff0-4a4e-933c-c64e934fe977\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:39:57.878641 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.878419    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d4597073-8ff0-4a4e-933c-c64e934fe977-config\") pod \"prometheus-k8s-0\" (UID: \"d4597073-8ff0-4a4e-933c-c64e934fe977\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:39:57.878641 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.878574    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d4597073-8ff0-4a4e-933c-c64e934fe977-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d4597073-8ff0-4a4e-933c-c64e934fe977\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:39:57.878641 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.878623    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d4597073-8ff0-4a4e-933c-c64e934fe977-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d4597073-8ff0-4a4e-933c-c64e934fe977\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:39:57.878826 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.878668    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d4597073-8ff0-4a4e-933c-c64e934fe977-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d4597073-8ff0-4a4e-933c-c64e934fe977\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:39:57.878826 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.878699    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d4597073-8ff0-4a4e-933c-c64e934fe977-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d4597073-8ff0-4a4e-933c-c64e934fe977\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:39:57.878826 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.878760    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d4597073-8ff0-4a4e-933c-c64e934fe977-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d4597073-8ff0-4a4e-933c-c64e934fe977\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:39:57.878826 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.878810    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d4597073-8ff0-4a4e-933c-c64e934fe977-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d4597073-8ff0-4a4e-933c-c64e934fe977\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:39:57.879029 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.878840    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d4597073-8ff0-4a4e-933c-c64e934fe977-config-out\") pod \"prometheus-k8s-0\" (UID: \"d4597073-8ff0-4a4e-933c-c64e934fe977\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:39:57.879029 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.878861    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d4597073-8ff0-4a4e-933c-c64e934fe977-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d4597073-8ff0-4a4e-933c-c64e934fe977\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:39:57.879029 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.878879    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d4597073-8ff0-4a4e-933c-c64e934fe977-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d4597073-8ff0-4a4e-933c-c64e934fe977\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:39:57.879029 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.878899    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4597073-8ff0-4a4e-933c-c64e934fe977-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d4597073-8ff0-4a4e-933c-c64e934fe977\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:39:57.879029 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.878945    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d4597073-8ff0-4a4e-933c-c64e934fe977-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d4597073-8ff0-4a4e-933c-c64e934fe977\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:39:57.879029 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.878968    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t9zq\" (UniqueName: \"kubernetes.io/projected/d4597073-8ff0-4a4e-933c-c64e934fe977-kube-api-access-8t9zq\") pod \"prometheus-k8s-0\" (UID: \"d4597073-8ff0-4a4e-933c-c64e934fe977\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:39:57.879029 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.878983    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d4597073-8ff0-4a4e-933c-c64e934fe977-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d4597073-8ff0-4a4e-933c-c64e934fe977\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:39:57.879526 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.879038    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4597073-8ff0-4a4e-933c-c64e934fe977-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d4597073-8ff0-4a4e-933c-c64e934fe977\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:39:57.879526 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.879076    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d4597073-8ff0-4a4e-933c-c64e934fe977-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d4597073-8ff0-4a4e-933c-c64e934fe977\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:39:57.879526 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.879110    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d4597073-8ff0-4a4e-933c-c64e934fe977-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d4597073-8ff0-4a4e-933c-c64e934fe977\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:39:57.879526 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.879136    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d4597073-8ff0-4a4e-933c-c64e934fe977-web-config\") pod \"prometheus-k8s-0\" (UID: \"d4597073-8ff0-4a4e-933c-c64e934fe977\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:39:57.879526 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.879164    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d4597073-8ff0-4a4e-933c-c64e934fe977-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d4597073-8ff0-4a4e-933c-c64e934fe977\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:39:57.879526 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.879343    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d4597073-8ff0-4a4e-933c-c64e934fe977-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d4597073-8ff0-4a4e-933c-c64e934fe977\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:39:57.879857 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.879644    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4597073-8ff0-4a4e-933c-c64e934fe977-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d4597073-8ff0-4a4e-933c-c64e934fe977\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:39:57.880200 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.880173    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d4597073-8ff0-4a4e-933c-c64e934fe977-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d4597073-8ff0-4a4e-933c-c64e934fe977\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:39:57.880420 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.880397    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4597073-8ff0-4a4e-933c-c64e934fe977-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d4597073-8ff0-4a4e-933c-c64e934fe977\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:39:57.881270 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.881241    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d4597073-8ff0-4a4e-933c-c64e934fe977-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d4597073-8ff0-4a4e-933c-c64e934fe977\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:39:57.883334 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.882638    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d4597073-8ff0-4a4e-933c-c64e934fe977-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d4597073-8ff0-4a4e-933c-c64e934fe977\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:39:57.883334 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.882882    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d4597073-8ff0-4a4e-933c-c64e934fe977-config-out\") pod \"prometheus-k8s-0\" (UID: \"d4597073-8ff0-4a4e-933c-c64e934fe977\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:39:57.883334 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.882953    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d4597073-8ff0-4a4e-933c-c64e934fe977-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d4597073-8ff0-4a4e-933c-c64e934fe977\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:39:57.883334 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.883291    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4597073-8ff0-4a4e-933c-c64e934fe977-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d4597073-8ff0-4a4e-933c-c64e934fe977\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:39:57.884243 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.884201    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d4597073-8ff0-4a4e-933c-c64e934fe977-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d4597073-8ff0-4a4e-933c-c64e934fe977\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:39:57.884340 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.884314    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d4597073-8ff0-4a4e-933c-c64e934fe977-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d4597073-8ff0-4a4e-933c-c64e934fe977\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:39:57.884700 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.884656    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d4597073-8ff0-4a4e-933c-c64e934fe977-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d4597073-8ff0-4a4e-933c-c64e934fe977\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:39:57.885447 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.885427    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d4597073-8ff0-4a4e-933c-c64e934fe977-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d4597073-8ff0-4a4e-933c-c64e934fe977\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:39:57.885550 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.885528    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d4597073-8ff0-4a4e-933c-c64e934fe977-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d4597073-8ff0-4a4e-933c-c64e934fe977\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:39:57.886272 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.886248    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d4597073-8ff0-4a4e-933c-c64e934fe977-web-config\") pod \"prometheus-k8s-0\" (UID: \"d4597073-8ff0-4a4e-933c-c64e934fe977\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:39:57.887281 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.887263    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/d4597073-8ff0-4a4e-933c-c64e934fe977-config\") pod \"prometheus-k8s-0\" (UID: \"d4597073-8ff0-4a4e-933c-c64e934fe977\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:39:57.889750 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.889727    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d4597073-8ff0-4a4e-933c-c64e934fe977-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d4597073-8ff0-4a4e-933c-c64e934fe977\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:39:57.891665 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.891640    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d4597073-8ff0-4a4e-933c-c64e934fe977-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d4597073-8ff0-4a4e-933c-c64e934fe977\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:39:57.953057 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:57.953026    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t9zq\" (UniqueName: \"kubernetes.io/projected/d4597073-8ff0-4a4e-933c-c64e934fe977-kube-api-access-8t9zq\") pod \"prometheus-k8s-0\" (UID: \"d4597073-8ff0-4a4e-933c-c64e934fe977\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:39:58.002873 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:58.002838    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:39:58.460052 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:58.460013    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-744b77f955-tzvzl" event={"ID":"f94cda1e-ed8b-4b22-9620-a0c3cf1af271","Type":"ContainerStarted","Data":"6e11a22cdbefb29c6e65e5820f3ddc001d55c7e10d9affb5e5c95c0a39a11c23"}
Oct 30 20:39:58.460535 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:58.460073    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-744b77f955-tzvzl" event={"ID":"f94cda1e-ed8b-4b22-9620-a0c3cf1af271","Type":"ContainerStarted","Data":"2d5fa8da417fac555b7a44f2473a888934aa1bd14d369d5a99704dd23642f621"}
Oct 30 20:39:58.460535 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:58.460093    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-744b77f955-tzvzl" event={"ID":"f94cda1e-ed8b-4b22-9620-a0c3cf1af271","Type":"ContainerStarted","Data":"5c986aa9ddc2c05d916c443d495829de4d8218d47a7381350329e41385acdb1a"}
Oct 30 20:39:58.483448 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:58.483419    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 20:39:58.486072 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:39:58.486042    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4597073_8ff0_4a4e_933c_c64e934fe977.slice/crio-6634976d01ec1a83fd0cc107c0527b66ff500c033ff44df8e3986946998000be WatchSource:0}: Error finding container 6634976d01ec1a83fd0cc107c0527b66ff500c033ff44df8e3986946998000be: Status 404 returned error can't find the container with id 6634976d01ec1a83fd0cc107c0527b66ff500c033ff44df8e3986946998000be
Oct 30 20:39:58.578122 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:58.578074    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-744b77f955-tzvzl" podStartSLOduration=2.238969156 podStartE2EDuration="4.578059374s" totalImagesPullingTime="2.339090218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:39:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:39:58.577328211 +0000 UTC m=+81.642104612" watchObservedRunningTime="2026-10-30 20:39:58.578059374 +0000 UTC m=+81.642835772"
Oct 30 20:39:59.061230 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:59.061195    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df" path="/var/lib/kubelet/pods/bbe18c50-e3ff-4684-b9a1-cb5a83f6d6df/volumes"
Oct 30 20:39:59.466741 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:59.466701    5713 generic.go:356] "Generic (PLEG): container finished" podID="d4597073-8ff0-4a4e-933c-c64e934fe977" containerID="27a49da2e25f14db0749d1bffaf35ac2a70c1abebbba9fb517f94aa7103d0824" exitCode=0
Oct 30 20:39:59.467122 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:59.466816    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d4597073-8ff0-4a4e-933c-c64e934fe977","Type":"ContainerDied","Data":"27a49da2e25f14db0749d1bffaf35ac2a70c1abebbba9fb517f94aa7103d0824"}
Oct 30 20:39:59.467122 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:39:59.466896    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d4597073-8ff0-4a4e-933c-c64e934fe977","Type":"ContainerStarted","Data":"6634976d01ec1a83fd0cc107c0527b66ff500c033ff44df8e3986946998000be"}
Oct 30 20:40:00.473620 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:40:00.473594    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d4597073-8ff0-4a4e-933c-c64e934fe977/prometheus/0.log"
Oct 30 20:40:00.474239 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:40:00.474025    5713 generic.go:356] "Generic (PLEG): container finished" podID="d4597073-8ff0-4a4e-933c-c64e934fe977" containerID="105e0e9c5b15b2d11832f99814a94fae89b1740d66ccd40ef2dd00a00544ecd8" exitCode=2
Oct 30 20:40:00.474239 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:40:00.474102    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d4597073-8ff0-4a4e-933c-c64e934fe977","Type":"ContainerStarted","Data":"78fdf33e9643291c35ed365c17014f8eccb2020cf7607bb796bf03d90b3a3369"}
Oct 30 20:40:00.474239 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:40:00.474178    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d4597073-8ff0-4a4e-933c-c64e934fe977","Type":"ContainerStarted","Data":"6fd98716d373c225f659b04253dbcb61f3bfd79b675d4b070709190a29282af4"}
Oct 30 20:40:00.474239 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:40:00.474205    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d4597073-8ff0-4a4e-933c-c64e934fe977","Type":"ContainerStarted","Data":"094c1e3073e25e42cb8eba5c6ee59c75a301aab11f46872c07244cbcf74a0781"}
Oct 30 20:40:00.474239 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:40:00.474226    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d4597073-8ff0-4a4e-933c-c64e934fe977","Type":"ContainerStarted","Data":"9dd4d2667e9392efeccbcbaf8d92fa9c9b1c4b806255903e7caeaedc21e67b5d"}
Oct 30 20:40:00.474422 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:40:00.474252    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d4597073-8ff0-4a4e-933c-c64e934fe977","Type":"ContainerStarted","Data":"f285fb277c5ffcf6d5c7e78081eac1bb414d3d29284aa9ba7cc3b38706b2a9a7"}
Oct 30 20:40:00.474422 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:40:00.474277    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d4597073-8ff0-4a4e-933c-c64e934fe977","Type":"ContainerDied","Data":"105e0e9c5b15b2d11832f99814a94fae89b1740d66ccd40ef2dd00a00544ecd8"}
Oct 30 20:40:00.474531 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:40:00.474519    5713 scope.go:122] "RemoveContainer" containerID="105e0e9c5b15b2d11832f99814a94fae89b1740d66ccd40ef2dd00a00544ecd8"
Oct 30 20:40:01.481880 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:40:01.481857    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d4597073-8ff0-4a4e-933c-c64e934fe977/prometheus/0.log"
Oct 30 20:40:01.482303 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:40:01.482236    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d4597073-8ff0-4a4e-933c-c64e934fe977","Type":"ContainerStarted","Data":"708b7216a59341b944dfff53d8591ea9f74b302965c9a2e26b326459aec9dd0e"}
Oct 30 20:40:01.666146 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:40:01.666093    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.649322289 podStartE2EDuration="4.666079876s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.757587ms" isStatefulPod=false podCreationTimestamp="2026-10-30 20:39:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:40:01.663678788 +0000 UTC m=+84.728455178" watchObservedRunningTime="2026-10-30 20:40:01.666079876 +0000 UTC m=+84.730856271"
Oct 30 20:40:03.003565 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:40:03.003517    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:40:24.456653 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:40:24.456578    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 20:40:54.457640 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:40:54.457605    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 20:40:58.003957 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:40:58.003926    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:40:58.020066 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:40:58.020042    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:40:58.705769 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:40:58.705742    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 20:41:24.456744 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:41:24.456708    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 20:41:54.457674 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:41:54.457597    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 20:42:24.456866 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:42:24.456828    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 20:42:54.456614 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:42:54.456573    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 20:43:24.456352 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:43:24.456317    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 20:43:27.904097 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:27.904064    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717dftx"]
Oct 30 20:43:27.908807 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:27.908758    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717dftx"
Oct 30 20:43:27.977574 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:27.977540    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b9acc6d-a438-4492-8415-c911ab6d8650-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717dftx\" (UID: \"3b9acc6d-a438-4492-8415-c911ab6d8650\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717dftx"
Oct 30 20:43:27.977728 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:27.977623    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b9acc6d-a438-4492-8415-c911ab6d8650-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717dftx\" (UID: \"3b9acc6d-a438-4492-8415-c911ab6d8650\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717dftx"
Oct 30 20:43:27.977770 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:27.977719    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24xtf\" (UniqueName: \"kubernetes.io/projected/3b9acc6d-a438-4492-8415-c911ab6d8650-kube-api-access-24xtf\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717dftx\" (UID: \"3b9acc6d-a438-4492-8415-c911ab6d8650\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717dftx"
Oct 30 20:43:28.010919 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:28.010876    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 20:43:28.010919 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:28.010881    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 20:43:28.011129 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:28.010991    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-ptbhp\""
Oct 30 20:43:28.011645 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:28.011623    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717dftx"]
Oct 30 20:43:28.079022 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:28.078987    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b9acc6d-a438-4492-8415-c911ab6d8650-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717dftx\" (UID: \"3b9acc6d-a438-4492-8415-c911ab6d8650\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717dftx"
Oct 30 20:43:28.079022 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:28.079029    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24xtf\" (UniqueName: \"kubernetes.io/projected/3b9acc6d-a438-4492-8415-c911ab6d8650-kube-api-access-24xtf\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717dftx\" (UID: \"3b9acc6d-a438-4492-8415-c911ab6d8650\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717dftx"
Oct 30 20:43:28.079237 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:28.079054    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b9acc6d-a438-4492-8415-c911ab6d8650-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717dftx\" (UID: \"3b9acc6d-a438-4492-8415-c911ab6d8650\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717dftx"
Oct 30 20:43:28.079398 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:28.079378    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b9acc6d-a438-4492-8415-c911ab6d8650-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717dftx\" (UID: \"3b9acc6d-a438-4492-8415-c911ab6d8650\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717dftx"
Oct 30 20:43:28.079398 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:28.079393    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b9acc6d-a438-4492-8415-c911ab6d8650-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717dftx\" (UID: \"3b9acc6d-a438-4492-8415-c911ab6d8650\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717dftx"
Oct 30 20:43:28.137122 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:28.137082    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24xtf\" (UniqueName: \"kubernetes.io/projected/3b9acc6d-a438-4492-8415-c911ab6d8650-kube-api-access-24xtf\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717dftx\" (UID: \"3b9acc6d-a438-4492-8415-c911ab6d8650\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717dftx"
Oct 30 20:43:28.225089 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:28.225058    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717dftx"
Oct 30 20:43:28.599045 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:28.599020    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717dftx"]
Oct 30 20:43:28.601582 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:43:28.601550    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b9acc6d_a438_4492_8415_c911ab6d8650.slice/crio-86a27f2259e66e7bc1156d2edc0e217948361b180ef2080f0437bbfbb8fd4920 WatchSource:0}: Error finding container 86a27f2259e66e7bc1156d2edc0e217948361b180ef2080f0437bbfbb8fd4920: Status 404 returned error can't find the container with id 86a27f2259e66e7bc1156d2edc0e217948361b180ef2080f0437bbfbb8fd4920
Oct 30 20:43:28.936564 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:28.936531    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vtz2n"]
Oct 30 20:43:28.941073 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:28.941056    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vtz2n"
Oct 30 20:43:29.006650 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:29.006615    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-t9dfz\""
Oct 30 20:43:29.006924 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:29.006905    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vtz2n"]
Oct 30 20:43:29.088254 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:29.088223    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/54df32fa-cbda-43f1-8159-eaf9151544b8-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vtz2n\" (UID: \"54df32fa-cbda-43f1-8159-eaf9151544b8\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vtz2n"
Oct 30 20:43:29.088254 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:29.088257    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd55k\" (UniqueName: \"kubernetes.io/projected/54df32fa-cbda-43f1-8159-eaf9151544b8-kube-api-access-nd55k\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vtz2n\" (UID: \"54df32fa-cbda-43f1-8159-eaf9151544b8\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vtz2n"
Oct 30 20:43:29.088467 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:29.088284    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/54df32fa-cbda-43f1-8159-eaf9151544b8-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vtz2n\" (UID: \"54df32fa-cbda-43f1-8159-eaf9151544b8\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vtz2n"
Oct 30 20:43:29.189133 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:29.189050    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/54df32fa-cbda-43f1-8159-eaf9151544b8-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vtz2n\" (UID: \"54df32fa-cbda-43f1-8159-eaf9151544b8\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vtz2n"
Oct 30 20:43:29.189133 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:29.189087    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd55k\" (UniqueName: \"kubernetes.io/projected/54df32fa-cbda-43f1-8159-eaf9151544b8-kube-api-access-nd55k\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vtz2n\" (UID: \"54df32fa-cbda-43f1-8159-eaf9151544b8\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vtz2n"
Oct 30 20:43:29.189133 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:29.189115    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/54df32fa-cbda-43f1-8159-eaf9151544b8-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vtz2n\" (UID: \"54df32fa-cbda-43f1-8159-eaf9151544b8\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vtz2n"
Oct 30 20:43:29.189436 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:29.189418    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/54df32fa-cbda-43f1-8159-eaf9151544b8-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vtz2n\" (UID: \"54df32fa-cbda-43f1-8159-eaf9151544b8\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vtz2n"
Oct 30 20:43:29.189490 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:29.189458    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/54df32fa-cbda-43f1-8159-eaf9151544b8-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vtz2n\" (UID: \"54df32fa-cbda-43f1-8159-eaf9151544b8\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vtz2n"
Oct 30 20:43:29.218191 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:29.218160    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717dftx" event={"ID":"3b9acc6d-a438-4492-8415-c911ab6d8650","Type":"ContainerStarted","Data":"86a27f2259e66e7bc1156d2edc0e217948361b180ef2080f0437bbfbb8fd4920"}
Oct 30 20:43:29.246828 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:29.246801    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd55k\" (UniqueName: \"kubernetes.io/projected/54df32fa-cbda-43f1-8159-eaf9151544b8-kube-api-access-nd55k\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vtz2n\" (UID: \"54df32fa-cbda-43f1-8159-eaf9151544b8\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vtz2n"
Oct 30 20:43:29.256623 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:29.256606    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vtz2n"
Oct 30 20:43:29.342288 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:29.342257    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cthhsr"]
Oct 30 20:43:29.345717 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:29.345702    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cthhsr"
Oct 30 20:43:29.398237 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:29.398213    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-chq6l\""
Oct 30 20:43:29.403376 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:29.403352    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cthhsr"]
Oct 30 20:43:29.448209 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:29.448138    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ackb29"]
Oct 30 20:43:29.451631 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:29.451607    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ackb29"
Oct 30 20:43:29.492195 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:29.492167    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0bc4c78-452e-474c-b22b-68a88cbdb74d-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cthhsr\" (UID: \"b0bc4c78-452e-474c-b22b-68a88cbdb74d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cthhsr"
Oct 30 20:43:29.492337 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:29.492217    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgbwb\" (UniqueName: \"kubernetes.io/projected/b0bc4c78-452e-474c-b22b-68a88cbdb74d-kube-api-access-wgbwb\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cthhsr\" (UID: \"b0bc4c78-452e-474c-b22b-68a88cbdb74d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cthhsr"
Oct 30 20:43:29.492337 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:29.492308    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0bc4c78-452e-474c-b22b-68a88cbdb74d-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cthhsr\" (UID: \"b0bc4c78-452e-474c-b22b-68a88cbdb74d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cthhsr"
Oct 30 20:43:29.504054 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:29.504035    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-64s92\""
Oct 30 20:43:29.513827 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:29.513778    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ackb29"]
Oct 30 20:43:29.593823 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:29.593772    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0bc4c78-452e-474c-b22b-68a88cbdb74d-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cthhsr\" (UID: \"b0bc4c78-452e-474c-b22b-68a88cbdb74d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cthhsr"
Oct 30 20:43:29.594009 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:29.593849    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/55c31017-c106-4d1a-a97f-086a1584f7d0-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ackb29\" (UID: \"55c31017-c106-4d1a-a97f-086a1584f7d0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ackb29"
Oct 30 20:43:29.594009 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:29.593899    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0bc4c78-452e-474c-b22b-68a88cbdb74d-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cthhsr\" (UID: \"b0bc4c78-452e-474c-b22b-68a88cbdb74d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cthhsr"
Oct 30 20:43:29.594009 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:29.593956    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/55c31017-c106-4d1a-a97f-086a1584f7d0-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ackb29\" (UID: \"55c31017-c106-4d1a-a97f-086a1584f7d0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ackb29"
Oct 30 20:43:29.594009 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:29.593996    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgbwb\" (UniqueName: \"kubernetes.io/projected/b0bc4c78-452e-474c-b22b-68a88cbdb74d-kube-api-access-wgbwb\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cthhsr\" (UID: \"b0bc4c78-452e-474c-b22b-68a88cbdb74d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cthhsr"
Oct 30 20:43:29.594252 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:29.594047    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z65dm\" (UniqueName: \"kubernetes.io/projected/55c31017-c106-4d1a-a97f-086a1584f7d0-kube-api-access-z65dm\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ackb29\" (UID: \"55c31017-c106-4d1a-a97f-086a1584f7d0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ackb29"
Oct 30 20:43:29.594252 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:29.594224    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0bc4c78-452e-474c-b22b-68a88cbdb74d-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cthhsr\" (UID: \"b0bc4c78-452e-474c-b22b-68a88cbdb74d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cthhsr"
Oct 30 20:43:29.594431 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:29.594409    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0bc4c78-452e-474c-b22b-68a88cbdb74d-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cthhsr\" (UID: \"b0bc4c78-452e-474c-b22b-68a88cbdb74d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cthhsr"
Oct 30 20:43:29.632363 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:29.632327    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vtz2n"]
Oct 30 20:43:29.638453 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:43:29.638422    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54df32fa_cbda_43f1_8159_eaf9151544b8.slice/crio-e9172758c814a8b61445fc3450b65f4cabe1e12ea4fcbd8d8f1f003ce533cead WatchSource:0}: Error finding container e9172758c814a8b61445fc3450b65f4cabe1e12ea4fcbd8d8f1f003ce533cead: Status 404 returned error can't find the container with id e9172758c814a8b61445fc3450b65f4cabe1e12ea4fcbd8d8f1f003ce533cead
Oct 30 20:43:29.652805 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:29.652768    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgbwb\" (UniqueName: \"kubernetes.io/projected/b0bc4c78-452e-474c-b22b-68a88cbdb74d-kube-api-access-wgbwb\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cthhsr\" (UID: \"b0bc4c78-452e-474c-b22b-68a88cbdb74d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cthhsr"
Oct 30 20:43:29.661552 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:29.661530    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cthhsr"
Oct 30 20:43:29.695150 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:29.695106    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z65dm\" (UniqueName: \"kubernetes.io/projected/55c31017-c106-4d1a-a97f-086a1584f7d0-kube-api-access-z65dm\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ackb29\" (UID: \"55c31017-c106-4d1a-a97f-086a1584f7d0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ackb29"
Oct 30 20:43:29.695385 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:29.695182    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/55c31017-c106-4d1a-a97f-086a1584f7d0-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ackb29\" (UID: \"55c31017-c106-4d1a-a97f-086a1584f7d0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ackb29"
Oct 30 20:43:29.695385 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:29.695266    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/55c31017-c106-4d1a-a97f-086a1584f7d0-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ackb29\" (UID: \"55c31017-c106-4d1a-a97f-086a1584f7d0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ackb29"
Oct 30 20:43:29.695716 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:29.695670    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/55c31017-c106-4d1a-a97f-086a1584f7d0-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ackb29\" (UID: \"55c31017-c106-4d1a-a97f-086a1584f7d0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ackb29"
Oct 30 20:43:29.696258 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:29.696235    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/55c31017-c106-4d1a-a97f-086a1584f7d0-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ackb29\" (UID: \"55c31017-c106-4d1a-a97f-086a1584f7d0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ackb29"
Oct 30 20:43:29.761588 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:29.761557    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z65dm\" (UniqueName: \"kubernetes.io/projected/55c31017-c106-4d1a-a97f-086a1584f7d0-kube-api-access-z65dm\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ackb29\" (UID: \"55c31017-c106-4d1a-a97f-086a1584f7d0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ackb29"
Oct 30 20:43:29.767364 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:29.767333    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ackb29"
Oct 30 20:43:30.046767 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:30.046697    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cthhsr"]
Oct 30 20:43:30.049697 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:43:30.049671    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0bc4c78_452e_474c_b22b_68a88cbdb74d.slice/crio-80310e27db16535e474c3a81ccb6fb8397cec3a80f71518cc2cdbc1670f0edc3 WatchSource:0}: Error finding container 80310e27db16535e474c3a81ccb6fb8397cec3a80f71518cc2cdbc1670f0edc3: Status 404 returned error can't find the container with id 80310e27db16535e474c3a81ccb6fb8397cec3a80f71518cc2cdbc1670f0edc3
Oct 30 20:43:30.154779 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:30.154749    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ackb29"]
Oct 30 20:43:30.158766 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:43:30.158737    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55c31017_c106_4d1a_a97f_086a1584f7d0.slice/crio-90993c9d49047e87660a8b0caab9bcc33dfea05e1f24863abd682a10205da515 WatchSource:0}: Error finding container 90993c9d49047e87660a8b0caab9bcc33dfea05e1f24863abd682a10205da515: Status 404 returned error can't find the container with id 90993c9d49047e87660a8b0caab9bcc33dfea05e1f24863abd682a10205da515
Oct 30 20:43:30.224685 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:30.224645    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cthhsr" event={"ID":"b0bc4c78-452e-474c-b22b-68a88cbdb74d","Type":"ContainerStarted","Data":"80310e27db16535e474c3a81ccb6fb8397cec3a80f71518cc2cdbc1670f0edc3"}
Oct 30 20:43:30.225919 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:30.225897    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vtz2n" event={"ID":"54df32fa-cbda-43f1-8159-eaf9151544b8","Type":"ContainerStarted","Data":"e9172758c814a8b61445fc3450b65f4cabe1e12ea4fcbd8d8f1f003ce533cead"}
Oct 30 20:43:30.226989 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:30.226969    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ackb29" event={"ID":"55c31017-c106-4d1a-a97f-086a1584f7d0","Type":"ContainerStarted","Data":"90993c9d49047e87660a8b0caab9bcc33dfea05e1f24863abd682a10205da515"}
Oct 30 20:43:34.761763 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:34.761718    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-msv5c"]
Oct 30 20:43:34.765770 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:34.765745    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-msv5c"
Oct 30 20:43:34.818115 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:34.818086    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 20:43:34.818115 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:34.818086    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 20:43:34.845673 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:34.845639    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45gg4\" (UniqueName: \"kubernetes.io/projected/2dd002cb-a7d3-4b19-bbf0-f3d2a04d9666-kube-api-access-45gg4\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-msv5c\" (UID: \"2dd002cb-a7d3-4b19-bbf0-f3d2a04d9666\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-msv5c"
Oct 30 20:43:34.845857 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:34.845754    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dd002cb-a7d3-4b19-bbf0-f3d2a04d9666-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-msv5c\" (UID: \"2dd002cb-a7d3-4b19-bbf0-f3d2a04d9666\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-msv5c"
Oct 30 20:43:34.845926 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:34.845853    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dd002cb-a7d3-4b19-bbf0-f3d2a04d9666-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-msv5c\" (UID: \"2dd002cb-a7d3-4b19-bbf0-f3d2a04d9666\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-msv5c"
Oct 30 20:43:34.846066 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:34.846007    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dd002cb-a7d3-4b19-bbf0-f3d2a04d9666-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-msv5c\" (UID: \"2dd002cb-a7d3-4b19-bbf0-f3d2a04d9666\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-msv5c"
Oct 30 20:43:34.846188 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:34.846133    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd002cb-a7d3-4b19-bbf0-f3d2a04d9666-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-msv5c\" (UID: \"2dd002cb-a7d3-4b19-bbf0-f3d2a04d9666\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-msv5c"
Oct 30 20:43:34.846249 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:34.846230    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dd002cb-a7d3-4b19-bbf0-f3d2a04d9666-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-msv5c\" (UID: \"2dd002cb-a7d3-4b19-bbf0-f3d2a04d9666\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-msv5c"
Oct 30 20:43:34.846315 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:34.846256    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dd002cb-a7d3-4b19-bbf0-f3d2a04d9666-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-msv5c\" (UID: \"2dd002cb-a7d3-4b19-bbf0-f3d2a04d9666\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-msv5c"
Oct 30 20:43:34.846374 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:34.846319    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dd002cb-a7d3-4b19-bbf0-f3d2a04d9666-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-msv5c\" (UID: \"2dd002cb-a7d3-4b19-bbf0-f3d2a04d9666\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-msv5c"
Oct 30 20:43:34.846428 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:34.846377    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd002cb-a7d3-4b19-bbf0-f3d2a04d9666-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-msv5c\" (UID: \"2dd002cb-a7d3-4b19-bbf0-f3d2a04d9666\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-msv5c"
Oct 30 20:43:34.851069 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:34.851044    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 20:43:34.867338 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:34.867316    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-4qhb2\""
Oct 30 20:43:34.868437 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:34.868417    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-msv5c"]
Oct 30 20:43:34.948528 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:34.948003    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dd002cb-a7d3-4b19-bbf0-f3d2a04d9666-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-msv5c\" (UID: \"2dd002cb-a7d3-4b19-bbf0-f3d2a04d9666\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-msv5c"
Oct 30 20:43:34.948528 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:34.948041    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dd002cb-a7d3-4b19-bbf0-f3d2a04d9666-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-msv5c\" (UID: \"2dd002cb-a7d3-4b19-bbf0-f3d2a04d9666\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-msv5c"
Oct 30 20:43:34.948528 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:34.948079    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dd002cb-a7d3-4b19-bbf0-f3d2a04d9666-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-msv5c\" (UID: \"2dd002cb-a7d3-4b19-bbf0-f3d2a04d9666\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-msv5c"
Oct 30 20:43:34.948528 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:34.948229    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd002cb-a7d3-4b19-bbf0-f3d2a04d9666-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-msv5c\" (UID: \"2dd002cb-a7d3-4b19-bbf0-f3d2a04d9666\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-msv5c"
Oct 30 20:43:34.948528 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:34.948284    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dd002cb-a7d3-4b19-bbf0-f3d2a04d9666-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-msv5c\" (UID: \"2dd002cb-a7d3-4b19-bbf0-f3d2a04d9666\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-msv5c"
Oct 30 20:43:34.948528 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:34.948308    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dd002cb-a7d3-4b19-bbf0-f3d2a04d9666-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-msv5c\" (UID: \"2dd002cb-a7d3-4b19-bbf0-f3d2a04d9666\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-msv5c"
Oct 30 20:43:34.948528 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:34.948332    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dd002cb-a7d3-4b19-bbf0-f3d2a04d9666-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-msv5c\" (UID: \"2dd002cb-a7d3-4b19-bbf0-f3d2a04d9666\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-msv5c"
Oct 30 20:43:34.948528 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:34.948384    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd002cb-a7d3-4b19-bbf0-f3d2a04d9666-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-msv5c\" (UID: \"2dd002cb-a7d3-4b19-bbf0-f3d2a04d9666\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-msv5c"
Oct 30 20:43:34.948528 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:34.948428    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45gg4\" (UniqueName: \"kubernetes.io/projected/2dd002cb-a7d3-4b19-bbf0-f3d2a04d9666-kube-api-access-45gg4\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-msv5c\" (UID: \"2dd002cb-a7d3-4b19-bbf0-f3d2a04d9666\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-msv5c"
Oct 30 20:43:34.949357 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:34.949156    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd002cb-a7d3-4b19-bbf0-f3d2a04d9666-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-msv5c\" (UID: \"2dd002cb-a7d3-4b19-bbf0-f3d2a04d9666\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-msv5c"
Oct 30 20:43:34.950394 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:34.950367    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd002cb-a7d3-4b19-bbf0-f3d2a04d9666-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-msv5c\" (UID: \"2dd002cb-a7d3-4b19-bbf0-f3d2a04d9666\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-msv5c"
Oct 30 20:43:34.950531 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:34.950508    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dd002cb-a7d3-4b19-bbf0-f3d2a04d9666-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-msv5c\" (UID: \"2dd002cb-a7d3-4b19-bbf0-f3d2a04d9666\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-msv5c"
Oct 30 20:43:34.950721 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:34.950669    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dd002cb-a7d3-4b19-bbf0-f3d2a04d9666-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-msv5c\" (UID: \"2dd002cb-a7d3-4b19-bbf0-f3d2a04d9666\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-msv5c"
Oct 30 20:43:34.951228 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:34.951171    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dd002cb-a7d3-4b19-bbf0-f3d2a04d9666-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-msv5c\" (UID: \"2dd002cb-a7d3-4b19-bbf0-f3d2a04d9666\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-msv5c"
Oct 30 20:43:34.953867 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:34.953821    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dd002cb-a7d3-4b19-bbf0-f3d2a04d9666-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-msv5c\" (UID: \"2dd002cb-a7d3-4b19-bbf0-f3d2a04d9666\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-msv5c"
Oct 30 20:43:34.956633 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:34.956573    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dd002cb-a7d3-4b19-bbf0-f3d2a04d9666-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-msv5c\" (UID: \"2dd002cb-a7d3-4b19-bbf0-f3d2a04d9666\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-msv5c"
Oct 30 20:43:35.006452 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:35.006397    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45gg4\" (UniqueName: \"kubernetes.io/projected/2dd002cb-a7d3-4b19-bbf0-f3d2a04d9666-kube-api-access-45gg4\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-msv5c\" (UID: \"2dd002cb-a7d3-4b19-bbf0-f3d2a04d9666\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-msv5c"
Oct 30 20:43:35.007048 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:35.006991    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dd002cb-a7d3-4b19-bbf0-f3d2a04d9666-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-msv5c\" (UID: \"2dd002cb-a7d3-4b19-bbf0-f3d2a04d9666\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-msv5c"
Oct 30 20:43:35.085676 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:35.085609    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-msv5c"
Oct 30 20:43:35.166849 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:35.166822    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqskt6"]
Oct 30 20:43:35.174994 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:35.174969    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqskt6"
Oct 30 20:43:35.200028 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:35.199997    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72c6f2"]
Oct 30 20:43:35.203760 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:35.203736    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72c6f2"
Oct 30 20:43:35.230327 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:35.230294    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bjvts"]
Oct 30 20:43:35.234491 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:35.234159    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bjvts"
Oct 30 20:43:35.236187 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:35.236135    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqskt6"]
Oct 30 20:43:35.264455 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:35.264429    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72c6f2"]
Oct 30 20:43:35.292458 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:35.292397    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bjvts"]
Oct 30 20:43:35.298268 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:35.295931    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-kglqw\""
Oct 30 20:43:35.298268 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:35.295988    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-jcnhp\""
Oct 30 20:43:35.352822 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:35.352676    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89dn6\" (UniqueName: \"kubernetes.io/projected/a802da95-53c9-4794-bf13-1d8ea25f28a7-kube-api-access-89dn6\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqskt6\" (UID: \"a802da95-53c9-4794-bf13-1d8ea25f28a7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqskt6"
Oct 30 20:43:35.352822 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:35.352726    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/13b57ffb-5feb-4441-8bf1-ee06e28999cf-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72c6f2\" (UID: \"13b57ffb-5feb-4441-8bf1-ee06e28999cf\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72c6f2"
Oct 30 20:43:35.353018 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:35.352832    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8387a3d9-8b95-4287-a98c-46fe0b2b17da-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bjvts\" (UID: \"8387a3d9-8b95-4287-a98c-46fe0b2b17da\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bjvts"
Oct 30 20:43:35.353018 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:35.352884    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a802da95-53c9-4794-bf13-1d8ea25f28a7-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqskt6\" (UID: \"a802da95-53c9-4794-bf13-1d8ea25f28a7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqskt6"
Oct 30 20:43:35.353018 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:35.352980    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/13b57ffb-5feb-4441-8bf1-ee06e28999cf-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72c6f2\" (UID: \"13b57ffb-5feb-4441-8bf1-ee06e28999cf\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72c6f2"
Oct 30 20:43:35.353116 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:35.353026    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8387a3d9-8b95-4287-a98c-46fe0b2b17da-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bjvts\" (UID: \"8387a3d9-8b95-4287-a98c-46fe0b2b17da\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bjvts"
Oct 30 20:43:35.353116 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:35.353059    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnr9b\" (UniqueName: \"kubernetes.io/projected/8387a3d9-8b95-4287-a98c-46fe0b2b17da-kube-api-access-tnr9b\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bjvts\" (UID: \"8387a3d9-8b95-4287-a98c-46fe0b2b17da\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bjvts"
Oct 30 20:43:35.353116 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:35.353089    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a802da95-53c9-4794-bf13-1d8ea25f28a7-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqskt6\" (UID: \"a802da95-53c9-4794-bf13-1d8ea25f28a7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqskt6"
Oct 30 20:43:35.353207 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:35.353143    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf7w8\" (UniqueName: \"kubernetes.io/projected/13b57ffb-5feb-4441-8bf1-ee06e28999cf-kube-api-access-mf7w8\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72c6f2\" (UID: \"13b57ffb-5feb-4441-8bf1-ee06e28999cf\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72c6f2"
Oct 30 20:43:35.455055 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:35.454877    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8387a3d9-8b95-4287-a98c-46fe0b2b17da-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bjvts\" (UID: \"8387a3d9-8b95-4287-a98c-46fe0b2b17da\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bjvts"
Oct 30 20:43:35.455055 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:35.454921    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnr9b\" (UniqueName: \"kubernetes.io/projected/8387a3d9-8b95-4287-a98c-46fe0b2b17da-kube-api-access-tnr9b\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bjvts\" (UID: \"8387a3d9-8b95-4287-a98c-46fe0b2b17da\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bjvts"
Oct 30 20:43:35.455055 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:35.454946    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a802da95-53c9-4794-bf13-1d8ea25f28a7-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqskt6\" (UID: \"a802da95-53c9-4794-bf13-1d8ea25f28a7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqskt6"
Oct 30 20:43:35.455055 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:35.454983    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf7w8\" (UniqueName: \"kubernetes.io/projected/13b57ffb-5feb-4441-8bf1-ee06e28999cf-kube-api-access-mf7w8\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72c6f2\" (UID: \"13b57ffb-5feb-4441-8bf1-ee06e28999cf\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72c6f2"
Oct 30 20:43:35.455055 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:35.455047    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89dn6\" (UniqueName: \"kubernetes.io/projected/a802da95-53c9-4794-bf13-1d8ea25f28a7-kube-api-access-89dn6\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqskt6\" (UID: \"a802da95-53c9-4794-bf13-1d8ea25f28a7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqskt6"
Oct 30 20:43:35.455450 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:35.455080    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/13b57ffb-5feb-4441-8bf1-ee06e28999cf-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72c6f2\" (UID: \"13b57ffb-5feb-4441-8bf1-ee06e28999cf\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72c6f2"
Oct 30 20:43:35.455450 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:35.455116    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8387a3d9-8b95-4287-a98c-46fe0b2b17da-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bjvts\" (UID: \"8387a3d9-8b95-4287-a98c-46fe0b2b17da\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bjvts"
Oct 30 20:43:35.455450 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:35.455165    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a802da95-53c9-4794-bf13-1d8ea25f28a7-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqskt6\" (UID: \"a802da95-53c9-4794-bf13-1d8ea25f28a7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqskt6"
Oct 30 20:43:35.455450 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:35.455206    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/13b57ffb-5feb-4441-8bf1-ee06e28999cf-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72c6f2\" (UID: \"13b57ffb-5feb-4441-8bf1-ee06e28999cf\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72c6f2"
Oct 30 20:43:35.455653 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:35.455561    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/13b57ffb-5feb-4441-8bf1-ee06e28999cf-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72c6f2\" (UID: \"13b57ffb-5feb-4441-8bf1-ee06e28999cf\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72c6f2"
Oct 30 20:43:35.456139 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:35.455878    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8387a3d9-8b95-4287-a98c-46fe0b2b17da-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bjvts\" (UID: \"8387a3d9-8b95-4287-a98c-46fe0b2b17da\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bjvts"
Oct 30 20:43:35.456504 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:35.456481    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a802da95-53c9-4794-bf13-1d8ea25f28a7-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqskt6\" (UID: \"a802da95-53c9-4794-bf13-1d8ea25f28a7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqskt6"
Oct 30 20:43:35.457124 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:35.457091    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/13b57ffb-5feb-4441-8bf1-ee06e28999cf-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72c6f2\" (UID: \"13b57ffb-5feb-4441-8bf1-ee06e28999cf\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72c6f2"
Oct 30 20:43:35.457337 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:35.457320    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8387a3d9-8b95-4287-a98c-46fe0b2b17da-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bjvts\" (UID: \"8387a3d9-8b95-4287-a98c-46fe0b2b17da\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bjvts"
Oct 30 20:43:35.457527 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:35.457508    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a802da95-53c9-4794-bf13-1d8ea25f28a7-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqskt6\" (UID: \"a802da95-53c9-4794-bf13-1d8ea25f28a7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqskt6"
Oct 30 20:43:35.514270 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:35.514211    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf7w8\" (UniqueName: \"kubernetes.io/projected/13b57ffb-5feb-4441-8bf1-ee06e28999cf-kube-api-access-mf7w8\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72c6f2\" (UID: \"13b57ffb-5feb-4441-8bf1-ee06e28999cf\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72c6f2"
Oct 30 20:43:35.514539 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:35.514503    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89dn6\" (UniqueName: \"kubernetes.io/projected/a802da95-53c9-4794-bf13-1d8ea25f28a7-kube-api-access-89dn6\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqskt6\" (UID: \"a802da95-53c9-4794-bf13-1d8ea25f28a7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqskt6"
Oct 30 20:43:35.515993 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:35.515947    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnr9b\" (UniqueName: \"kubernetes.io/projected/8387a3d9-8b95-4287-a98c-46fe0b2b17da-kube-api-access-tnr9b\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bjvts\" (UID: \"8387a3d9-8b95-4287-a98c-46fe0b2b17da\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bjvts"
Oct 30 20:43:35.529578 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:35.524227    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72c6f2"
Oct 30 20:43:35.529578 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:35.529244    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-msv5c"]
Oct 30 20:43:35.535337 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:43:35.535307    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dd002cb_a7d3_4b19_bbf0_f3d2a04d9666.slice/crio-86daa6a97f821b858960c2ae339094101812d89578bd39205baf438dc0def706 WatchSource:0}: Error finding container 86daa6a97f821b858960c2ae339094101812d89578bd39205baf438dc0def706: Status 404 returned error can't find the container with id 86daa6a97f821b858960c2ae339094101812d89578bd39205baf438dc0def706
Oct 30 20:43:35.556758 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:35.556523    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bjvts"
Oct 30 20:43:35.802279 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:35.802240    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqskt6"
Oct 30 20:43:35.948707 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:35.948677    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72c6f2"]
Oct 30 20:43:35.951555 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:43:35.951523    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13b57ffb_5feb_4441_8bf1_ee06e28999cf.slice/crio-3432ff5018328405f7deb7080321a3eaaed3bc0a8e64edd16e4e7f4147d1b874 WatchSource:0}: Error finding container 3432ff5018328405f7deb7080321a3eaaed3bc0a8e64edd16e4e7f4147d1b874: Status 404 returned error can't find the container with id 3432ff5018328405f7deb7080321a3eaaed3bc0a8e64edd16e4e7f4147d1b874
Oct 30 20:43:35.998489 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:35.998454    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bjvts"]
Oct 30 20:43:36.001906 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:43:36.001878    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8387a3d9_8b95_4287_a98c_46fe0b2b17da.slice/crio-197b444f8f349c89402756914d04a8963e5df3eee67caaebda9780068adaf15b WatchSource:0}: Error finding container 197b444f8f349c89402756914d04a8963e5df3eee67caaebda9780068adaf15b: Status 404 returned error can't find the container with id 197b444f8f349c89402756914d04a8963e5df3eee67caaebda9780068adaf15b
Oct 30 20:43:36.201277 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:36.201201    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqskt6"]
Oct 30 20:43:36.206064 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:43:36.206031    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda802da95_53c9_4794_bf13_1d8ea25f28a7.slice/crio-71d74db9088057d4e6327ec15e5a3e0e93b3e97610f26b967f3e84898e222e84 WatchSource:0}: Error finding container 71d74db9088057d4e6327ec15e5a3e0e93b3e97610f26b967f3e84898e222e84: Status 404 returned error can't find the container with id 71d74db9088057d4e6327ec15e5a3e0e93b3e97610f26b967f3e84898e222e84
Oct 30 20:43:36.265405 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:36.265368    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bjvts" event={"ID":"8387a3d9-8b95-4287-a98c-46fe0b2b17da","Type":"ContainerStarted","Data":"197b444f8f349c89402756914d04a8963e5df3eee67caaebda9780068adaf15b"}
Oct 30 20:43:36.267041 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:36.266980    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72c6f2" event={"ID":"13b57ffb-5feb-4441-8bf1-ee06e28999cf","Type":"ContainerStarted","Data":"3432ff5018328405f7deb7080321a3eaaed3bc0a8e64edd16e4e7f4147d1b874"}
Oct 30 20:43:36.268638 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:36.268575    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqskt6" event={"ID":"a802da95-53c9-4794-bf13-1d8ea25f28a7","Type":"ContainerStarted","Data":"71d74db9088057d4e6327ec15e5a3e0e93b3e97610f26b967f3e84898e222e84"}
Oct 30 20:43:36.270126 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:36.270059    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-msv5c" event={"ID":"2dd002cb-a7d3-4b19-bbf0-f3d2a04d9666","Type":"ContainerStarted","Data":"86daa6a97f821b858960c2ae339094101812d89578bd39205baf438dc0def706"}
Oct 30 20:43:37.074888 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:37.074782    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d4597073-8ff0-4a4e-933c-c64e934fe977/prometheus/0.log"
Oct 30 20:43:37.076998 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:37.076776    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d4597073-8ff0-4a4e-933c-c64e934fe977/prometheus/0.log"
Oct 30 20:43:37.106586 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:37.106560    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r7lpf_764461d5-e0b2-4856-a97e-0c5d3d34a497/ovn-acl-logging/0.log"
Oct 30 20:43:37.109756 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:37.109688    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r7lpf_764461d5-e0b2-4856-a97e-0c5d3d34a497/ovn-acl-logging/0.log"
Oct 30 20:43:41.323343 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:41.323291    5713 generic.go:356] "Generic (PLEG): container finished" podID="55c31017-c106-4d1a-a97f-086a1584f7d0" containerID="6ddc374ea03205d3140e61ae98bcae1ccb7a57d9656682fc2be5b83e7784ccb5" exitCode=0
Oct 30 20:43:41.323851 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:41.323367    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ackb29" event={"ID":"55c31017-c106-4d1a-a97f-086a1584f7d0","Type":"ContainerDied","Data":"6ddc374ea03205d3140e61ae98bcae1ccb7a57d9656682fc2be5b83e7784ccb5"}
Oct 30 20:43:42.331318 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:42.331273    5713 generic.go:356] "Generic (PLEG): container finished" podID="3b9acc6d-a438-4492-8415-c911ab6d8650" containerID="35d4ba0da07bbf94793b96b415f8805e76736dd97b8123ac3cb7d0fa8668b688" exitCode=0
Oct 30 20:43:42.331779 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:42.331358    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717dftx" event={"ID":"3b9acc6d-a438-4492-8415-c911ab6d8650","Type":"ContainerDied","Data":"35d4ba0da07bbf94793b96b415f8805e76736dd97b8123ac3cb7d0fa8668b688"}
Oct 30 20:43:43.165558 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:43.165519    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 20:43:43.165686 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:43.165606    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 20:43:43.165686 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:43.165647    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 20:43:43.350363 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:43.349833    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-msv5c" event={"ID":"2dd002cb-a7d3-4b19-bbf0-f3d2a04d9666","Type":"ContainerStarted","Data":"837f5e4515ea2954cb01019b5f16e7682f094dadc4eb89a4b05945c760102a1f"}
Oct 30 20:43:44.086625 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:44.086556    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-msv5c"
Oct 30 20:43:44.092474 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:44.092450    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-msv5c"
Oct 30 20:43:44.210783 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:44.210725    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-msv5c" podStartSLOduration=2.6002303209999997 podStartE2EDuration="10.210706024s" totalImagesPullingTime="7.610475703s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:43:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:43:43.557030069 +0000 UTC m=+306.621806471" watchObservedRunningTime="2026-10-30 20:43:44.210706024 +0000 UTC m=+307.275482424"
Oct 30 20:43:44.359035 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:44.358807    5713 generic.go:356] "Generic (PLEG): container finished" podID="54df32fa-cbda-43f1-8159-eaf9151544b8" containerID="1942a7b9cfb3b67171e3dbf6c3815e0060b73c2a7b73579f3fa8aaf6b2e61656" exitCode=0
Oct 30 20:43:44.359035 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:44.358917    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vtz2n" event={"ID":"54df32fa-cbda-43f1-8159-eaf9151544b8","Type":"ContainerDied","Data":"1942a7b9cfb3b67171e3dbf6c3815e0060b73c2a7b73579f3fa8aaf6b2e61656"}
Oct 30 20:43:44.363288 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:44.363264    5713 generic.go:356] "Generic (PLEG): container finished" podID="b0bc4c78-452e-474c-b22b-68a88cbdb74d" containerID="99281d41da2cb48061b1f711c6a5fe4c4f1b79115692b1d78b612b4617d7bca7" exitCode=0
Oct 30 20:43:44.363407 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:44.363387    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cthhsr" event={"ID":"b0bc4c78-452e-474c-b22b-68a88cbdb74d","Type":"ContainerDied","Data":"99281d41da2cb48061b1f711c6a5fe4c4f1b79115692b1d78b612b4617d7bca7"}
Oct 30 20:43:44.366276 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:44.366253    5713 generic.go:356] "Generic (PLEG): container finished" podID="55c31017-c106-4d1a-a97f-086a1584f7d0" containerID="b1ec4fbb1aeebe0ef57787135a907e14dc0a9f3a0d21ee948f5c27ef2baa69d2" exitCode=0
Oct 30 20:43:44.366469 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:44.366398    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ackb29" event={"ID":"55c31017-c106-4d1a-a97f-086a1584f7d0","Type":"ContainerDied","Data":"b1ec4fbb1aeebe0ef57787135a907e14dc0a9f3a0d21ee948f5c27ef2baa69d2"}
Oct 30 20:43:44.370271 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:44.370183    5713 generic.go:356] "Generic (PLEG): container finished" podID="3b9acc6d-a438-4492-8415-c911ab6d8650" containerID="6fe8a767dd770a4dc951f1864d9bea8e05d47f1959111dc4072a1d8f8db146e4" exitCode=0
Oct 30 20:43:44.370461 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:44.370418    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717dftx" event={"ID":"3b9acc6d-a438-4492-8415-c911ab6d8650","Type":"ContainerDied","Data":"6fe8a767dd770a4dc951f1864d9bea8e05d47f1959111dc4072a1d8f8db146e4"}
Oct 30 20:43:44.370815 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:44.370714    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-msv5c"
Oct 30 20:43:44.372892 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:44.371869    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-msv5c"
Oct 30 20:43:45.377624 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:45.377587    5713 generic.go:356] "Generic (PLEG): container finished" podID="13b57ffb-5feb-4441-8bf1-ee06e28999cf" containerID="342a3db47bd0aa7ce809fca537df8a3c8f9fde2cd34dc033c0f70bcc88aa853a" exitCode=0
Oct 30 20:43:45.378091 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:45.377743    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72c6f2" event={"ID":"13b57ffb-5feb-4441-8bf1-ee06e28999cf","Type":"ContainerDied","Data":"342a3db47bd0aa7ce809fca537df8a3c8f9fde2cd34dc033c0f70bcc88aa853a"}
Oct 30 20:43:45.380092 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:45.380063    5713 generic.go:356] "Generic (PLEG): container finished" podID="a802da95-53c9-4794-bf13-1d8ea25f28a7" containerID="b93d6b7fb1a227f45a8b5d86cf3720b896cf56d4d76b6f67899bba14e369ba84" exitCode=0
Oct 30 20:43:45.380205 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:45.380183    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqskt6" event={"ID":"a802da95-53c9-4794-bf13-1d8ea25f28a7","Type":"ContainerDied","Data":"b93d6b7fb1a227f45a8b5d86cf3720b896cf56d4d76b6f67899bba14e369ba84"}
Oct 30 20:43:45.383111 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:45.383088    5713 generic.go:356] "Generic (PLEG): container finished" podID="8387a3d9-8b95-4287-a98c-46fe0b2b17da" containerID="1ec58b388b84d85a44d6683e08d963dbd3c05be41e91e113ae6d044e5aa23adf" exitCode=0
Oct 30 20:43:45.383198 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:45.383176    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bjvts" event={"ID":"8387a3d9-8b95-4287-a98c-46fe0b2b17da","Type":"ContainerDied","Data":"1ec58b388b84d85a44d6683e08d963dbd3c05be41e91e113ae6d044e5aa23adf"}
Oct 30 20:43:46.392441 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:46.392399    5713 generic.go:356] "Generic (PLEG): container finished" podID="a802da95-53c9-4794-bf13-1d8ea25f28a7" containerID="1768e0707a9117d7926efc6c842ffc6d132fd6e2656a863726af07f81989a0f6" exitCode=0
Oct 30 20:43:46.393006 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:46.392533    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqskt6" event={"ID":"a802da95-53c9-4794-bf13-1d8ea25f28a7","Type":"ContainerDied","Data":"1768e0707a9117d7926efc6c842ffc6d132fd6e2656a863726af07f81989a0f6"}
Oct 30 20:43:46.395963 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:46.395849    5713 generic.go:356] "Generic (PLEG): container finished" podID="b0bc4c78-452e-474c-b22b-68a88cbdb74d" containerID="d2d4a7a592dafa914015e3b6de770b48a09c14702c8e11a4d53557cafc5cc3b9" exitCode=0
Oct 30 20:43:46.395963 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:46.395953    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cthhsr" event={"ID":"b0bc4c78-452e-474c-b22b-68a88cbdb74d","Type":"ContainerDied","Data":"d2d4a7a592dafa914015e3b6de770b48a09c14702c8e11a4d53557cafc5cc3b9"}
Oct 30 20:43:47.410864 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:47.410619    5713 generic.go:356] "Generic (PLEG): container finished" podID="8387a3d9-8b95-4287-a98c-46fe0b2b17da" containerID="e7f8815003f0522d0bcba5edc8db3ece71640ddbea81de7a8389b39cd0b55114" exitCode=0
Oct 30 20:43:47.410864 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:47.410736    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bjvts" event={"ID":"8387a3d9-8b95-4287-a98c-46fe0b2b17da","Type":"ContainerDied","Data":"e7f8815003f0522d0bcba5edc8db3ece71640ddbea81de7a8389b39cd0b55114"}
Oct 30 20:43:47.418810 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:47.418165    5713 generic.go:356] "Generic (PLEG): container finished" podID="13b57ffb-5feb-4441-8bf1-ee06e28999cf" containerID="4db2595f7c08f0e0c458f45bc7ccce206307b6cffec1279c6a944ee4d4246ad8" exitCode=0
Oct 30 20:43:47.418810 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:47.418277    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72c6f2" event={"ID":"13b57ffb-5feb-4441-8bf1-ee06e28999cf","Type":"ContainerDied","Data":"4db2595f7c08f0e0c458f45bc7ccce206307b6cffec1279c6a944ee4d4246ad8"}
Oct 30 20:43:50.445860 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:50.444878    5713 generic.go:356] "Generic (PLEG): container finished" podID="54df32fa-cbda-43f1-8159-eaf9151544b8" containerID="0421d1617396546e1fefd82809344938e2c6f9b62f557de6e571ee35c7955b1e" exitCode=0
Oct 30 20:43:50.445860 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:50.444997    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vtz2n" event={"ID":"54df32fa-cbda-43f1-8159-eaf9151544b8","Type":"ContainerDied","Data":"0421d1617396546e1fefd82809344938e2c6f9b62f557de6e571ee35c7955b1e"}
Oct 30 20:43:54.482434 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:43:54.482396    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 20:43:58.515034 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:58.514994    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ackb29" event={"ID":"55c31017-c106-4d1a-a97f-086a1584f7d0","Type":"ContainerStarted","Data":"e51a692b970dff84718a0e95cb7a2650aa123f7e791fcc97ec3b64ec781bb9e7"}
Oct 30 20:43:58.519753 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:58.519722    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717dftx" event={"ID":"3b9acc6d-a438-4492-8415-c911ab6d8650","Type":"ContainerStarted","Data":"24706b37f84a25e6208c53105b27e45325695d7fee08500a412c6f3975588e5b"}
Oct 30 20:43:58.632726 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:58.632658    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ackb29" podStartSLOduration=3.120079152 podStartE2EDuration="29.632639829s" totalImagesPullingTime="26.512560677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:43:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:43:58.631358093 +0000 UTC m=+321.696134514" watchObservedRunningTime="2026-10-30 20:43:58.632639829 +0000 UTC m=+321.697416228"
Oct 30 20:43:58.749952 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:58.749386    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717dftx" podStartSLOduration=3.513782604 podStartE2EDuration="31.749367233s" totalImagesPullingTime="28.235584629s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:43:27 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:43:58.749045615 +0000 UTC m=+321.813822015" watchObservedRunningTime="2026-10-30 20:43:58.749367233 +0000 UTC m=+321.814143632"
Oct 30 20:43:59.530901 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:59.530864    5713 generic.go:356] "Generic (PLEG): container finished" podID="55c31017-c106-4d1a-a97f-086a1584f7d0" containerID="e51a692b970dff84718a0e95cb7a2650aa123f7e791fcc97ec3b64ec781bb9e7" exitCode=0
Oct 30 20:43:59.531382 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:59.530961    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ackb29" event={"ID":"55c31017-c106-4d1a-a97f-086a1584f7d0","Type":"ContainerDied","Data":"e51a692b970dff84718a0e95cb7a2650aa123f7e791fcc97ec3b64ec781bb9e7"}
Oct 30 20:43:59.534397 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:59.534192    5713 generic.go:356] "Generic (PLEG): container finished" podID="3b9acc6d-a438-4492-8415-c911ab6d8650" containerID="24706b37f84a25e6208c53105b27e45325695d7fee08500a412c6f3975588e5b" exitCode=0
Oct 30 20:43:59.534397 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:43:59.534234    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717dftx" event={"ID":"3b9acc6d-a438-4492-8415-c911ab6d8650","Type":"ContainerDied","Data":"24706b37f84a25e6208c53105b27e45325695d7fee08500a412c6f3975588e5b"}
Oct 30 20:44:00.543517 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:00.543269    5713 generic.go:356] "Generic (PLEG): container finished" podID="54df32fa-cbda-43f1-8159-eaf9151544b8" containerID="5892b9171a2524a593a66955e853f87c1318b85570e53f1a628914c7d4353dec" exitCode=0
Oct 30 20:44:00.543517 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:00.543451    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vtz2n" event={"ID":"54df32fa-cbda-43f1-8159-eaf9151544b8","Type":"ContainerDied","Data":"5892b9171a2524a593a66955e853f87c1318b85570e53f1a628914c7d4353dec"}
Oct 30 20:44:00.755449 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:00.755418    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ackb29"
Oct 30 20:44:00.763194 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:00.763102    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717dftx"
Oct 30 20:44:00.859413 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:00.859377    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55c31017-c106-4d1a-a97f-086a1584f7d0-kube-api-access-z65dm\" (UniqueName: \"kubernetes.io/projected/55c31017-c106-4d1a-a97f-086a1584f7d0-kube-api-access-z65dm\") pod \"55c31017-c106-4d1a-a97f-086a1584f7d0\" (UID: \"55c31017-c106-4d1a-a97f-086a1584f7d0\") "
Oct 30 20:44:00.859601 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:00.859431    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b9acc6d-a438-4492-8415-c911ab6d8650-kube-api-access-24xtf\" (UniqueName: \"kubernetes.io/projected/3b9acc6d-a438-4492-8415-c911ab6d8650-kube-api-access-24xtf\") pod \"3b9acc6d-a438-4492-8415-c911ab6d8650\" (UID: \"3b9acc6d-a438-4492-8415-c911ab6d8650\") "
Oct 30 20:44:00.859601 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:00.859522    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b9acc6d-a438-4492-8415-c911ab6d8650-bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b9acc6d-a438-4492-8415-c911ab6d8650-bundle\") pod \"3b9acc6d-a438-4492-8415-c911ab6d8650\" (UID: \"3b9acc6d-a438-4492-8415-c911ab6d8650\") "
Oct 30 20:44:00.859601 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:00.859568    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55c31017-c106-4d1a-a97f-086a1584f7d0-bundle\" (UniqueName: \"kubernetes.io/empty-dir/55c31017-c106-4d1a-a97f-086a1584f7d0-bundle\") pod \"55c31017-c106-4d1a-a97f-086a1584f7d0\" (UID: \"55c31017-c106-4d1a-a97f-086a1584f7d0\") "
Oct 30 20:44:00.859601 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:00.859596    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55c31017-c106-4d1a-a97f-086a1584f7d0-util\" (UniqueName: \"kubernetes.io/empty-dir/55c31017-c106-4d1a-a97f-086a1584f7d0-util\") pod \"55c31017-c106-4d1a-a97f-086a1584f7d0\" (UID: \"55c31017-c106-4d1a-a97f-086a1584f7d0\") "
Oct 30 20:44:00.859830 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:00.859652    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b9acc6d-a438-4492-8415-c911ab6d8650-util\" (UniqueName: \"kubernetes.io/empty-dir/3b9acc6d-a438-4492-8415-c911ab6d8650-util\") pod \"3b9acc6d-a438-4492-8415-c911ab6d8650\" (UID: \"3b9acc6d-a438-4492-8415-c911ab6d8650\") "
Oct 30 20:44:00.860513 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:00.860479    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b9acc6d-a438-4492-8415-c911ab6d8650-bundle" pod "3b9acc6d-a438-4492-8415-c911ab6d8650" (UID: "3b9acc6d-a438-4492-8415-c911ab6d8650"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:44:00.862454 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:00.862409    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55c31017-c106-4d1a-a97f-086a1584f7d0-bundle" pod "55c31017-c106-4d1a-a97f-086a1584f7d0" (UID: "55c31017-c106-4d1a-a97f-086a1584f7d0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:44:00.864076 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:00.864045    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b9acc6d-a438-4492-8415-c911ab6d8650-kube-api-access-24xtf" pod "3b9acc6d-a438-4492-8415-c911ab6d8650" (UID: "3b9acc6d-a438-4492-8415-c911ab6d8650"). InnerVolumeSpecName "kube-api-access-24xtf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:44:00.866463 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:00.866414    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55c31017-c106-4d1a-a97f-086a1584f7d0-util" pod "55c31017-c106-4d1a-a97f-086a1584f7d0" (UID: "55c31017-c106-4d1a-a97f-086a1584f7d0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:44:00.866575 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:00.866455    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55c31017-c106-4d1a-a97f-086a1584f7d0-kube-api-access-z65dm" pod "55c31017-c106-4d1a-a97f-086a1584f7d0" (UID: "55c31017-c106-4d1a-a97f-086a1584f7d0"). InnerVolumeSpecName "kube-api-access-z65dm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:44:00.866829 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:00.866804    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b9acc6d-a438-4492-8415-c911ab6d8650-util" pod "3b9acc6d-a438-4492-8415-c911ab6d8650" (UID: "3b9acc6d-a438-4492-8415-c911ab6d8650"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:44:00.961211 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:00.961175    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z65dm\" (UniqueName: \"kubernetes.io/projected/55c31017-c106-4d1a-a97f-086a1584f7d0-kube-api-access-z65dm\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:00.961211 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:00.961214    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24xtf\" (UniqueName: \"kubernetes.io/projected/3b9acc6d-a438-4492-8415-c911ab6d8650-kube-api-access-24xtf\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:00.961437 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:00.961231    5713 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b9acc6d-a438-4492-8415-c911ab6d8650-bundle\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:00.961437 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:00.961272    5713 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/55c31017-c106-4d1a-a97f-086a1584f7d0-bundle\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:00.961437 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:00.961285    5713 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/55c31017-c106-4d1a-a97f-086a1584f7d0-util\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:00.961437 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:00.961299    5713 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b9acc6d-a438-4492-8415-c911ab6d8650-util\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:01.551850 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:01.551812    5713 generic.go:356] "Generic (PLEG): container finished" podID="b0bc4c78-452e-474c-b22b-68a88cbdb74d" containerID="c3d2d03c2106dd435ef3ecd7fc3a93db8bc5722b0678eb50fa8b84ea4c16bf04" exitCode=0
Oct 30 20:44:01.552282 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:01.551906    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cthhsr" event={"ID":"b0bc4c78-452e-474c-b22b-68a88cbdb74d","Type":"ContainerDied","Data":"c3d2d03c2106dd435ef3ecd7fc3a93db8bc5722b0678eb50fa8b84ea4c16bf04"}
Oct 30 20:44:01.554247 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:01.554180    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ackb29" event={"ID":"55c31017-c106-4d1a-a97f-086a1584f7d0","Type":"ContainerDied","Data":"90993c9d49047e87660a8b0caab9bcc33dfea05e1f24863abd682a10205da515"}
Oct 30 20:44:01.554247 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:01.554195    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ackb29"
Oct 30 20:44:01.554247 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:01.554227    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90993c9d49047e87660a8b0caab9bcc33dfea05e1f24863abd682a10205da515"
Oct 30 20:44:01.556904 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:01.556399    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717dftx"
Oct 30 20:44:01.556904 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:01.556397    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717dftx" event={"ID":"3b9acc6d-a438-4492-8415-c911ab6d8650","Type":"ContainerDied","Data":"86a27f2259e66e7bc1156d2edc0e217948361b180ef2080f0437bbfbb8fd4920"}
Oct 30 20:44:01.556904 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:01.556525    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86a27f2259e66e7bc1156d2edc0e217948361b180ef2080f0437bbfbb8fd4920"
Oct 30 20:44:01.559760 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:01.559735    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqskt6" event={"ID":"a802da95-53c9-4794-bf13-1d8ea25f28a7","Type":"ContainerStarted","Data":"79c4df777092f74e312d4a97b15a92809875c3e28f4cde717f157e5a6ac56454"}
Oct 30 20:44:01.752571 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:01.752546    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vtz2n"
Oct 30 20:44:01.871250 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:01.871173    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54df32fa-cbda-43f1-8159-eaf9151544b8-util\" (UniqueName: \"kubernetes.io/empty-dir/54df32fa-cbda-43f1-8159-eaf9151544b8-util\") pod \"54df32fa-cbda-43f1-8159-eaf9151544b8\" (UID: \"54df32fa-cbda-43f1-8159-eaf9151544b8\") "
Oct 30 20:44:01.871250 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:01.871209    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54df32fa-cbda-43f1-8159-eaf9151544b8-kube-api-access-nd55k\" (UniqueName: \"kubernetes.io/projected/54df32fa-cbda-43f1-8159-eaf9151544b8-kube-api-access-nd55k\") pod \"54df32fa-cbda-43f1-8159-eaf9151544b8\" (UID: \"54df32fa-cbda-43f1-8159-eaf9151544b8\") "
Oct 30 20:44:01.871454 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:01.871264    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54df32fa-cbda-43f1-8159-eaf9151544b8-bundle\" (UniqueName: \"kubernetes.io/empty-dir/54df32fa-cbda-43f1-8159-eaf9151544b8-bundle\") pod \"54df32fa-cbda-43f1-8159-eaf9151544b8\" (UID: \"54df32fa-cbda-43f1-8159-eaf9151544b8\") "
Oct 30 20:44:01.871698 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:01.871666    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54df32fa-cbda-43f1-8159-eaf9151544b8-bundle" pod "54df32fa-cbda-43f1-8159-eaf9151544b8" (UID: "54df32fa-cbda-43f1-8159-eaf9151544b8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:44:01.873454 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:01.873423    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54df32fa-cbda-43f1-8159-eaf9151544b8-kube-api-access-nd55k" pod "54df32fa-cbda-43f1-8159-eaf9151544b8" (UID: "54df32fa-cbda-43f1-8159-eaf9151544b8"). InnerVolumeSpecName "kube-api-access-nd55k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:44:01.876756 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:01.876737    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54df32fa-cbda-43f1-8159-eaf9151544b8-util" pod "54df32fa-cbda-43f1-8159-eaf9151544b8" (UID: "54df32fa-cbda-43f1-8159-eaf9151544b8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:44:01.972811 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:01.972756    5713 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/54df32fa-cbda-43f1-8159-eaf9151544b8-util\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:01.972811 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:01.972781    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd55k\" (UniqueName: \"kubernetes.io/projected/54df32fa-cbda-43f1-8159-eaf9151544b8-kube-api-access-nd55k\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:01.973006 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:01.972821    5713 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/54df32fa-cbda-43f1-8159-eaf9151544b8-bundle\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:02.011808 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:02.011744    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqskt6" podStartSLOduration=3.839760171 podStartE2EDuration="27.011728037s" totalImagesPullingTime="23.171967866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:43:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:44:01.792845311 +0000 UTC m=+324.857621709" watchObservedRunningTime="2026-10-30 20:44:02.011728037 +0000 UTC m=+325.076504435"
Oct 30 20:44:02.566232 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:02.566201    5713 generic.go:356] "Generic (PLEG): container finished" podID="8387a3d9-8b95-4287-a98c-46fe0b2b17da" containerID="1a6c9d3b863dc9697a806fd5faaf7b36d0bfebf8b1f8662a4cbd0e120d83d384" exitCode=0
Oct 30 20:44:02.566651 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:02.566277    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bjvts" event={"ID":"8387a3d9-8b95-4287-a98c-46fe0b2b17da","Type":"ContainerDied","Data":"1a6c9d3b863dc9697a806fd5faaf7b36d0bfebf8b1f8662a4cbd0e120d83d384"}
Oct 30 20:44:02.568135 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:02.568115    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vtz2n" event={"ID":"54df32fa-cbda-43f1-8159-eaf9151544b8","Type":"ContainerDied","Data":"e9172758c814a8b61445fc3450b65f4cabe1e12ea4fcbd8d8f1f003ce533cead"}
Oct 30 20:44:02.568135 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:02.568129    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644vtz2n"
Oct 30 20:44:02.568322 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:02.568156    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9172758c814a8b61445fc3450b65f4cabe1e12ea4fcbd8d8f1f003ce533cead"
Oct 30 20:44:02.570221 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:02.570203    5713 generic.go:356] "Generic (PLEG): container finished" podID="13b57ffb-5feb-4441-8bf1-ee06e28999cf" containerID="86e407964a8ad5b8b4bee001418f1df1d12749f6addced7fe29c14497560541c" exitCode=0
Oct 30 20:44:02.570312 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:02.570292    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72c6f2" event={"ID":"13b57ffb-5feb-4441-8bf1-ee06e28999cf","Type":"ContainerDied","Data":"86e407964a8ad5b8b4bee001418f1df1d12749f6addced7fe29c14497560541c"}
Oct 30 20:44:02.572408 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:02.572376    5713 generic.go:356] "Generic (PLEG): container finished" podID="a802da95-53c9-4794-bf13-1d8ea25f28a7" containerID="79c4df777092f74e312d4a97b15a92809875c3e28f4cde717f157e5a6ac56454" exitCode=0
Oct 30 20:44:02.572490 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:02.572427    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqskt6" event={"ID":"a802da95-53c9-4794-bf13-1d8ea25f28a7","Type":"ContainerDied","Data":"79c4df777092f74e312d4a97b15a92809875c3e28f4cde717f157e5a6ac56454"}
Oct 30 20:44:02.704567 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:02.698580    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cthhsr"
Oct 30 20:44:02.779417 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:02.779385    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0bc4c78-452e-474c-b22b-68a88cbdb74d-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0bc4c78-452e-474c-b22b-68a88cbdb74d-bundle\") pod \"b0bc4c78-452e-474c-b22b-68a88cbdb74d\" (UID: \"b0bc4c78-452e-474c-b22b-68a88cbdb74d\") "
Oct 30 20:44:02.779618 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:02.779490    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0bc4c78-452e-474c-b22b-68a88cbdb74d-util\" (UniqueName: \"kubernetes.io/empty-dir/b0bc4c78-452e-474c-b22b-68a88cbdb74d-util\") pod \"b0bc4c78-452e-474c-b22b-68a88cbdb74d\" (UID: \"b0bc4c78-452e-474c-b22b-68a88cbdb74d\") "
Oct 30 20:44:02.779618 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:02.779573    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0bc4c78-452e-474c-b22b-68a88cbdb74d-kube-api-access-wgbwb\" (UniqueName: \"kubernetes.io/projected/b0bc4c78-452e-474c-b22b-68a88cbdb74d-kube-api-access-wgbwb\") pod \"b0bc4c78-452e-474c-b22b-68a88cbdb74d\" (UID: \"b0bc4c78-452e-474c-b22b-68a88cbdb74d\") "
Oct 30 20:44:02.780063 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:02.780031    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0bc4c78-452e-474c-b22b-68a88cbdb74d-bundle" pod "b0bc4c78-452e-474c-b22b-68a88cbdb74d" (UID: "b0bc4c78-452e-474c-b22b-68a88cbdb74d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:44:02.782321 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:02.782287    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0bc4c78-452e-474c-b22b-68a88cbdb74d-kube-api-access-wgbwb" pod "b0bc4c78-452e-474c-b22b-68a88cbdb74d" (UID: "b0bc4c78-452e-474c-b22b-68a88cbdb74d"). InnerVolumeSpecName "kube-api-access-wgbwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:44:02.786638 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:02.786612    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0bc4c78-452e-474c-b22b-68a88cbdb74d-util" pod "b0bc4c78-452e-474c-b22b-68a88cbdb74d" (UID: "b0bc4c78-452e-474c-b22b-68a88cbdb74d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:44:02.880851 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:02.880766    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgbwb\" (UniqueName: \"kubernetes.io/projected/b0bc4c78-452e-474c-b22b-68a88cbdb74d-kube-api-access-wgbwb\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:02.880851 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:02.880818    5713 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0bc4c78-452e-474c-b22b-68a88cbdb74d-bundle\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:02.880851 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:02.880829    5713 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0bc4c78-452e-474c-b22b-68a88cbdb74d-util\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:03.579063 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:03.579033    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cthhsr"
Oct 30 20:44:03.579511 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:03.579081    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cthhsr" event={"ID":"b0bc4c78-452e-474c-b22b-68a88cbdb74d","Type":"ContainerDied","Data":"80310e27db16535e474c3a81ccb6fb8397cec3a80f71518cc2cdbc1670f0edc3"}
Oct 30 20:44:03.579511 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:03.579145    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80310e27db16535e474c3a81ccb6fb8397cec3a80f71518cc2cdbc1670f0edc3"
Oct 30 20:44:03.714207 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:03.714174    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqskt6"
Oct 30 20:44:03.756565 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:03.756540    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72c6f2"
Oct 30 20:44:03.759286 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:03.759269    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bjvts"
Oct 30 20:44:03.788457 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:03.788439    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a802da95-53c9-4794-bf13-1d8ea25f28a7-kube-api-access-89dn6\" (UniqueName: \"kubernetes.io/projected/a802da95-53c9-4794-bf13-1d8ea25f28a7-kube-api-access-89dn6\") pod \"a802da95-53c9-4794-bf13-1d8ea25f28a7\" (UID: \"a802da95-53c9-4794-bf13-1d8ea25f28a7\") "
Oct 30 20:44:03.788595 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:03.788480    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a802da95-53c9-4794-bf13-1d8ea25f28a7-util\" (UniqueName: \"kubernetes.io/empty-dir/a802da95-53c9-4794-bf13-1d8ea25f28a7-util\") pod \"a802da95-53c9-4794-bf13-1d8ea25f28a7\" (UID: \"a802da95-53c9-4794-bf13-1d8ea25f28a7\") "
Oct 30 20:44:03.788595 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:03.788501    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a802da95-53c9-4794-bf13-1d8ea25f28a7-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a802da95-53c9-4794-bf13-1d8ea25f28a7-bundle\") pod \"a802da95-53c9-4794-bf13-1d8ea25f28a7\" (UID: \"a802da95-53c9-4794-bf13-1d8ea25f28a7\") "
Oct 30 20:44:03.789173 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:03.789151    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a802da95-53c9-4794-bf13-1d8ea25f28a7-bundle" pod "a802da95-53c9-4794-bf13-1d8ea25f28a7" (UID: "a802da95-53c9-4794-bf13-1d8ea25f28a7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:44:03.790838 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:03.790814    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a802da95-53c9-4794-bf13-1d8ea25f28a7-kube-api-access-89dn6" pod "a802da95-53c9-4794-bf13-1d8ea25f28a7" (UID: "a802da95-53c9-4794-bf13-1d8ea25f28a7"). InnerVolumeSpecName "kube-api-access-89dn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:44:03.792913 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:03.792890    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a802da95-53c9-4794-bf13-1d8ea25f28a7-util" pod "a802da95-53c9-4794-bf13-1d8ea25f28a7" (UID: "a802da95-53c9-4794-bf13-1d8ea25f28a7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:44:03.889503 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:03.889408    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8387a3d9-8b95-4287-a98c-46fe0b2b17da-util\" (UniqueName: \"kubernetes.io/empty-dir/8387a3d9-8b95-4287-a98c-46fe0b2b17da-util\") pod \"8387a3d9-8b95-4287-a98c-46fe0b2b17da\" (UID: \"8387a3d9-8b95-4287-a98c-46fe0b2b17da\") "
Oct 30 20:44:03.889503 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:03.889456    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8387a3d9-8b95-4287-a98c-46fe0b2b17da-kube-api-access-tnr9b\" (UniqueName: \"kubernetes.io/projected/8387a3d9-8b95-4287-a98c-46fe0b2b17da-kube-api-access-tnr9b\") pod \"8387a3d9-8b95-4287-a98c-46fe0b2b17da\" (UID: \"8387a3d9-8b95-4287-a98c-46fe0b2b17da\") "
Oct 30 20:44:03.889503 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:03.889491    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13b57ffb-5feb-4441-8bf1-ee06e28999cf-kube-api-access-mf7w8\" (UniqueName: \"kubernetes.io/projected/13b57ffb-5feb-4441-8bf1-ee06e28999cf-kube-api-access-mf7w8\") pod \"13b57ffb-5feb-4441-8bf1-ee06e28999cf\" (UID: \"13b57ffb-5feb-4441-8bf1-ee06e28999cf\") "
Oct 30 20:44:03.889829 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:03.889516    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13b57ffb-5feb-4441-8bf1-ee06e28999cf-bundle\" (UniqueName: \"kubernetes.io/empty-dir/13b57ffb-5feb-4441-8bf1-ee06e28999cf-bundle\") pod \"13b57ffb-5feb-4441-8bf1-ee06e28999cf\" (UID: \"13b57ffb-5feb-4441-8bf1-ee06e28999cf\") "
Oct 30 20:44:03.889829 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:03.889534    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8387a3d9-8b95-4287-a98c-46fe0b2b17da-bundle\" (UniqueName: \"kubernetes.io/empty-dir/8387a3d9-8b95-4287-a98c-46fe0b2b17da-bundle\") pod \"8387a3d9-8b95-4287-a98c-46fe0b2b17da\" (UID: \"8387a3d9-8b95-4287-a98c-46fe0b2b17da\") "
Oct 30 20:44:03.889829 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:03.889565    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13b57ffb-5feb-4441-8bf1-ee06e28999cf-util\" (UniqueName: \"kubernetes.io/empty-dir/13b57ffb-5feb-4441-8bf1-ee06e28999cf-util\") pod \"13b57ffb-5feb-4441-8bf1-ee06e28999cf\" (UID: \"13b57ffb-5feb-4441-8bf1-ee06e28999cf\") "
Oct 30 20:44:03.889829 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:03.889815    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89dn6\" (UniqueName: \"kubernetes.io/projected/a802da95-53c9-4794-bf13-1d8ea25f28a7-kube-api-access-89dn6\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:03.890029 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:03.889835    5713 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a802da95-53c9-4794-bf13-1d8ea25f28a7-util\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:03.890029 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:03.889850    5713 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a802da95-53c9-4794-bf13-1d8ea25f28a7-bundle\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:03.890204 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:03.890171    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13b57ffb-5feb-4441-8bf1-ee06e28999cf-bundle" pod "13b57ffb-5feb-4441-8bf1-ee06e28999cf" (UID: "13b57ffb-5feb-4441-8bf1-ee06e28999cf"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:44:03.890285 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:03.890173    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8387a3d9-8b95-4287-a98c-46fe0b2b17da-bundle" pod "8387a3d9-8b95-4287-a98c-46fe0b2b17da" (UID: "8387a3d9-8b95-4287-a98c-46fe0b2b17da"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:44:03.891889 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:03.891856    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8387a3d9-8b95-4287-a98c-46fe0b2b17da-kube-api-access-tnr9b" pod "8387a3d9-8b95-4287-a98c-46fe0b2b17da" (UID: "8387a3d9-8b95-4287-a98c-46fe0b2b17da"). InnerVolumeSpecName "kube-api-access-tnr9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:44:03.892009 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:03.891901    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13b57ffb-5feb-4441-8bf1-ee06e28999cf-kube-api-access-mf7w8" pod "13b57ffb-5feb-4441-8bf1-ee06e28999cf" (UID: "13b57ffb-5feb-4441-8bf1-ee06e28999cf"). InnerVolumeSpecName "kube-api-access-mf7w8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:44:03.894617 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:03.894596    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8387a3d9-8b95-4287-a98c-46fe0b2b17da-util" pod "8387a3d9-8b95-4287-a98c-46fe0b2b17da" (UID: "8387a3d9-8b95-4287-a98c-46fe0b2b17da"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:44:03.898497 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:03.898464    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13b57ffb-5feb-4441-8bf1-ee06e28999cf-util" pod "13b57ffb-5feb-4441-8bf1-ee06e28999cf" (UID: "13b57ffb-5feb-4441-8bf1-ee06e28999cf"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:44:03.990518 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:03.990482    5713 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8387a3d9-8b95-4287-a98c-46fe0b2b17da-util\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:03.990518 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:03.990514    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnr9b\" (UniqueName: \"kubernetes.io/projected/8387a3d9-8b95-4287-a98c-46fe0b2b17da-kube-api-access-tnr9b\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:03.990518 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:03.990526    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mf7w8\" (UniqueName: \"kubernetes.io/projected/13b57ffb-5feb-4441-8bf1-ee06e28999cf-kube-api-access-mf7w8\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:03.990755 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:03.990535    5713 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/13b57ffb-5feb-4441-8bf1-ee06e28999cf-bundle\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:03.990755 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:03.990544    5713 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8387a3d9-8b95-4287-a98c-46fe0b2b17da-bundle\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:03.990755 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:03.990551    5713 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/13b57ffb-5feb-4441-8bf1-ee06e28999cf-util\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:04.586124 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:04.586095    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqskt6"
Oct 30 20:44:04.586581 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:04.586089    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqskt6" event={"ID":"a802da95-53c9-4794-bf13-1d8ea25f28a7","Type":"ContainerDied","Data":"71d74db9088057d4e6327ec15e5a3e0e93b3e97610f26b967f3e84898e222e84"}
Oct 30 20:44:04.586581 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:04.586224    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71d74db9088057d4e6327ec15e5a3e0e93b3e97610f26b967f3e84898e222e84"
Oct 30 20:44:04.588223 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:04.588205    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bjvts"
Oct 30 20:44:04.588369 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:04.588204    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bjvts" event={"ID":"8387a3d9-8b95-4287-a98c-46fe0b2b17da","Type":"ContainerDied","Data":"197b444f8f349c89402756914d04a8963e5df3eee67caaebda9780068adaf15b"}
Oct 30 20:44:04.588369 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:04.588317    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="197b444f8f349c89402756914d04a8963e5df3eee67caaebda9780068adaf15b"
Oct 30 20:44:04.590196 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:04.590171    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72c6f2" event={"ID":"13b57ffb-5feb-4441-8bf1-ee06e28999cf","Type":"ContainerDied","Data":"3432ff5018328405f7deb7080321a3eaaed3bc0a8e64edd16e4e7f4147d1b874"}
Oct 30 20:44:04.590288 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:04.590209    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3432ff5018328405f7deb7080321a3eaaed3bc0a8e64edd16e4e7f4147d1b874"
Oct 30 20:44:04.590288 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:04.590224    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72c6f2"
Oct 30 20:44:10.383460 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.383423    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-57f958cfdb-h4pqh"]
Oct 30 20:44:10.383947 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.383721    5713 cpu_manager.go:414] "containerMap: removing container" podUID="55c31017-c106-4d1a-a97f-086a1584f7d0" containerName="util"
Oct 30 20:44:10.383947 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.383732    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55c31017-c106-4d1a-a97f-086a1584f7d0" containerName="util"
Oct 30 20:44:10.383947 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.383741    5713 cpu_manager.go:414] "containerMap: removing container" podUID="3b9acc6d-a438-4492-8415-c911ab6d8650" containerName="util"
Oct 30 20:44:10.383947 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.383745    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b9acc6d-a438-4492-8415-c911ab6d8650" containerName="util"
Oct 30 20:44:10.383947 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.383753    5713 cpu_manager.go:414] "containerMap: removing container" podUID="3b9acc6d-a438-4492-8415-c911ab6d8650" containerName="pull"
Oct 30 20:44:10.383947 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.383759    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b9acc6d-a438-4492-8415-c911ab6d8650" containerName="pull"
Oct 30 20:44:10.383947 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.383765    5713 cpu_manager.go:414] "containerMap: removing container" podUID="54df32fa-cbda-43f1-8159-eaf9151544b8" containerName="util"
Oct 30 20:44:10.383947 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.383769    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54df32fa-cbda-43f1-8159-eaf9151544b8" containerName="util"
Oct 30 20:44:10.383947 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.383774    5713 cpu_manager.go:414] "containerMap: removing container" podUID="8387a3d9-8b95-4287-a98c-46fe0b2b17da" containerName="util"
Oct 30 20:44:10.383947 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.383778    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8387a3d9-8b95-4287-a98c-46fe0b2b17da" containerName="util"
Oct 30 20:44:10.383947 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.383803    5713 cpu_manager.go:414] "containerMap: removing container" podUID="13b57ffb-5feb-4441-8bf1-ee06e28999cf" containerName="pull"
Oct 30 20:44:10.383947 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.383808    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13b57ffb-5feb-4441-8bf1-ee06e28999cf" containerName="pull"
Oct 30 20:44:10.383947 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.383816    5713 cpu_manager.go:414] "containerMap: removing container" podUID="54df32fa-cbda-43f1-8159-eaf9151544b8" containerName="extract"
Oct 30 20:44:10.383947 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.383820    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54df32fa-cbda-43f1-8159-eaf9151544b8" containerName="extract"
Oct 30 20:44:10.383947 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.383834    5713 cpu_manager.go:414] "containerMap: removing container" podUID="a802da95-53c9-4794-bf13-1d8ea25f28a7" containerName="util"
Oct 30 20:44:10.383947 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.383838    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a802da95-53c9-4794-bf13-1d8ea25f28a7" containerName="util"
Oct 30 20:44:10.383947 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.383845    5713 cpu_manager.go:414] "containerMap: removing container" podUID="54df32fa-cbda-43f1-8159-eaf9151544b8" containerName="pull"
Oct 30 20:44:10.383947 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.383848    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54df32fa-cbda-43f1-8159-eaf9151544b8" containerName="pull"
Oct 30 20:44:10.383947 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.383858    5713 cpu_manager.go:414] "containerMap: removing container" podUID="b0bc4c78-452e-474c-b22b-68a88cbdb74d" containerName="extract"
Oct 30 20:44:10.383947 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.383863    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0bc4c78-452e-474c-b22b-68a88cbdb74d" containerName="extract"
Oct 30 20:44:10.383947 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.383870    5713 cpu_manager.go:414] "containerMap: removing container" podUID="b0bc4c78-452e-474c-b22b-68a88cbdb74d" containerName="pull"
Oct 30 20:44:10.383947 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.383874    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0bc4c78-452e-474c-b22b-68a88cbdb74d" containerName="pull"
Oct 30 20:44:10.383947 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.383881    5713 cpu_manager.go:414] "containerMap: removing container" podUID="8387a3d9-8b95-4287-a98c-46fe0b2b17da" containerName="extract"
Oct 30 20:44:10.383947 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.383884    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8387a3d9-8b95-4287-a98c-46fe0b2b17da" containerName="extract"
Oct 30 20:44:10.383947 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.383891    5713 cpu_manager.go:414] "containerMap: removing container" podUID="8387a3d9-8b95-4287-a98c-46fe0b2b17da" containerName="pull"
Oct 30 20:44:10.383947 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.383895    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8387a3d9-8b95-4287-a98c-46fe0b2b17da" containerName="pull"
Oct 30 20:44:10.383947 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.383901    5713 cpu_manager.go:414] "containerMap: removing container" podUID="13b57ffb-5feb-4441-8bf1-ee06e28999cf" containerName="extract"
Oct 30 20:44:10.383947 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.383906    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13b57ffb-5feb-4441-8bf1-ee06e28999cf" containerName="extract"
Oct 30 20:44:10.383947 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.383917    5713 cpu_manager.go:414] "containerMap: removing container" podUID="55c31017-c106-4d1a-a97f-086a1584f7d0" containerName="pull"
Oct 30 20:44:10.383947 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.383922    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55c31017-c106-4d1a-a97f-086a1584f7d0" containerName="pull"
Oct 30 20:44:10.383947 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.383927    5713 cpu_manager.go:414] "containerMap: removing container" podUID="a802da95-53c9-4794-bf13-1d8ea25f28a7" containerName="pull"
Oct 30 20:44:10.383947 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.383931    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a802da95-53c9-4794-bf13-1d8ea25f28a7" containerName="pull"
Oct 30 20:44:10.383947 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.383938    5713 cpu_manager.go:414] "containerMap: removing container" podUID="55c31017-c106-4d1a-a97f-086a1584f7d0" containerName="extract"
Oct 30 20:44:10.383947 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.383942    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55c31017-c106-4d1a-a97f-086a1584f7d0" containerName="extract"
Oct 30 20:44:10.383947 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.383947    5713 cpu_manager.go:414] "containerMap: removing container" podUID="a802da95-53c9-4794-bf13-1d8ea25f28a7" containerName="extract"
Oct 30 20:44:10.383947 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.383952    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a802da95-53c9-4794-bf13-1d8ea25f28a7" containerName="extract"
Oct 30 20:44:10.383947 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.383961    5713 cpu_manager.go:414] "containerMap: removing container" podUID="b0bc4c78-452e-474c-b22b-68a88cbdb74d" containerName="util"
Oct 30 20:44:10.383947 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.383966    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0bc4c78-452e-474c-b22b-68a88cbdb74d" containerName="util"
Oct 30 20:44:10.385063 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.383971    5713 cpu_manager.go:414] "containerMap: removing container" podUID="13b57ffb-5feb-4441-8bf1-ee06e28999cf" containerName="util"
Oct 30 20:44:10.385063 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.383975    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13b57ffb-5feb-4441-8bf1-ee06e28999cf" containerName="util"
Oct 30 20:44:10.385063 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.383981    5713 cpu_manager.go:414] "containerMap: removing container" podUID="3b9acc6d-a438-4492-8415-c911ab6d8650" containerName="extract"
Oct 30 20:44:10.385063 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.383985    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b9acc6d-a438-4492-8415-c911ab6d8650" containerName="extract"
Oct 30 20:44:10.385063 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.384029    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b9acc6d-a438-4492-8415-c911ab6d8650" containerName="extract"
Oct 30 20:44:10.385063 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.384037    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="54df32fa-cbda-43f1-8159-eaf9151544b8" containerName="extract"
Oct 30 20:44:10.385063 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.384044    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0bc4c78-452e-474c-b22b-68a88cbdb74d" containerName="extract"
Oct 30 20:44:10.385063 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.384049    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="8387a3d9-8b95-4287-a98c-46fe0b2b17da" containerName="extract"
Oct 30 20:44:10.385063 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.384054    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="13b57ffb-5feb-4441-8bf1-ee06e28999cf" containerName="extract"
Oct 30 20:44:10.385063 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.384060    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="55c31017-c106-4d1a-a97f-086a1584f7d0" containerName="extract"
Oct 30 20:44:10.385063 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.384065    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="a802da95-53c9-4794-bf13-1d8ea25f28a7" containerName="extract"
Oct 30 20:44:10.386747 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.386732    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f958cfdb-h4pqh"
Oct 30 20:44:10.517364 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.517331    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 20:44:10.517364 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.517339    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 30 20:44:10.517550 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.517504    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 30 20:44:10.517595 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.517552    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 20:44:10.517635 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.517606    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-wrjh7\""
Oct 30 20:44:10.517635 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.517615    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 30 20:44:10.527741 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.527717    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-57f958cfdb-h4pqh"]
Oct 30 20:44:10.551553 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.551532    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a69a1de-fd21-4eb4-8961-4ad2d2aee5e3-tmp\") pod \"opendatahub-operator-controller-manager-57f958cfdb-h4pqh\" (UID: \"9a69a1de-fd21-4eb4-8961-4ad2d2aee5e3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f958cfdb-h4pqh"
Oct 30 20:44:10.551686 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.551561    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9a69a1de-fd21-4eb4-8961-4ad2d2aee5e3-manifests\") pod \"opendatahub-operator-controller-manager-57f958cfdb-h4pqh\" (UID: \"9a69a1de-fd21-4eb4-8961-4ad2d2aee5e3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f958cfdb-h4pqh"
Oct 30 20:44:10.551686 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.551598    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfnpg\" (UniqueName: \"kubernetes.io/projected/9a69a1de-fd21-4eb4-8961-4ad2d2aee5e3-kube-api-access-sfnpg\") pod \"opendatahub-operator-controller-manager-57f958cfdb-h4pqh\" (UID: \"9a69a1de-fd21-4eb4-8961-4ad2d2aee5e3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f958cfdb-h4pqh"
Oct 30 20:44:10.551765 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.551731    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9a69a1de-fd21-4eb4-8961-4ad2d2aee5e3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57f958cfdb-h4pqh\" (UID: \"9a69a1de-fd21-4eb4-8961-4ad2d2aee5e3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f958cfdb-h4pqh"
Oct 30 20:44:10.551816 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.551766    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9a69a1de-fd21-4eb4-8961-4ad2d2aee5e3-metrics-cert\") pod \"opendatahub-operator-controller-manager-57f958cfdb-h4pqh\" (UID: \"9a69a1de-fd21-4eb4-8961-4ad2d2aee5e3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f958cfdb-h4pqh"
Oct 30 20:44:10.551816 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.551807    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9a69a1de-fd21-4eb4-8961-4ad2d2aee5e3-webhook-cert\") pod \"opendatahub-operator-controller-manager-57f958cfdb-h4pqh\" (UID: \"9a69a1de-fd21-4eb4-8961-4ad2d2aee5e3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f958cfdb-h4pqh"
Oct 30 20:44:10.652624 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.652540    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfnpg\" (UniqueName: \"kubernetes.io/projected/9a69a1de-fd21-4eb4-8961-4ad2d2aee5e3-kube-api-access-sfnpg\") pod \"opendatahub-operator-controller-manager-57f958cfdb-h4pqh\" (UID: \"9a69a1de-fd21-4eb4-8961-4ad2d2aee5e3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f958cfdb-h4pqh"
Oct 30 20:44:10.652624 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.652589    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9a69a1de-fd21-4eb4-8961-4ad2d2aee5e3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57f958cfdb-h4pqh\" (UID: \"9a69a1de-fd21-4eb4-8961-4ad2d2aee5e3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f958cfdb-h4pqh"
Oct 30 20:44:10.652624 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.652620    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9a69a1de-fd21-4eb4-8961-4ad2d2aee5e3-metrics-cert\") pod \"opendatahub-operator-controller-manager-57f958cfdb-h4pqh\" (UID: \"9a69a1de-fd21-4eb4-8961-4ad2d2aee5e3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f958cfdb-h4pqh"
Oct 30 20:44:10.652935 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.652645    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9a69a1de-fd21-4eb4-8961-4ad2d2aee5e3-webhook-cert\") pod \"opendatahub-operator-controller-manager-57f958cfdb-h4pqh\" (UID: \"9a69a1de-fd21-4eb4-8961-4ad2d2aee5e3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f958cfdb-h4pqh"
Oct 30 20:44:10.652935 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.652704    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a69a1de-fd21-4eb4-8961-4ad2d2aee5e3-tmp\") pod \"opendatahub-operator-controller-manager-57f958cfdb-h4pqh\" (UID: \"9a69a1de-fd21-4eb4-8961-4ad2d2aee5e3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f958cfdb-h4pqh"
Oct 30 20:44:10.652935 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.652734    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9a69a1de-fd21-4eb4-8961-4ad2d2aee5e3-manifests\") pod \"opendatahub-operator-controller-manager-57f958cfdb-h4pqh\" (UID: \"9a69a1de-fd21-4eb4-8961-4ad2d2aee5e3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f958cfdb-h4pqh"
Oct 30 20:44:10.653146 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.653128    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9a69a1de-fd21-4eb4-8961-4ad2d2aee5e3-manifests\") pod \"opendatahub-operator-controller-manager-57f958cfdb-h4pqh\" (UID: \"9a69a1de-fd21-4eb4-8961-4ad2d2aee5e3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f958cfdb-h4pqh"
Oct 30 20:44:10.655274 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.655251    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a69a1de-fd21-4eb4-8961-4ad2d2aee5e3-tmp\") pod \"opendatahub-operator-controller-manager-57f958cfdb-h4pqh\" (UID: \"9a69a1de-fd21-4eb4-8961-4ad2d2aee5e3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f958cfdb-h4pqh"
Oct 30 20:44:10.655536 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.655515    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9a69a1de-fd21-4eb4-8961-4ad2d2aee5e3-metrics-cert\") pod \"opendatahub-operator-controller-manager-57f958cfdb-h4pqh\" (UID: \"9a69a1de-fd21-4eb4-8961-4ad2d2aee5e3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f958cfdb-h4pqh"
Oct 30 20:44:10.655662 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.655644    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9a69a1de-fd21-4eb4-8961-4ad2d2aee5e3-webhook-cert\") pod \"opendatahub-operator-controller-manager-57f958cfdb-h4pqh\" (UID: \"9a69a1de-fd21-4eb4-8961-4ad2d2aee5e3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f958cfdb-h4pqh"
Oct 30 20:44:10.655715 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.655685    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9a69a1de-fd21-4eb4-8961-4ad2d2aee5e3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57f958cfdb-h4pqh\" (UID: \"9a69a1de-fd21-4eb4-8961-4ad2d2aee5e3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f958cfdb-h4pqh"
Oct 30 20:44:10.724253 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.724214    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfnpg\" (UniqueName: \"kubernetes.io/projected/9a69a1de-fd21-4eb4-8961-4ad2d2aee5e3-kube-api-access-sfnpg\") pod \"opendatahub-operator-controller-manager-57f958cfdb-h4pqh\" (UID: \"9a69a1de-fd21-4eb4-8961-4ad2d2aee5e3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f958cfdb-h4pqh"
Oct 30 20:44:10.999523 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:10.999491    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f958cfdb-h4pqh"
Oct 30 20:44:11.562879 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:11.562847    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-57f958cfdb-h4pqh"]
Oct 30 20:44:11.566196 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:44:11.566172    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a69a1de_fd21_4eb4_8961_4ad2d2aee5e3.slice/crio-1cb2e67cb5b445a5d1850b35d9f7ad8a7fb2727a6ce649f29c469aac6101decd WatchSource:0}: Error finding container 1cb2e67cb5b445a5d1850b35d9f7ad8a7fb2727a6ce649f29c469aac6101decd: Status 404 returned error can't find the container with id 1cb2e67cb5b445a5d1850b35d9f7ad8a7fb2727a6ce649f29c469aac6101decd
Oct 30 20:44:11.623433 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:11.623402    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f958cfdb-h4pqh" event={"ID":"9a69a1de-fd21-4eb4-8961-4ad2d2aee5e3","Type":"ContainerStarted","Data":"1cb2e67cb5b445a5d1850b35d9f7ad8a7fb2727a6ce649f29c469aac6101decd"}
Oct 30 20:44:11.928043 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:11.927970    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-6xr6r"]
Oct 30 20:44:11.930759 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:11.930740    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-6xr6r"
Oct 30 20:44:11.989569 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:11.989542    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 30 20:44:11.989569 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:11.989565    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 30 20:44:11.990988 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:11.990968    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-6xr6r"]
Oct 30 20:44:12.007569 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:12.007549    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-rcng2\""
Oct 30 20:44:12.066774 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:12.066751    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9d3d4b64-23c4-4b61-8df5-c6460d068c15-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-6xr6r\" (UID: \"9d3d4b64-23c4-4b61-8df5-c6460d068c15\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-6xr6r"
Oct 30 20:44:12.066907 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:12.066874    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv75z\" (UniqueName: \"kubernetes.io/projected/9d3d4b64-23c4-4b61-8df5-c6460d068c15-kube-api-access-vv75z\") pod \"cert-manager-cainjector-5c57bbd4fc-6xr6r\" (UID: \"9d3d4b64-23c4-4b61-8df5-c6460d068c15\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-6xr6r"
Oct 30 20:44:12.167811 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:12.167760    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9d3d4b64-23c4-4b61-8df5-c6460d068c15-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-6xr6r\" (UID: \"9d3d4b64-23c4-4b61-8df5-c6460d068c15\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-6xr6r"
Oct 30 20:44:12.167960 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:12.167874    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv75z\" (UniqueName: \"kubernetes.io/projected/9d3d4b64-23c4-4b61-8df5-c6460d068c15-kube-api-access-vv75z\") pod \"cert-manager-cainjector-5c57bbd4fc-6xr6r\" (UID: \"9d3d4b64-23c4-4b61-8df5-c6460d068c15\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-6xr6r"
Oct 30 20:44:12.224656 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:12.224627    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9d3d4b64-23c4-4b61-8df5-c6460d068c15-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-6xr6r\" (UID: \"9d3d4b64-23c4-4b61-8df5-c6460d068c15\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-6xr6r"
Oct 30 20:44:12.224782 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:12.224694    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv75z\" (UniqueName: \"kubernetes.io/projected/9d3d4b64-23c4-4b61-8df5-c6460d068c15-kube-api-access-vv75z\") pod \"cert-manager-cainjector-5c57bbd4fc-6xr6r\" (UID: \"9d3d4b64-23c4-4b61-8df5-c6460d068c15\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-6xr6r"
Oct 30 20:44:12.257350 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:12.257330    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-6xr6r"
Oct 30 20:44:12.646067 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:12.645909    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-6xr6r"]
Oct 30 20:44:12.649007 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:44:12.648978    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9d3d4b64_23c4_4b61_8df5_c6460d068c15.slice/crio-0ceb2e85ed6f29e7096053d6819ed906bf58bbecb2b8c9100129b092b434356d WatchSource:0}: Error finding container 0ceb2e85ed6f29e7096053d6819ed906bf58bbecb2b8c9100129b092b434356d: Status 404 returned error can't find the container with id 0ceb2e85ed6f29e7096053d6819ed906bf58bbecb2b8c9100129b092b434356d
Oct 30 20:44:13.634832 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:13.634777    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-6xr6r" event={"ID":"9d3d4b64-23c4-4b61-8df5-c6460d068c15","Type":"ContainerStarted","Data":"0ceb2e85ed6f29e7096053d6819ed906bf58bbecb2b8c9100129b092b434356d"}
Oct 30 20:44:13.830006 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:13.829974    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-g78zk"]
Oct 30 20:44:13.835119 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:13.835099    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-g78zk"
Oct 30 20:44:13.939413 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:13.939387    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Oct 30 20:44:13.939413 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:13.939391    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Oct 30 20:44:13.939665 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:13.939462    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-d5h4x\""
Oct 30 20:44:13.941027 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:13.941005    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-g78zk"]
Oct 30 20:44:13.984985 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:13.984959    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ecb8ca8-2877-425d-80a7-cdac9d4c3cfb-tmp\") pod \"jobset-operator-674cd59dc5-g78zk\" (UID: \"9ecb8ca8-2877-425d-80a7-cdac9d4c3cfb\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-g78zk"
Oct 30 20:44:13.985136 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:13.985064    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmnds\" (UniqueName: \"kubernetes.io/projected/9ecb8ca8-2877-425d-80a7-cdac9d4c3cfb-kube-api-access-dmnds\") pod \"jobset-operator-674cd59dc5-g78zk\" (UID: \"9ecb8ca8-2877-425d-80a7-cdac9d4c3cfb\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-g78zk"
Oct 30 20:44:14.085986 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:14.085955    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmnds\" (UniqueName: \"kubernetes.io/projected/9ecb8ca8-2877-425d-80a7-cdac9d4c3cfb-kube-api-access-dmnds\") pod \"jobset-operator-674cd59dc5-g78zk\" (UID: \"9ecb8ca8-2877-425d-80a7-cdac9d4c3cfb\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-g78zk"
Oct 30 20:44:14.086140 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:14.086018    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ecb8ca8-2877-425d-80a7-cdac9d4c3cfb-tmp\") pod \"jobset-operator-674cd59dc5-g78zk\" (UID: \"9ecb8ca8-2877-425d-80a7-cdac9d4c3cfb\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-g78zk"
Oct 30 20:44:14.086431 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:14.086410    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ecb8ca8-2877-425d-80a7-cdac9d4c3cfb-tmp\") pod \"jobset-operator-674cd59dc5-g78zk\" (UID: \"9ecb8ca8-2877-425d-80a7-cdac9d4c3cfb\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-g78zk"
Oct 30 20:44:14.144535 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:14.144500    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmnds\" (UniqueName: \"kubernetes.io/projected/9ecb8ca8-2877-425d-80a7-cdac9d4c3cfb-kube-api-access-dmnds\") pod \"jobset-operator-674cd59dc5-g78zk\" (UID: \"9ecb8ca8-2877-425d-80a7-cdac9d4c3cfb\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-g78zk"
Oct 30 20:44:14.147272 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:14.147246    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-g78zk"
Oct 30 20:44:14.527382 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:14.527351    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-g78zk"]
Oct 30 20:44:14.530229 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:44:14.530195    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9ecb8ca8_2877_425d_80a7_cdac9d4c3cfb.slice/crio-c0a42ed42bd66a9f97cc456f08e19ccc5f8451d096ec534486c4a5e5d5fe7e0b WatchSource:0}: Error finding container c0a42ed42bd66a9f97cc456f08e19ccc5f8451d096ec534486c4a5e5d5fe7e0b: Status 404 returned error can't find the container with id c0a42ed42bd66a9f97cc456f08e19ccc5f8451d096ec534486c4a5e5d5fe7e0b
Oct 30 20:44:14.642885 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:14.642721    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-g78zk" event={"ID":"9ecb8ca8-2877-425d-80a7-cdac9d4c3cfb","Type":"ContainerStarted","Data":"c0a42ed42bd66a9f97cc456f08e19ccc5f8451d096ec534486c4a5e5d5fe7e0b"}
Oct 30 20:44:15.650859 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:15.650759    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f958cfdb-h4pqh" event={"ID":"9a69a1de-fd21-4eb4-8961-4ad2d2aee5e3","Type":"ContainerStarted","Data":"1d76760fa447c47f67d9d02b7f1df25560c90e1139231740c36cf01c6cf341b0"}
Oct 30 20:44:16.657766 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:16.657721    5713 generic.go:356] "Generic (PLEG): container finished" podID="9a69a1de-fd21-4eb4-8961-4ad2d2aee5e3" containerID="1d76760fa447c47f67d9d02b7f1df25560c90e1139231740c36cf01c6cf341b0" exitCode=0
Oct 30 20:44:16.658258 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:16.657827    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f958cfdb-h4pqh" event={"ID":"9a69a1de-fd21-4eb4-8961-4ad2d2aee5e3","Type":"ContainerDied","Data":"1d76760fa447c47f67d9d02b7f1df25560c90e1139231740c36cf01c6cf341b0"}
Oct 30 20:44:17.664602 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:17.664566    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f958cfdb-h4pqh" event={"ID":"9a69a1de-fd21-4eb4-8961-4ad2d2aee5e3","Type":"ContainerStarted","Data":"54f535da8bd8c58b8b77ee562da942300ebc40f64e12e5a38fc6ac34d46987f5"}
Oct 30 20:44:17.665095 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:17.664693    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f958cfdb-h4pqh"
Oct 30 20:44:17.666126 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:17.666103    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-6xr6r" event={"ID":"9d3d4b64-23c4-4b61-8df5-c6460d068c15","Type":"ContainerStarted","Data":"232369d4fff6d5962bb36f5f0290d7e7f2eb624ee7ef2f182665f52b3a1885e0"}
Oct 30 20:44:17.843158 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:17.843092    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f958cfdb-h4pqh" podStartSLOduration=2.79050653 podStartE2EDuration="7.843074937s" totalImagesPullingTime="4.397109345s" totalInitContainerRuntime="655.459062ms" isStatefulPod=false podCreationTimestamp="2026-10-30 20:44:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:44:17.792738564 +0000 UTC m=+340.857514962" watchObservedRunningTime="2026-10-30 20:44:17.843074937 +0000 UTC m=+340.907851335"
Oct 30 20:44:17.906159 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:17.906095    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-6xr6r" podStartSLOduration=2.405835751 podStartE2EDuration="6.906072284s" totalImagesPullingTime="4.500236533s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:44:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:44:17.904936851 +0000 UTC m=+340.969713254" watchObservedRunningTime="2026-10-30 20:44:17.906072284 +0000 UTC m=+340.970848683"
Oct 30 20:44:18.673385 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:18.673293    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-g78zk" event={"ID":"9ecb8ca8-2877-425d-80a7-cdac9d4c3cfb","Type":"ContainerStarted","Data":"6553a01cc7729d2b88308350aab39a01315c4e7cc9c6ab1b0c5e740306725238"}
Oct 30 20:44:18.785595 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:18.785541    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-g78zk" podStartSLOduration=1.914957839 podStartE2EDuration="5.785526319s" totalImagesPullingTime="3.87056848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:44:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:44:18.784977518 +0000 UTC m=+341.849753917" watchObservedRunningTime="2026-10-30 20:44:18.785526319 +0000 UTC m=+341.850302720"
Oct 30 20:44:19.896282 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:19.896251    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-t6ksw"]
Oct 30 20:44:19.898436 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:19.898420    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-t6ksw"
Oct 30 20:44:19.940587 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:19.940562    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/09e7d4ed-673e-4d00-9697-2e018287373b-bound-sa-token\") pod \"cert-manager-597b95cb84-t6ksw\" (UID: \"09e7d4ed-673e-4d00-9697-2e018287373b\") " pod="cert-manager/cert-manager-597b95cb84-t6ksw"
Oct 30 20:44:19.940728 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:19.940594    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ljg2\" (UniqueName: \"kubernetes.io/projected/09e7d4ed-673e-4d00-9697-2e018287373b-kube-api-access-9ljg2\") pod \"cert-manager-597b95cb84-t6ksw\" (UID: \"09e7d4ed-673e-4d00-9697-2e018287373b\") " pod="cert-manager/cert-manager-597b95cb84-t6ksw"
Oct 30 20:44:20.000194 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:20.000160    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-nvcll\""
Oct 30 20:44:20.003714 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:20.002395    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-t6ksw"]
Oct 30 20:44:20.041397 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:20.041364    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/09e7d4ed-673e-4d00-9697-2e018287373b-bound-sa-token\") pod \"cert-manager-597b95cb84-t6ksw\" (UID: \"09e7d4ed-673e-4d00-9697-2e018287373b\") " pod="cert-manager/cert-manager-597b95cb84-t6ksw"
Oct 30 20:44:20.041397 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:20.041402    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ljg2\" (UniqueName: \"kubernetes.io/projected/09e7d4ed-673e-4d00-9697-2e018287373b-kube-api-access-9ljg2\") pod \"cert-manager-597b95cb84-t6ksw\" (UID: \"09e7d4ed-673e-4d00-9697-2e018287373b\") " pod="cert-manager/cert-manager-597b95cb84-t6ksw"
Oct 30 20:44:20.098343 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:20.098309    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/09e7d4ed-673e-4d00-9697-2e018287373b-bound-sa-token\") pod \"cert-manager-597b95cb84-t6ksw\" (UID: \"09e7d4ed-673e-4d00-9697-2e018287373b\") " pod="cert-manager/cert-manager-597b95cb84-t6ksw"
Oct 30 20:44:20.098478 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:20.098405    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ljg2\" (UniqueName: \"kubernetes.io/projected/09e7d4ed-673e-4d00-9697-2e018287373b-kube-api-access-9ljg2\") pod \"cert-manager-597b95cb84-t6ksw\" (UID: \"09e7d4ed-673e-4d00-9697-2e018287373b\") " pod="cert-manager/cert-manager-597b95cb84-t6ksw"
Oct 30 20:44:20.209199 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:20.209167    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-t6ksw"
Oct 30 20:44:20.583595 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:20.583474    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-t6ksw"]
Oct 30 20:44:20.586678 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:44:20.586645    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod09e7d4ed_673e_4d00_9697_2e018287373b.slice/crio-133754fb169c64e7656222e03cd234c5e7d02bd49ae757b927d198d4bcfaf4ad WatchSource:0}: Error finding container 133754fb169c64e7656222e03cd234c5e7d02bd49ae757b927d198d4bcfaf4ad: Status 404 returned error can't find the container with id 133754fb169c64e7656222e03cd234c5e7d02bd49ae757b927d198d4bcfaf4ad
Oct 30 20:44:20.683808 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:20.683760    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-t6ksw" event={"ID":"09e7d4ed-673e-4d00-9697-2e018287373b","Type":"ContainerStarted","Data":"f8917649a2329fdadfce6f9bc4226d4bccda4a5576f4ea9066c72589939b1b86"}
Oct 30 20:44:20.683967 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:20.683824    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-t6ksw" event={"ID":"09e7d4ed-673e-4d00-9697-2e018287373b","Type":"ContainerStarted","Data":"133754fb169c64e7656222e03cd234c5e7d02bd49ae757b927d198d4bcfaf4ad"}
Oct 30 20:44:20.801047 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:20.800993    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-t6ksw" podStartSLOduration=1.8009722419999998 podStartE2EDuration="1.800972242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:44:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:44:20.799190346 +0000 UTC m=+343.863966745" watchObservedRunningTime="2026-10-30 20:44:20.800972242 +0000 UTC m=+343.865748643"
Oct 30 20:44:24.463121 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:44:24.463087    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 20:44:28.676053 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:28.676023    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f958cfdb-h4pqh"
Oct 30 20:44:36.110169 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:36.110136    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-4hh9b"]
Oct 30 20:44:36.113015 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:36.112998    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4hh9b"
Oct 30 20:44:36.182261 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:36.182227    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zmjc\" (UniqueName: \"kubernetes.io/projected/d2d9bfed-999d-4d51-91a8-30b0e938ee1d-kube-api-access-4zmjc\") pod \"limitador-operator-controller-manager-5778bb4449-4hh9b\" (UID: \"d2d9bfed-999d-4d51-91a8-30b0e938ee1d\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4hh9b"
Oct 30 20:44:36.215014 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:36.214983    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 20:44:36.215167 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:36.215067    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 20:44:36.215167 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:36.215159    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-49lkv\""
Oct 30 20:44:36.216578 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:36.216559    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-4hh9b"]
Oct 30 20:44:36.283706 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:36.283667    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zmjc\" (UniqueName: \"kubernetes.io/projected/d2d9bfed-999d-4d51-91a8-30b0e938ee1d-kube-api-access-4zmjc\") pod \"limitador-operator-controller-manager-5778bb4449-4hh9b\" (UID: \"d2d9bfed-999d-4d51-91a8-30b0e938ee1d\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4hh9b"
Oct 30 20:44:36.341037 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:36.341009    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zmjc\" (UniqueName: \"kubernetes.io/projected/d2d9bfed-999d-4d51-91a8-30b0e938ee1d-kube-api-access-4zmjc\") pod \"limitador-operator-controller-manager-5778bb4449-4hh9b\" (UID: \"d2d9bfed-999d-4d51-91a8-30b0e938ee1d\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4hh9b"
Oct 30 20:44:36.424917 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:36.424828    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4hh9b"
Oct 30 20:44:36.797970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:36.797941    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-4hh9b"]
Oct 30 20:44:36.801769 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:44:36.801737    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2d9bfed_999d_4d51_91a8_30b0e938ee1d.slice/crio-515fcaa20c6f0a36b4de43a6823605381d43dbc521ac29cc897bb626678109a7 WatchSource:0}: Error finding container 515fcaa20c6f0a36b4de43a6823605381d43dbc521ac29cc897bb626678109a7: Status 404 returned error can't find the container with id 515fcaa20c6f0a36b4de43a6823605381d43dbc521ac29cc897bb626678109a7
Oct 30 20:44:37.765355 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:37.765318    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4hh9b" event={"ID":"d2d9bfed-999d-4d51-91a8-30b0e938ee1d","Type":"ContainerStarted","Data":"515fcaa20c6f0a36b4de43a6823605381d43dbc521ac29cc897bb626678109a7"}
Oct 30 20:44:43.795933 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:43.795900    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4hh9b" event={"ID":"d2d9bfed-999d-4d51-91a8-30b0e938ee1d","Type":"ContainerStarted","Data":"4cf2040ab4dee414c36617759d8ac97f193810cc31b4f1c051fe3d340bdae72e"}
Oct 30 20:44:43.796429 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:43.795992    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4hh9b"
Oct 30 20:44:43.916266 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:43.916216    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4hh9b" podStartSLOduration=1.565626773 podStartE2EDuration="7.916202226s" totalImagesPullingTime="6.350575453s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:44:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:44:43.91105914 +0000 UTC m=+366.975835540" watchObservedRunningTime="2026-10-30 20:44:43.916202226 +0000 UTC m=+366.980978623"
Oct 30 20:44:47.643283 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:47.643190    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7988bdb8dc-nzmcx"]
Oct 30 20:44:47.661595 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:47.661560    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7988bdb8dc-nzmcx"
Oct 30 20:44:47.765799 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:47.765751    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 30 20:44:47.765970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:47.765932    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-zk9dx\""
Oct 30 20:44:47.766025 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:47.765997    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 30 20:44:47.771745 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:47.771724    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7988bdb8dc-nzmcx"]
Oct 30 20:44:47.790069 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:47.790039    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5csfb\" (UniqueName: \"kubernetes.io/projected/cfe209a2-82e3-4c1e-99d0-615d8294737a-kube-api-access-5csfb\") pod \"kube-auth-proxy-7988bdb8dc-nzmcx\" (UID: \"cfe209a2-82e3-4c1e-99d0-615d8294737a\") " pod="openshift-ingress/kube-auth-proxy-7988bdb8dc-nzmcx"
Oct 30 20:44:47.790243 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:47.790152    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cfe209a2-82e3-4c1e-99d0-615d8294737a-tmp\") pod \"kube-auth-proxy-7988bdb8dc-nzmcx\" (UID: \"cfe209a2-82e3-4c1e-99d0-615d8294737a\") " pod="openshift-ingress/kube-auth-proxy-7988bdb8dc-nzmcx"
Oct 30 20:44:47.790243 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:47.790211    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfe209a2-82e3-4c1e-99d0-615d8294737a-tls-certs\") pod \"kube-auth-proxy-7988bdb8dc-nzmcx\" (UID: \"cfe209a2-82e3-4c1e-99d0-615d8294737a\") " pod="openshift-ingress/kube-auth-proxy-7988bdb8dc-nzmcx"
Oct 30 20:44:47.891152 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:47.891121    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfe209a2-82e3-4c1e-99d0-615d8294737a-tls-certs\") pod \"kube-auth-proxy-7988bdb8dc-nzmcx\" (UID: \"cfe209a2-82e3-4c1e-99d0-615d8294737a\") " pod="openshift-ingress/kube-auth-proxy-7988bdb8dc-nzmcx"
Oct 30 20:44:47.891336 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:47.891171    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5csfb\" (UniqueName: \"kubernetes.io/projected/cfe209a2-82e3-4c1e-99d0-615d8294737a-kube-api-access-5csfb\") pod \"kube-auth-proxy-7988bdb8dc-nzmcx\" (UID: \"cfe209a2-82e3-4c1e-99d0-615d8294737a\") " pod="openshift-ingress/kube-auth-proxy-7988bdb8dc-nzmcx"
Oct 30 20:44:47.891336 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:47.891219    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cfe209a2-82e3-4c1e-99d0-615d8294737a-tmp\") pod \"kube-auth-proxy-7988bdb8dc-nzmcx\" (UID: \"cfe209a2-82e3-4c1e-99d0-615d8294737a\") " pod="openshift-ingress/kube-auth-proxy-7988bdb8dc-nzmcx"
Oct 30 20:44:47.893658 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:47.893580    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cfe209a2-82e3-4c1e-99d0-615d8294737a-tmp\") pod \"kube-auth-proxy-7988bdb8dc-nzmcx\" (UID: \"cfe209a2-82e3-4c1e-99d0-615d8294737a\") " pod="openshift-ingress/kube-auth-proxy-7988bdb8dc-nzmcx"
Oct 30 20:44:47.893921 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:47.893901    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfe209a2-82e3-4c1e-99d0-615d8294737a-tls-certs\") pod \"kube-auth-proxy-7988bdb8dc-nzmcx\" (UID: \"cfe209a2-82e3-4c1e-99d0-615d8294737a\") " pod="openshift-ingress/kube-auth-proxy-7988bdb8dc-nzmcx"
Oct 30 20:44:47.949598 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:47.949563    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5csfb\" (UniqueName: \"kubernetes.io/projected/cfe209a2-82e3-4c1e-99d0-615d8294737a-kube-api-access-5csfb\") pod \"kube-auth-proxy-7988bdb8dc-nzmcx\" (UID: \"cfe209a2-82e3-4c1e-99d0-615d8294737a\") " pod="openshift-ingress/kube-auth-proxy-7988bdb8dc-nzmcx"
Oct 30 20:44:47.990570 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:47.990540    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7988bdb8dc-nzmcx"
Oct 30 20:44:48.385086 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:48.385060    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7988bdb8dc-nzmcx"]
Oct 30 20:44:48.387729 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:44:48.387703    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfe209a2_82e3_4c1e_99d0_615d8294737a.slice/crio-6dcf429c9d96e0bbe8fdaf42e3c04c2cad79a061b600d8dc2e450f229051a11b WatchSource:0}: Error finding container 6dcf429c9d96e0bbe8fdaf42e3c04c2cad79a061b600d8dc2e450f229051a11b: Status 404 returned error can't find the container with id 6dcf429c9d96e0bbe8fdaf42e3c04c2cad79a061b600d8dc2e450f229051a11b
Oct 30 20:44:48.822864 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:48.822829    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7988bdb8dc-nzmcx" event={"ID":"cfe209a2-82e3-4c1e-99d0-615d8294737a","Type":"ContainerStarted","Data":"6dcf429c9d96e0bbe8fdaf42e3c04c2cad79a061b600d8dc2e450f229051a11b"}
Oct 30 20:44:50.940765 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:50.940734    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5b94c4c576-t9qdd"]
Oct 30 20:44:50.946634 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:50.946611    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-5b94c4c576-t9qdd"
Oct 30 20:44:50.999064 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:50.999030    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-bbbw8\""
Oct 30 20:44:51.023254 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:51.023227    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92834d63-fbc2-4fcc-aca8-93f66f7e6d26-metrics-certs\") pod \"jobset-controller-manager-5b94c4c576-t9qdd\" (UID: \"92834d63-fbc2-4fcc-aca8-93f66f7e6d26\") " pod="openshift-jobset-operator/jobset-controller-manager-5b94c4c576-t9qdd"
Oct 30 20:44:51.023425 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:51.023268    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/92834d63-fbc2-4fcc-aca8-93f66f7e6d26-manager-config\") pod \"jobset-controller-manager-5b94c4c576-t9qdd\" (UID: \"92834d63-fbc2-4fcc-aca8-93f66f7e6d26\") " pod="openshift-jobset-operator/jobset-controller-manager-5b94c4c576-t9qdd"
Oct 30 20:44:51.023491 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:51.023403    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwcxs\" (UniqueName: \"kubernetes.io/projected/92834d63-fbc2-4fcc-aca8-93f66f7e6d26-kube-api-access-dwcxs\") pod \"jobset-controller-manager-5b94c4c576-t9qdd\" (UID: \"92834d63-fbc2-4fcc-aca8-93f66f7e6d26\") " pod="openshift-jobset-operator/jobset-controller-manager-5b94c4c576-t9qdd"
Oct 30 20:44:51.023548 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:51.023519    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/92834d63-fbc2-4fcc-aca8-93f66f7e6d26-cert\") pod \"jobset-controller-manager-5b94c4c576-t9qdd\" (UID: \"92834d63-fbc2-4fcc-aca8-93f66f7e6d26\") " pod="openshift-jobset-operator/jobset-controller-manager-5b94c4c576-t9qdd"
Oct 30 20:44:51.049126 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:51.049099    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Oct 30 20:44:51.049343 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:51.049322    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Oct 30 20:44:51.049682 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:51.049659    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Oct 30 20:44:51.051314 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:51.051288    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5b94c4c576-t9qdd"]
Oct 30 20:44:51.124447 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:51.124415    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/92834d63-fbc2-4fcc-aca8-93f66f7e6d26-cert\") pod \"jobset-controller-manager-5b94c4c576-t9qdd\" (UID: \"92834d63-fbc2-4fcc-aca8-93f66f7e6d26\") " pod="openshift-jobset-operator/jobset-controller-manager-5b94c4c576-t9qdd"
Oct 30 20:44:51.124621 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:51.124510    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92834d63-fbc2-4fcc-aca8-93f66f7e6d26-metrics-certs\") pod \"jobset-controller-manager-5b94c4c576-t9qdd\" (UID: \"92834d63-fbc2-4fcc-aca8-93f66f7e6d26\") " pod="openshift-jobset-operator/jobset-controller-manager-5b94c4c576-t9qdd"
Oct 30 20:44:51.124621 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:51.124557    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/92834d63-fbc2-4fcc-aca8-93f66f7e6d26-manager-config\") pod \"jobset-controller-manager-5b94c4c576-t9qdd\" (UID: \"92834d63-fbc2-4fcc-aca8-93f66f7e6d26\") " pod="openshift-jobset-operator/jobset-controller-manager-5b94c4c576-t9qdd"
Oct 30 20:44:51.124621 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:51.124603    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwcxs\" (UniqueName: \"kubernetes.io/projected/92834d63-fbc2-4fcc-aca8-93f66f7e6d26-kube-api-access-dwcxs\") pod \"jobset-controller-manager-5b94c4c576-t9qdd\" (UID: \"92834d63-fbc2-4fcc-aca8-93f66f7e6d26\") " pod="openshift-jobset-operator/jobset-controller-manager-5b94c4c576-t9qdd"
Oct 30 20:44:51.125648 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:51.125626    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/92834d63-fbc2-4fcc-aca8-93f66f7e6d26-manager-config\") pod \"jobset-controller-manager-5b94c4c576-t9qdd\" (UID: \"92834d63-fbc2-4fcc-aca8-93f66f7e6d26\") " pod="openshift-jobset-operator/jobset-controller-manager-5b94c4c576-t9qdd"
Oct 30 20:44:51.127766 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:51.127742    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/92834d63-fbc2-4fcc-aca8-93f66f7e6d26-cert\") pod \"jobset-controller-manager-5b94c4c576-t9qdd\" (UID: \"92834d63-fbc2-4fcc-aca8-93f66f7e6d26\") " pod="openshift-jobset-operator/jobset-controller-manager-5b94c4c576-t9qdd"
Oct 30 20:44:51.128340 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:51.128320    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92834d63-fbc2-4fcc-aca8-93f66f7e6d26-metrics-certs\") pod \"jobset-controller-manager-5b94c4c576-t9qdd\" (UID: \"92834d63-fbc2-4fcc-aca8-93f66f7e6d26\") " pod="openshift-jobset-operator/jobset-controller-manager-5b94c4c576-t9qdd"
Oct 30 20:44:51.185322 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:51.185292    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwcxs\" (UniqueName: \"kubernetes.io/projected/92834d63-fbc2-4fcc-aca8-93f66f7e6d26-kube-api-access-dwcxs\") pod \"jobset-controller-manager-5b94c4c576-t9qdd\" (UID: \"92834d63-fbc2-4fcc-aca8-93f66f7e6d26\") " pod="openshift-jobset-operator/jobset-controller-manager-5b94c4c576-t9qdd"
Oct 30 20:44:51.261481 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:51.261395    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-5b94c4c576-t9qdd"
Oct 30 20:44:51.656547 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:51.656453    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5b94c4c576-t9qdd"]
Oct 30 20:44:51.662328 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:44:51.662296    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92834d63_fbc2_4fcc_aca8_93f66f7e6d26.slice/crio-33c4b9d25e2074f51f3f9227c947b3fe3db034dd3b22193c0138fde555e403e8 WatchSource:0}: Error finding container 33c4b9d25e2074f51f3f9227c947b3fe3db034dd3b22193c0138fde555e403e8: Status 404 returned error can't find the container with id 33c4b9d25e2074f51f3f9227c947b3fe3db034dd3b22193c0138fde555e403e8
Oct 30 20:44:51.846303 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:51.846273    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5b94c4c576-t9qdd" event={"ID":"92834d63-fbc2-4fcc-aca8-93f66f7e6d26","Type":"ContainerStarted","Data":"33c4b9d25e2074f51f3f9227c947b3fe3db034dd3b22193c0138fde555e403e8"}
Oct 30 20:44:51.960678 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:51.960626    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7988bdb8dc-nzmcx" podStartSLOduration=1.566661769 podStartE2EDuration="4.960611665s" totalImagesPullingTime="3.393949896s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:44:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:44:51.958632438 +0000 UTC m=+375.023408835" watchObservedRunningTime="2026-10-30 20:44:51.960611665 +0000 UTC m=+375.025388062"
Oct 30 20:44:52.851917 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:52.851873    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7988bdb8dc-nzmcx" event={"ID":"cfe209a2-82e3-4c1e-99d0-615d8294737a","Type":"ContainerStarted","Data":"d25275538926133ea3612e3cd4780ee9a38e69d549e309d4a44234d5fc320ee1"}
Oct 30 20:44:54.465560 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:44:54.465518    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 20:44:54.804993 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:54.804927    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-4hh9b"
Oct 30 20:44:56.878168 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:56.878128    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5b94c4c576-t9qdd" event={"ID":"92834d63-fbc2-4fcc-aca8-93f66f7e6d26","Type":"ContainerStarted","Data":"4cb88bd86223884a222fb237475ef6759ee47383429876387f54cd692af78fc4"}
Oct 30 20:44:56.878557 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:56.878267    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-5b94c4c576-t9qdd"
Oct 30 20:44:56.993922 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:44:56.993863    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-5b94c4c576-t9qdd" podStartSLOduration=2.463288006 podStartE2EDuration="6.993844869s" totalImagesPullingTime="4.530556863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:44:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:44:56.993139926 +0000 UTC m=+380.057916350" watchObservedRunningTime="2026-10-30 20:44:56.993844869 +0000 UTC m=+380.058621268"
Oct 30 20:45:07.888378 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:45:07.888349    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-5b94c4c576-t9qdd"
Oct 30 20:45:24.457869 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:45:24.457831    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 20:45:54.456886 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:45:54.456850    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 20:46:24.458417 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:46:24.458382    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 20:46:54.457510 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:46:54.457460    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 20:47:24.457747 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:47:24.457709    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 20:47:54.457987 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:47:54.457887    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 20:48:13.105592 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:13.105558    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l"]
Oct 30 20:48:13.108246 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:13.108231    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l"
Oct 30 20:48:13.148343 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:13.148311    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc9510c5-2180-41d6-90bc-9acdff0f3616-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l\" (UID: \"fc9510c5-2180-41d6-90bc-9acdff0f3616\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l"
Oct 30 20:48:13.148343 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:13.148348    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc9510c5-2180-41d6-90bc-9acdff0f3616-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l\" (UID: \"fc9510c5-2180-41d6-90bc-9acdff0f3616\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l"
Oct 30 20:48:13.148608 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:13.148390    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc9510c5-2180-41d6-90bc-9acdff0f3616-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l\" (UID: \"fc9510c5-2180-41d6-90bc-9acdff0f3616\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l"
Oct 30 20:48:13.148608 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:13.148559    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thpl6\" (UniqueName: \"kubernetes.io/projected/fc9510c5-2180-41d6-90bc-9acdff0f3616-kube-api-access-thpl6\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l\" (UID: \"fc9510c5-2180-41d6-90bc-9acdff0f3616\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l"
Oct 30 20:48:13.148723 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:13.148650    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc9510c5-2180-41d6-90bc-9acdff0f3616-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l\" (UID: \"fc9510c5-2180-41d6-90bc-9acdff0f3616\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l"
Oct 30 20:48:13.148723 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:13.148684    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc9510c5-2180-41d6-90bc-9acdff0f3616-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l\" (UID: \"fc9510c5-2180-41d6-90bc-9acdff0f3616\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l"
Oct 30 20:48:13.164761 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:13.164734    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 30 20:48:13.164761 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:13.164736    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 30 20:48:13.214670 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:13.214634    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Oct 30 20:48:13.214900 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:13.214719    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-r2j5p\""
Oct 30 20:48:13.216172 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:13.216151    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l"]
Oct 30 20:48:13.249804 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:13.249752    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc9510c5-2180-41d6-90bc-9acdff0f3616-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l\" (UID: \"fc9510c5-2180-41d6-90bc-9acdff0f3616\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l"
Oct 30 20:48:13.249977 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:13.249833    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc9510c5-2180-41d6-90bc-9acdff0f3616-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l\" (UID: \"fc9510c5-2180-41d6-90bc-9acdff0f3616\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l"
Oct 30 20:48:13.249977 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:13.249915    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc9510c5-2180-41d6-90bc-9acdff0f3616-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l\" (UID: \"fc9510c5-2180-41d6-90bc-9acdff0f3616\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l"
Oct 30 20:48:13.249977 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:13.249944    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc9510c5-2180-41d6-90bc-9acdff0f3616-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l\" (UID: \"fc9510c5-2180-41d6-90bc-9acdff0f3616\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l"
Oct 30 20:48:13.250160 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:13.249991    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc9510c5-2180-41d6-90bc-9acdff0f3616-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l\" (UID: \"fc9510c5-2180-41d6-90bc-9acdff0f3616\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l"
Oct 30 20:48:13.250160 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:13.250032    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thpl6\" (UniqueName: \"kubernetes.io/projected/fc9510c5-2180-41d6-90bc-9acdff0f3616-kube-api-access-thpl6\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l\" (UID: \"fc9510c5-2180-41d6-90bc-9acdff0f3616\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l"
Oct 30 20:48:13.250444 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:13.250417    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc9510c5-2180-41d6-90bc-9acdff0f3616-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l\" (UID: \"fc9510c5-2180-41d6-90bc-9acdff0f3616\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l"
Oct 30 20:48:13.250538 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:13.250460    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc9510c5-2180-41d6-90bc-9acdff0f3616-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l\" (UID: \"fc9510c5-2180-41d6-90bc-9acdff0f3616\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l"
Oct 30 20:48:13.250538 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:13.250425    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc9510c5-2180-41d6-90bc-9acdff0f3616-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l\" (UID: \"fc9510c5-2180-41d6-90bc-9acdff0f3616\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l"
Oct 30 20:48:13.252418 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:13.252398    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc9510c5-2180-41d6-90bc-9acdff0f3616-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l\" (UID: \"fc9510c5-2180-41d6-90bc-9acdff0f3616\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l"
Oct 30 20:48:13.252746 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:13.252725    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc9510c5-2180-41d6-90bc-9acdff0f3616-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l\" (UID: \"fc9510c5-2180-41d6-90bc-9acdff0f3616\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l"
Oct 30 20:48:13.311863 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:13.311828    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thpl6\" (UniqueName: \"kubernetes.io/projected/fc9510c5-2180-41d6-90bc-9acdff0f3616-kube-api-access-thpl6\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l\" (UID: \"fc9510c5-2180-41d6-90bc-9acdff0f3616\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l"
Oct 30 20:48:13.418747 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:13.418661    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l"
Oct 30 20:48:13.790998 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:13.790974    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l"]
Oct 30 20:48:13.793644 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:48:13.793608    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfc9510c5_2180_41d6_90bc_9acdff0f3616.slice/crio-af41aa7f945488daf4ee4cfac6141a3ec4245300205188ccc338890cbe15793a WatchSource:0}: Error finding container af41aa7f945488daf4ee4cfac6141a3ec4245300205188ccc338890cbe15793a: Status 404 returned error can't find the container with id af41aa7f945488daf4ee4cfac6141a3ec4245300205188ccc338890cbe15793a
Oct 30 20:48:13.795020 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:13.795003    5713 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:48:13.869756 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:13.869723    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l" event={"ID":"fc9510c5-2180-41d6-90bc-9acdff0f3616","Type":"ContainerStarted","Data":"af41aa7f945488daf4ee4cfac6141a3ec4245300205188ccc338890cbe15793a"}
Oct 30 20:48:15.883556 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:15.883517    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l" event={"ID":"fc9510c5-2180-41d6-90bc-9acdff0f3616","Type":"ContainerStarted","Data":"74722994788c6ea323c95792427692da083a44d1579f6f8dc1f0ba612a51aaa6"}
Oct 30 20:48:16.005468 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:16.005408    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l" podStartSLOduration=1.027734908 podStartE2EDuration="3.00539236s" totalImagesPullingTime="1.977657452s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:48:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:48:16.003770597 +0000 UTC m=+579.068546994" watchObservedRunningTime="2026-10-30 20:48:16.00539236 +0000 UTC m=+579.070168758"
Oct 30 20:48:16.419623 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:16.419582    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l"
Oct 30 20:48:19.831684 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:19.831649    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hmv85"]
Oct 30 20:48:19.834980 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:19.834963    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hmv85"
Oct 30 20:48:19.912773 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:19.912737    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb3b89e5-8e1b-4881-9400-a8cd97ff6b19-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hmv85\" (UID: \"fb3b89e5-8e1b-4881-9400-a8cd97ff6b19\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hmv85"
Oct 30 20:48:19.912773 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:19.912773    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjllj\" (UniqueName: \"kubernetes.io/projected/fb3b89e5-8e1b-4881-9400-a8cd97ff6b19-kube-api-access-vjllj\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hmv85\" (UID: \"fb3b89e5-8e1b-4881-9400-a8cd97ff6b19\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hmv85"
Oct 30 20:48:19.913041 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:19.912816    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb3b89e5-8e1b-4881-9400-a8cd97ff6b19-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hmv85\" (UID: \"fb3b89e5-8e1b-4881-9400-a8cd97ff6b19\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hmv85"
Oct 30 20:48:19.913041 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:19.912863    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb3b89e5-8e1b-4881-9400-a8cd97ff6b19-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hmv85\" (UID: \"fb3b89e5-8e1b-4881-9400-a8cd97ff6b19\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hmv85"
Oct 30 20:48:19.913041 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:19.912937    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb3b89e5-8e1b-4881-9400-a8cd97ff6b19-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hmv85\" (UID: \"fb3b89e5-8e1b-4881-9400-a8cd97ff6b19\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hmv85"
Oct 30 20:48:19.913041 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:19.913003    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb3b89e5-8e1b-4881-9400-a8cd97ff6b19-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hmv85\" (UID: \"fb3b89e5-8e1b-4881-9400-a8cd97ff6b19\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hmv85"
Oct 30 20:48:19.941834 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:19.941807    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Oct 30 20:48:19.943508 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:19.943483    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hmv85"]
Oct 30 20:48:20.013609 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:20.013573    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb3b89e5-8e1b-4881-9400-a8cd97ff6b19-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hmv85\" (UID: \"fb3b89e5-8e1b-4881-9400-a8cd97ff6b19\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hmv85"
Oct 30 20:48:20.013840 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:20.013618    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb3b89e5-8e1b-4881-9400-a8cd97ff6b19-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hmv85\" (UID: \"fb3b89e5-8e1b-4881-9400-a8cd97ff6b19\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hmv85"
Oct 30 20:48:20.013840 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:20.013637    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb3b89e5-8e1b-4881-9400-a8cd97ff6b19-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hmv85\" (UID: \"fb3b89e5-8e1b-4881-9400-a8cd97ff6b19\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hmv85"
Oct 30 20:48:20.013840 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:20.013655    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjllj\" (UniqueName: \"kubernetes.io/projected/fb3b89e5-8e1b-4881-9400-a8cd97ff6b19-kube-api-access-vjllj\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hmv85\" (UID: \"fb3b89e5-8e1b-4881-9400-a8cd97ff6b19\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hmv85"
Oct 30 20:48:20.013840 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:20.013680    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb3b89e5-8e1b-4881-9400-a8cd97ff6b19-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hmv85\" (UID: \"fb3b89e5-8e1b-4881-9400-a8cd97ff6b19\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hmv85"
Oct 30 20:48:20.013840 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:20.013820    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb3b89e5-8e1b-4881-9400-a8cd97ff6b19-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hmv85\" (UID: \"fb3b89e5-8e1b-4881-9400-a8cd97ff6b19\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hmv85"
Oct 30 20:48:20.014122 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:20.014089    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb3b89e5-8e1b-4881-9400-a8cd97ff6b19-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hmv85\" (UID: \"fb3b89e5-8e1b-4881-9400-a8cd97ff6b19\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hmv85"
Oct 30 20:48:20.014122 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:20.014100    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb3b89e5-8e1b-4881-9400-a8cd97ff6b19-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hmv85\" (UID: \"fb3b89e5-8e1b-4881-9400-a8cd97ff6b19\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hmv85"
Oct 30 20:48:20.014226 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:20.014202    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb3b89e5-8e1b-4881-9400-a8cd97ff6b19-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hmv85\" (UID: \"fb3b89e5-8e1b-4881-9400-a8cd97ff6b19\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hmv85"
Oct 30 20:48:20.016265 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:20.016243    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb3b89e5-8e1b-4881-9400-a8cd97ff6b19-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hmv85\" (UID: \"fb3b89e5-8e1b-4881-9400-a8cd97ff6b19\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hmv85"
Oct 30 20:48:20.016396 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:20.016381    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb3b89e5-8e1b-4881-9400-a8cd97ff6b19-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hmv85\" (UID: \"fb3b89e5-8e1b-4881-9400-a8cd97ff6b19\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hmv85"
Oct 30 20:48:20.072010 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:20.071983    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjllj\" (UniqueName: \"kubernetes.io/projected/fb3b89e5-8e1b-4881-9400-a8cd97ff6b19-kube-api-access-vjllj\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hmv85\" (UID: \"fb3b89e5-8e1b-4881-9400-a8cd97ff6b19\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hmv85"
Oct 30 20:48:20.147097 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:20.147006    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hmv85"
Oct 30 20:48:20.523942 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:20.523914    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hmv85"]
Oct 30 20:48:20.526294 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:48:20.526268    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb3b89e5_8e1b_4881_9400_a8cd97ff6b19.slice/crio-edf29d71337bfec6e2c38e549b29b9417343ae814ca3dd5986555a4f7d84749e WatchSource:0}: Error finding container edf29d71337bfec6e2c38e549b29b9417343ae814ca3dd5986555a4f7d84749e: Status 404 returned error can't find the container with id edf29d71337bfec6e2c38e549b29b9417343ae814ca3dd5986555a4f7d84749e
Oct 30 20:48:20.909381 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:20.909300    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hmv85" event={"ID":"fb3b89e5-8e1b-4881-9400-a8cd97ff6b19","Type":"ContainerStarted","Data":"edf29d71337bfec6e2c38e549b29b9417343ae814ca3dd5986555a4f7d84749e"}
Oct 30 20:48:21.915733 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:21.915696    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hmv85" event={"ID":"fb3b89e5-8e1b-4881-9400-a8cd97ff6b19","Type":"ContainerStarted","Data":"705be916a5a43f8956759d162564615cc38769c5b4c74f13a8fb78b3d6156f4f"}
Oct 30 20:48:22.035566 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:22.035514    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hmv85" podStartSLOduration=2.438655366 podStartE2EDuration="3.035497937s" totalImagesPullingTime="596.842571ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:48:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:48:22.03404431 +0000 UTC m=+585.098820708" watchObservedRunningTime="2026-10-30 20:48:22.035497937 +0000 UTC m=+585.100274334"
Oct 30 20:48:22.148179 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:22.148147    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hmv85"
Oct 30 20:48:23.418932 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:23.418892    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l"
Oct 30 20:48:23.431547 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:23.431521    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l"
Oct 30 20:48:23.935853 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:23.935827    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l"
Oct 30 20:48:24.465034 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:48:24.464983    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 20:48:26.438027 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:26.437988    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v2lfl"]
Oct 30 20:48:26.441616 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:26.441599    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v2lfl"
Oct 30 20:48:26.544332 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:26.544298    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Oct 30 20:48:26.546251 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:26.546228    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v2lfl"]
Oct 30 20:48:26.578140 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:26.578113    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cff5280-b60c-4285-8250-9482152de0d8-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-v2lfl\" (UID: \"2cff5280-b60c-4285-8250-9482152de0d8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v2lfl"
Oct 30 20:48:26.578318 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:26.578170    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cff5280-b60c-4285-8250-9482152de0d8-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-v2lfl\" (UID: \"2cff5280-b60c-4285-8250-9482152de0d8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v2lfl"
Oct 30 20:48:26.578318 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:26.578244    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cff5280-b60c-4285-8250-9482152de0d8-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-v2lfl\" (UID: \"2cff5280-b60c-4285-8250-9482152de0d8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v2lfl"
Oct 30 20:48:26.578318 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:26.578265    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cff5280-b60c-4285-8250-9482152de0d8-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-v2lfl\" (UID: \"2cff5280-b60c-4285-8250-9482152de0d8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v2lfl"
Oct 30 20:48:26.578318 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:26.578291    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cff5280-b60c-4285-8250-9482152de0d8-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-v2lfl\" (UID: \"2cff5280-b60c-4285-8250-9482152de0d8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v2lfl"
Oct 30 20:48:26.578454 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:26.578385    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62jdg\" (UniqueName: \"kubernetes.io/projected/2cff5280-b60c-4285-8250-9482152de0d8-kube-api-access-62jdg\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-v2lfl\" (UID: \"2cff5280-b60c-4285-8250-9482152de0d8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v2lfl"
Oct 30 20:48:26.679487 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:26.679436    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cff5280-b60c-4285-8250-9482152de0d8-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-v2lfl\" (UID: \"2cff5280-b60c-4285-8250-9482152de0d8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v2lfl"
Oct 30 20:48:26.679487 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:26.679488    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cff5280-b60c-4285-8250-9482152de0d8-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-v2lfl\" (UID: \"2cff5280-b60c-4285-8250-9482152de0d8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v2lfl"
Oct 30 20:48:26.679694 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:26.679506    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cff5280-b60c-4285-8250-9482152de0d8-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-v2lfl\" (UID: \"2cff5280-b60c-4285-8250-9482152de0d8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v2lfl"
Oct 30 20:48:26.679694 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:26.679532    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cff5280-b60c-4285-8250-9482152de0d8-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-v2lfl\" (UID: \"2cff5280-b60c-4285-8250-9482152de0d8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v2lfl"
Oct 30 20:48:26.679694 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:26.679607    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62jdg\" (UniqueName: \"kubernetes.io/projected/2cff5280-b60c-4285-8250-9482152de0d8-kube-api-access-62jdg\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-v2lfl\" (UID: \"2cff5280-b60c-4285-8250-9482152de0d8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v2lfl"
Oct 30 20:48:26.679694 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:26.679646    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cff5280-b60c-4285-8250-9482152de0d8-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-v2lfl\" (UID: \"2cff5280-b60c-4285-8250-9482152de0d8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v2lfl"
Oct 30 20:48:26.679938 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:26.679892    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cff5280-b60c-4285-8250-9482152de0d8-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-v2lfl\" (UID: \"2cff5280-b60c-4285-8250-9482152de0d8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v2lfl"
Oct 30 20:48:26.679989 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:26.679945    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cff5280-b60c-4285-8250-9482152de0d8-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-v2lfl\" (UID: \"2cff5280-b60c-4285-8250-9482152de0d8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v2lfl"
Oct 30 20:48:26.680041 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:26.680027    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cff5280-b60c-4285-8250-9482152de0d8-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-v2lfl\" (UID: \"2cff5280-b60c-4285-8250-9482152de0d8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v2lfl"
Oct 30 20:48:26.682060 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:26.682036    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cff5280-b60c-4285-8250-9482152de0d8-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-v2lfl\" (UID: \"2cff5280-b60c-4285-8250-9482152de0d8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v2lfl"
Oct 30 20:48:26.682293 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:26.682276    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cff5280-b60c-4285-8250-9482152de0d8-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-v2lfl\" (UID: \"2cff5280-b60c-4285-8250-9482152de0d8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v2lfl"
Oct 30 20:48:26.740471 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:26.740394    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62jdg\" (UniqueName: \"kubernetes.io/projected/2cff5280-b60c-4285-8250-9482152de0d8-kube-api-access-62jdg\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-v2lfl\" (UID: \"2cff5280-b60c-4285-8250-9482152de0d8\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v2lfl"
Oct 30 20:48:26.754327 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:26.754289    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v2lfl"
Oct 30 20:48:27.131364 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:27.131315    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v2lfl"]
Oct 30 20:48:27.133921 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:48:27.133894    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cff5280_b60c_4285_8250_9482152de0d8.slice/crio-86e9ca56dbdfe786ba4458345ddfcbb249c3bf4809c65047f65eb7bd44dea17f WatchSource:0}: Error finding container 86e9ca56dbdfe786ba4458345ddfcbb249c3bf4809c65047f65eb7bd44dea17f: Status 404 returned error can't find the container with id 86e9ca56dbdfe786ba4458345ddfcbb249c3bf4809c65047f65eb7bd44dea17f
Oct 30 20:48:27.947846 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:27.947815    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v2lfl" event={"ID":"2cff5280-b60c-4285-8250-9482152de0d8","Type":"ContainerStarted","Data":"8a4d05777ece6cf079b1242cb32bca4ad30db21cbc631274d213ca194544b7bd"}
Oct 30 20:48:27.948261 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:27.947868    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v2lfl" event={"ID":"2cff5280-b60c-4285-8250-9482152de0d8","Type":"ContainerStarted","Data":"86e9ca56dbdfe786ba4458345ddfcbb249c3bf4809c65047f65eb7bd44dea17f"}
Oct 30 20:48:28.073708 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:28.073594    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v2lfl" podStartSLOduration=1.488181445 podStartE2EDuration="2.073580035s" totalImagesPullingTime="585.39859ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:48:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:48:28.072441434 +0000 UTC m=+591.137217832" watchObservedRunningTime="2026-10-30 20:48:28.073580035 +0000 UTC m=+591.138356432"
Oct 30 20:48:28.755186 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:28.755153    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v2lfl"
Oct 30 20:48:30.147487 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:30.147456    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hmv85"
Oct 30 20:48:30.159924 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:30.159901    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hmv85"
Oct 30 20:48:30.980120 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:30.980082    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hmv85"
Oct 30 20:48:36.755185 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:36.755144    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v2lfl"
Oct 30 20:48:36.767840 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:36.767808    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v2lfl"
Oct 30 20:48:37.012346 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:37.012263    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v2lfl"
Oct 30 20:48:37.126886 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:37.126860    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d4597073-8ff0-4a4e-933c-c64e934fe977/prometheus/0.log"
Oct 30 20:48:37.134599 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:37.134581    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d4597073-8ff0-4a4e-933c-c64e934fe977/prometheus/0.log"
Oct 30 20:48:37.140578 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:37.140557    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r7lpf_764461d5-e0b2-4856-a97e-0c5d3d34a497/ovn-acl-logging/0.log"
Oct 30 20:48:37.149934 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:37.149914    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r7lpf_764461d5-e0b2-4856-a97e-0c5d3d34a497/ovn-acl-logging/0.log"
Oct 30 20:48:37.242571 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:37.242542    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-mb6fg"]
Oct 30 20:48:37.246133 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:37.246111    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mb6fg"
Oct 30 20:48:37.298724 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:37.298654    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Oct 30 20:48:37.310111 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:37.310088    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-mb6fg"]
Oct 30 20:48:37.380925 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:37.380896    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ae4eaba-11ff-45e4-95d9-9bf25c06d42f-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mb6fg\" (UID: \"1ae4eaba-11ff-45e4-95d9-9bf25c06d42f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mb6fg"
Oct 30 20:48:37.380925 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:37.380929    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ae4eaba-11ff-45e4-95d9-9bf25c06d42f-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mb6fg\" (UID: \"1ae4eaba-11ff-45e4-95d9-9bf25c06d42f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mb6fg"
Oct 30 20:48:37.381124 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:37.380951    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ae4eaba-11ff-45e4-95d9-9bf25c06d42f-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mb6fg\" (UID: \"1ae4eaba-11ff-45e4-95d9-9bf25c06d42f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mb6fg"
Oct 30 20:48:37.381124 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:37.381023    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ae4eaba-11ff-45e4-95d9-9bf25c06d42f-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mb6fg\" (UID: \"1ae4eaba-11ff-45e4-95d9-9bf25c06d42f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mb6fg"
Oct 30 20:48:37.381124 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:37.381070    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qcm8\" (UniqueName: \"kubernetes.io/projected/1ae4eaba-11ff-45e4-95d9-9bf25c06d42f-kube-api-access-8qcm8\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mb6fg\" (UID: \"1ae4eaba-11ff-45e4-95d9-9bf25c06d42f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mb6fg"
Oct 30 20:48:37.381124 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:37.381114    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ae4eaba-11ff-45e4-95d9-9bf25c06d42f-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mb6fg\" (UID: \"1ae4eaba-11ff-45e4-95d9-9bf25c06d42f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mb6fg"
Oct 30 20:48:37.482368 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:37.482336    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ae4eaba-11ff-45e4-95d9-9bf25c06d42f-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mb6fg\" (UID: \"1ae4eaba-11ff-45e4-95d9-9bf25c06d42f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mb6fg"
Oct 30 20:48:37.482368 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:37.482370    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ae4eaba-11ff-45e4-95d9-9bf25c06d42f-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mb6fg\" (UID: \"1ae4eaba-11ff-45e4-95d9-9bf25c06d42f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mb6fg"
Oct 30 20:48:37.482627 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:37.482386    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ae4eaba-11ff-45e4-95d9-9bf25c06d42f-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mb6fg\" (UID: \"1ae4eaba-11ff-45e4-95d9-9bf25c06d42f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mb6fg"
Oct 30 20:48:37.482627 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:37.482403    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ae4eaba-11ff-45e4-95d9-9bf25c06d42f-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mb6fg\" (UID: \"1ae4eaba-11ff-45e4-95d9-9bf25c06d42f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mb6fg"
Oct 30 20:48:37.482627 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:37.482458    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qcm8\" (UniqueName: \"kubernetes.io/projected/1ae4eaba-11ff-45e4-95d9-9bf25c06d42f-kube-api-access-8qcm8\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mb6fg\" (UID: \"1ae4eaba-11ff-45e4-95d9-9bf25c06d42f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mb6fg"
Oct 30 20:48:37.482627 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:37.482490    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ae4eaba-11ff-45e4-95d9-9bf25c06d42f-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mb6fg\" (UID: \"1ae4eaba-11ff-45e4-95d9-9bf25c06d42f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mb6fg"
Oct 30 20:48:37.482825 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:37.482771    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ae4eaba-11ff-45e4-95d9-9bf25c06d42f-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mb6fg\" (UID: \"1ae4eaba-11ff-45e4-95d9-9bf25c06d42f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mb6fg"
Oct 30 20:48:37.482911 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:37.482895    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ae4eaba-11ff-45e4-95d9-9bf25c06d42f-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mb6fg\" (UID: \"1ae4eaba-11ff-45e4-95d9-9bf25c06d42f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mb6fg"
Oct 30 20:48:37.482947 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:37.482926    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ae4eaba-11ff-45e4-95d9-9bf25c06d42f-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mb6fg\" (UID: \"1ae4eaba-11ff-45e4-95d9-9bf25c06d42f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mb6fg"
Oct 30 20:48:37.484921 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:37.484900    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ae4eaba-11ff-45e4-95d9-9bf25c06d42f-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mb6fg\" (UID: \"1ae4eaba-11ff-45e4-95d9-9bf25c06d42f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mb6fg"
Oct 30 20:48:37.485213 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:37.485196    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ae4eaba-11ff-45e4-95d9-9bf25c06d42f-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mb6fg\" (UID: \"1ae4eaba-11ff-45e4-95d9-9bf25c06d42f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mb6fg"
Oct 30 20:48:37.546888 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:37.546856    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qcm8\" (UniqueName: \"kubernetes.io/projected/1ae4eaba-11ff-45e4-95d9-9bf25c06d42f-kube-api-access-8qcm8\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-mb6fg\" (UID: \"1ae4eaba-11ff-45e4-95d9-9bf25c06d42f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mb6fg"
Oct 30 20:48:37.560763 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:37.560706    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mb6fg"
Oct 30 20:48:37.939042 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:37.939022    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-mb6fg"]
Oct 30 20:48:37.941112 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:48:37.941078    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ae4eaba_11ff_45e4_95d9_9bf25c06d42f.slice/crio-0ee73c9a624d5e170945f761bcf7ac04ca8acb9e1a0ff51af0a3188f7f498c36 WatchSource:0}: Error finding container 0ee73c9a624d5e170945f761bcf7ac04ca8acb9e1a0ff51af0a3188f7f498c36: Status 404 returned error can't find the container with id 0ee73c9a624d5e170945f761bcf7ac04ca8acb9e1a0ff51af0a3188f7f498c36
Oct 30 20:48:38.008342 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:38.008309    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mb6fg" event={"ID":"1ae4eaba-11ff-45e4-95d9-9bf25c06d42f","Type":"ContainerStarted","Data":"0ee73c9a624d5e170945f761bcf7ac04ca8acb9e1a0ff51af0a3188f7f498c36"}
Oct 30 20:48:39.016300 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:39.016264    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mb6fg" event={"ID":"1ae4eaba-11ff-45e4-95d9-9bf25c06d42f","Type":"ContainerStarted","Data":"835b0c4f245ed0b40203dce5690dc87fbd8dd6316a8c95a3b7556008353deb6b"}
Oct 30 20:48:39.145308 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:39.145251    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mb6fg" podStartSLOduration=1.549520641 podStartE2EDuration="2.145235052s" totalImagesPullingTime="595.714411ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:48:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:48:39.143931548 +0000 UTC m=+602.208707947" watchObservedRunningTime="2026-10-30 20:48:39.145235052 +0000 UTC m=+602.210011484"
Oct 30 20:48:39.561880 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:39.561845    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mb6fg"
Oct 30 20:48:40.596057 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:40.596026    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-dvl5x"]
Oct 30 20:48:40.599617 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:40.599601    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dvl5x"
Oct 30 20:48:40.663659 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:40.663627    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Oct 30 20:48:40.705806 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:40.705742    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-dvl5x"]
Oct 30 20:48:40.712164 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:40.712134    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cc10a3b-ecd9-4dba-a799-7c4de06b7523-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-dvl5x\" (UID: \"4cc10a3b-ecd9-4dba-a799-7c4de06b7523\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dvl5x"
Oct 30 20:48:40.712313 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:40.712167    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cc10a3b-ecd9-4dba-a799-7c4de06b7523-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-dvl5x\" (UID: \"4cc10a3b-ecd9-4dba-a799-7c4de06b7523\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dvl5x"
Oct 30 20:48:40.712354 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:40.712312    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cc10a3b-ecd9-4dba-a799-7c4de06b7523-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-dvl5x\" (UID: \"4cc10a3b-ecd9-4dba-a799-7c4de06b7523\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dvl5x"
Oct 30 20:48:40.712354 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:40.712332    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szt7s\" (UniqueName: \"kubernetes.io/projected/4cc10a3b-ecd9-4dba-a799-7c4de06b7523-kube-api-access-szt7s\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-dvl5x\" (UID: \"4cc10a3b-ecd9-4dba-a799-7c4de06b7523\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dvl5x"
Oct 30 20:48:40.712418 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:40.712389    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cc10a3b-ecd9-4dba-a799-7c4de06b7523-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-dvl5x\" (UID: \"4cc10a3b-ecd9-4dba-a799-7c4de06b7523\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dvl5x"
Oct 30 20:48:40.712454 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:40.712424    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cc10a3b-ecd9-4dba-a799-7c4de06b7523-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-dvl5x\" (UID: \"4cc10a3b-ecd9-4dba-a799-7c4de06b7523\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dvl5x"
Oct 30 20:48:40.813125 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:40.813088    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cc10a3b-ecd9-4dba-a799-7c4de06b7523-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-dvl5x\" (UID: \"4cc10a3b-ecd9-4dba-a799-7c4de06b7523\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dvl5x"
Oct 30 20:48:40.813316 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:40.813134    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cc10a3b-ecd9-4dba-a799-7c4de06b7523-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-dvl5x\" (UID: \"4cc10a3b-ecd9-4dba-a799-7c4de06b7523\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dvl5x"
Oct 30 20:48:40.813316 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:40.813155    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cc10a3b-ecd9-4dba-a799-7c4de06b7523-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-dvl5x\" (UID: \"4cc10a3b-ecd9-4dba-a799-7c4de06b7523\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dvl5x"
Oct 30 20:48:40.813316 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:40.813245    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cc10a3b-ecd9-4dba-a799-7c4de06b7523-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-dvl5x\" (UID: \"4cc10a3b-ecd9-4dba-a799-7c4de06b7523\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dvl5x"
Oct 30 20:48:40.813316 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:40.813268    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szt7s\" (UniqueName: \"kubernetes.io/projected/4cc10a3b-ecd9-4dba-a799-7c4de06b7523-kube-api-access-szt7s\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-dvl5x\" (UID: \"4cc10a3b-ecd9-4dba-a799-7c4de06b7523\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dvl5x"
Oct 30 20:48:40.813316 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:40.813305    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cc10a3b-ecd9-4dba-a799-7c4de06b7523-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-dvl5x\" (UID: \"4cc10a3b-ecd9-4dba-a799-7c4de06b7523\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dvl5x"
Oct 30 20:48:40.813550 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:40.813503    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cc10a3b-ecd9-4dba-a799-7c4de06b7523-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-dvl5x\" (UID: \"4cc10a3b-ecd9-4dba-a799-7c4de06b7523\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dvl5x"
Oct 30 20:48:40.813608 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:40.813551    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cc10a3b-ecd9-4dba-a799-7c4de06b7523-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-dvl5x\" (UID: \"4cc10a3b-ecd9-4dba-a799-7c4de06b7523\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dvl5x"
Oct 30 20:48:40.813661 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:40.813616    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cc10a3b-ecd9-4dba-a799-7c4de06b7523-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-dvl5x\" (UID: \"4cc10a3b-ecd9-4dba-a799-7c4de06b7523\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dvl5x"
Oct 30 20:48:40.815741 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:40.815715    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cc10a3b-ecd9-4dba-a799-7c4de06b7523-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-dvl5x\" (UID: \"4cc10a3b-ecd9-4dba-a799-7c4de06b7523\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dvl5x"
Oct 30 20:48:40.816151 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:40.816130    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cc10a3b-ecd9-4dba-a799-7c4de06b7523-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-dvl5x\" (UID: \"4cc10a3b-ecd9-4dba-a799-7c4de06b7523\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dvl5x"
Oct 30 20:48:40.872464 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:40.872393    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szt7s\" (UniqueName: \"kubernetes.io/projected/4cc10a3b-ecd9-4dba-a799-7c4de06b7523-kube-api-access-szt7s\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-dvl5x\" (UID: \"4cc10a3b-ecd9-4dba-a799-7c4de06b7523\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dvl5x"
Oct 30 20:48:40.912068 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:40.912035    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dvl5x"
Oct 30 20:48:41.291810 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:41.291768    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-dvl5x"]
Oct 30 20:48:41.294338 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:48:41.294314    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cc10a3b_ecd9_4dba_a799_7c4de06b7523.slice/crio-374450999e0ea586bab2d13255ad05d079ae58754c599e770c93bf07354bb912 WatchSource:0}: Error finding container 374450999e0ea586bab2d13255ad05d079ae58754c599e770c93bf07354bb912: Status 404 returned error can't find the container with id 374450999e0ea586bab2d13255ad05d079ae58754c599e770c93bf07354bb912
Oct 30 20:48:42.032968 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:42.032933    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dvl5x" event={"ID":"4cc10a3b-ecd9-4dba-a799-7c4de06b7523","Type":"ContainerStarted","Data":"05b536b5d18e658cc0d3ca4c3f5eb1aaae3b9883845b4c1c5bf0f64005f81474"}
Oct 30 20:48:42.033432 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:42.032987    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dvl5x" event={"ID":"4cc10a3b-ecd9-4dba-a799-7c4de06b7523","Type":"ContainerStarted","Data":"374450999e0ea586bab2d13255ad05d079ae58754c599e770c93bf07354bb912"}
Oct 30 20:48:42.155327 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:42.155228    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dvl5x" podStartSLOduration=1.545272356 podStartE2EDuration="2.155212068s" totalImagesPullingTime="609.939712ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:48:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:48:42.154475103 +0000 UTC m=+605.219251503" watchObservedRunningTime="2026-10-30 20:48:42.155212068 +0000 UTC m=+605.219988466"
Oct 30 20:48:42.913036 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:42.913000    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dvl5x"
Oct 30 20:48:47.561521 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:47.561480    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mb6fg"
Oct 30 20:48:47.574230 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:47.574199    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mb6fg"
Oct 30 20:48:48.076193 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:48.076166    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-mb6fg"
Oct 30 20:48:50.912181 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:50.912138    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dvl5x"
Oct 30 20:48:50.925272 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:50.925244    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dvl5x"
Oct 30 20:48:51.092198 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:51.092171    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-dvl5x"
Oct 30 20:48:52.038368 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:52.038327    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-sjpxz"]
Oct 30 20:48:52.044652 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:52.044628    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-sjpxz"
Oct 30 20:48:52.090766 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:52.090739    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7jfsh"]
Oct 30 20:48:52.094591 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:52.094568    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7jfsh"
Oct 30 20:48:52.110719 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:52.110697    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-qbqvq\""
Oct 30 20:48:52.146651 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:52.146622    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-sjpxz"]
Oct 30 20:48:52.195384 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:52.195350    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7jfsh"]
Oct 30 20:48:52.221722 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:52.221690    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f321ee6-f63d-43b5-a1e9-983985a789ff-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-7jfsh\" (UID: \"2f321ee6-f63d-43b5-a1e9-983985a789ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7jfsh"
Oct 30 20:48:52.221920 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:52.221743    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f543e9a0-a5a6-417a-8f00-b0627a5c1776-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-sjpxz\" (UID: \"f543e9a0-a5a6-417a-8f00-b0627a5c1776\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-sjpxz"
Oct 30 20:48:52.221920 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:52.221896    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f543e9a0-a5a6-417a-8f00-b0627a5c1776-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-sjpxz\" (UID: \"f543e9a0-a5a6-417a-8f00-b0627a5c1776\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-sjpxz"
Oct 30 20:48:52.222045 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:52.221942    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f321ee6-f63d-43b5-a1e9-983985a789ff-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-7jfsh\" (UID: \"2f321ee6-f63d-43b5-a1e9-983985a789ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7jfsh"
Oct 30 20:48:52.222045 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:52.222030    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f543e9a0-a5a6-417a-8f00-b0627a5c1776-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-sjpxz\" (UID: \"f543e9a0-a5a6-417a-8f00-b0627a5c1776\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-sjpxz"
Oct 30 20:48:52.222160 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:52.222059    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f543e9a0-a5a6-417a-8f00-b0627a5c1776-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-sjpxz\" (UID: \"f543e9a0-a5a6-417a-8f00-b0627a5c1776\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-sjpxz"
Oct 30 20:48:52.222160 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:52.222116    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f321ee6-f63d-43b5-a1e9-983985a789ff-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-7jfsh\" (UID: \"2f321ee6-f63d-43b5-a1e9-983985a789ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7jfsh"
Oct 30 20:48:52.222160 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:52.222141    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f321ee6-f63d-43b5-a1e9-983985a789ff-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-7jfsh\" (UID: \"2f321ee6-f63d-43b5-a1e9-983985a789ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7jfsh"
Oct 30 20:48:52.222300 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:52.222170    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f543e9a0-a5a6-417a-8f00-b0627a5c1776-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-sjpxz\" (UID: \"f543e9a0-a5a6-417a-8f00-b0627a5c1776\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-sjpxz"
Oct 30 20:48:52.222300 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:52.222186    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f543e9a0-a5a6-417a-8f00-b0627a5c1776-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-sjpxz\" (UID: \"f543e9a0-a5a6-417a-8f00-b0627a5c1776\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-sjpxz"
Oct 30 20:48:52.222402 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:52.222296    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f321ee6-f63d-43b5-a1e9-983985a789ff-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-7jfsh\" (UID: \"2f321ee6-f63d-43b5-a1e9-983985a789ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7jfsh"
Oct 30 20:48:52.222402 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:52.222359    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f321ee6-f63d-43b5-a1e9-983985a789ff-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-7jfsh\" (UID: \"2f321ee6-f63d-43b5-a1e9-983985a789ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7jfsh"
Oct 30 20:48:52.222504 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:52.222447    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f321ee6-f63d-43b5-a1e9-983985a789ff-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-7jfsh\" (UID: \"2f321ee6-f63d-43b5-a1e9-983985a789ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7jfsh"
Oct 30 20:48:52.222504 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:52.222481    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f543e9a0-a5a6-417a-8f00-b0627a5c1776-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-sjpxz\" (UID: \"f543e9a0-a5a6-417a-8f00-b0627a5c1776\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-sjpxz"
Oct 30 20:48:52.222621 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:52.222604    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p88qr\" (UniqueName: \"kubernetes.io/projected/2f321ee6-f63d-43b5-a1e9-983985a789ff-kube-api-access-p88qr\") pod \"maas-default-gateway-openshift-default-769df7846c-7jfsh\" (UID: \"2f321ee6-f63d-43b5-a1e9-983985a789ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7jfsh"
Oct 30 20:48:52.222657 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:52.222633    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ddbn\" (UniqueName: \"kubernetes.io/projected/f543e9a0-a5a6-417a-8f00-b0627a5c1776-kube-api-access-4ddbn\") pod \"maas-default-gateway-openshift-default-769df7846c-sjpxz\" (UID: \"f543e9a0-a5a6-417a-8f00-b0627a5c1776\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-sjpxz"
Oct 30 20:48:52.222657 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:52.222653    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f321ee6-f63d-43b5-a1e9-983985a789ff-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-7jfsh\" (UID: \"2f321ee6-f63d-43b5-a1e9-983985a789ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7jfsh"
Oct 30 20:48:52.222759 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:52.222673    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f543e9a0-a5a6-417a-8f00-b0627a5c1776-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-sjpxz\" (UID: \"f543e9a0-a5a6-417a-8f00-b0627a5c1776\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-sjpxz"
Oct 30 20:48:52.323957 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:52.323871    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f321ee6-f63d-43b5-a1e9-983985a789ff-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-7jfsh\" (UID: \"2f321ee6-f63d-43b5-a1e9-983985a789ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7jfsh"
Oct 30 20:48:52.323957 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:52.323915    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f321ee6-f63d-43b5-a1e9-983985a789ff-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-7jfsh\" (UID: \"2f321ee6-f63d-43b5-a1e9-983985a789ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7jfsh"
Oct 30 20:48:52.324209 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:52.324044    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f321ee6-f63d-43b5-a1e9-983985a789ff-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-7jfsh\" (UID: \"2f321ee6-f63d-43b5-a1e9-983985a789ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7jfsh"
Oct 30 20:48:52.324209 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:52.324079    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f543e9a0-a5a6-417a-8f00-b0627a5c1776-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-sjpxz\" (UID: \"f543e9a0-a5a6-417a-8f00-b0627a5c1776\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-sjpxz"
Oct 30 20:48:52.324209 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:52.324114    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p88qr\" (UniqueName: \"kubernetes.io/projected/2f321ee6-f63d-43b5-a1e9-983985a789ff-kube-api-access-p88qr\") pod \"maas-default-gateway-openshift-default-769df7846c-7jfsh\" (UID: \"2f321ee6-f63d-43b5-a1e9-983985a789ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7jfsh"
Oct 30 20:48:52.324209 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:52.324146    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ddbn\" (UniqueName: \"kubernetes.io/projected/f543e9a0-a5a6-417a-8f00-b0627a5c1776-kube-api-access-4ddbn\") pod \"maas-default-gateway-openshift-default-769df7846c-sjpxz\" (UID: \"f543e9a0-a5a6-417a-8f00-b0627a5c1776\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-sjpxz"
Oct 30 20:48:52.324209 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:52.324167    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f321ee6-f63d-43b5-a1e9-983985a789ff-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-7jfsh\" (UID: \"2f321ee6-f63d-43b5-a1e9-983985a789ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7jfsh"
Oct 30 20:48:52.324209 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:52.324189    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f543e9a0-a5a6-417a-8f00-b0627a5c1776-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-sjpxz\" (UID: \"f543e9a0-a5a6-417a-8f00-b0627a5c1776\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-sjpxz"
Oct 30 20:48:52.324526 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:52.324228    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f321ee6-f63d-43b5-a1e9-983985a789ff-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-7jfsh\" (UID: \"2f321ee6-f63d-43b5-a1e9-983985a789ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7jfsh"
Oct 30 20:48:52.324526 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:52.324351    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f321ee6-f63d-43b5-a1e9-983985a789ff-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-7jfsh\" (UID: \"2f321ee6-f63d-43b5-a1e9-983985a789ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7jfsh"
Oct 30 20:48:52.324526 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:52.324408    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f543e9a0-a5a6-417a-8f00-b0627a5c1776-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-sjpxz\" (UID: \"f543e9a0-a5a6-417a-8f00-b0627a5c1776\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-sjpxz"
Oct 30 20:48:52.324526 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:52.324456    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f543e9a0-a5a6-417a-8f00-b0627a5c1776-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-sjpxz\" (UID: \"f543e9a0-a5a6-417a-8f00-b0627a5c1776\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-sjpxz"
Oct 30 20:48:52.324526 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:52.324494    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f321ee6-f63d-43b5-a1e9-983985a789ff-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-7jfsh\" (UID: \"2f321ee6-f63d-43b5-a1e9-983985a789ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7jfsh"
Oct 30 20:48:52.324823 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:52.324560    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f543e9a0-a5a6-417a-8f00-b0627a5c1776-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-sjpxz\" (UID: \"f543e9a0-a5a6-417a-8f00-b0627a5c1776\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-sjpxz"
Oct 30 20:48:52.324823 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:52.324595    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f543e9a0-a5a6-417a-8f00-b0627a5c1776-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-sjpxz\" (UID: \"f543e9a0-a5a6-417a-8f00-b0627a5c1776\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-sjpxz"
Oct 30 20:48:52.324823 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:52.324614    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f321ee6-f63d-43b5-a1e9-983985a789ff-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-7jfsh\" (UID: \"2f321ee6-f63d-43b5-a1e9-983985a789ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7jfsh"
Oct 30 20:48:52.324998 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:52.324618    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f321ee6-f63d-43b5-a1e9-983985a789ff-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-7jfsh\" (UID: \"2f321ee6-f63d-43b5-a1e9-983985a789ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7jfsh"
Oct 30 20:48:52.324998 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:52.324874    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f321ee6-f63d-43b5-a1e9-983985a789ff-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-7jfsh\" (UID: \"2f321ee6-f63d-43b5-a1e9-983985a789ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7jfsh"
Oct 30 20:48:52.324998 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:52.324893    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f321ee6-f63d-43b5-a1e9-983985a789ff-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-7jfsh\" (UID: \"2f321ee6-f63d-43b5-a1e9-983985a789ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7jfsh"
Oct 30 20:48:52.324998 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:52.324975    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f321ee6-f63d-43b5-a1e9-983985a789ff-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-7jfsh\" (UID: \"2f321ee6-f63d-43b5-a1e9-983985a789ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7jfsh"
Oct 30 20:48:52.325217 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:52.325156    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f543e9a0-a5a6-417a-8f00-b0627a5c1776-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-sjpxz\" (UID: \"f543e9a0-a5a6-417a-8f00-b0627a5c1776\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-sjpxz"
Oct 30 20:48:52.325659 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:52.325360    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f321ee6-f63d-43b5-a1e9-983985a789ff-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-7jfsh\" (UID: \"2f321ee6-f63d-43b5-a1e9-983985a789ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7jfsh"
Oct 30 20:48:52.325659 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:52.325438    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f543e9a0-a5a6-417a-8f00-b0627a5c1776-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-sjpxz\" (UID: \"f543e9a0-a5a6-417a-8f00-b0627a5c1776\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-sjpxz"
Oct 30 20:48:52.325659 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:52.325470    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f543e9a0-a5a6-417a-8f00-b0627a5c1776-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-sjpxz\" (UID: \"f543e9a0-a5a6-417a-8f00-b0627a5c1776\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-sjpxz"
Oct 30 20:48:52.325659 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:52.325614    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f543e9a0-a5a6-417a-8f00-b0627a5c1776-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-sjpxz\" (UID: \"f543e9a0-a5a6-417a-8f00-b0627a5c1776\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-sjpxz"
Oct 30 20:48:52.326027 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:52.325768    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f543e9a0-a5a6-417a-8f00-b0627a5c1776-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-sjpxz\" (UID: \"f543e9a0-a5a6-417a-8f00-b0627a5c1776\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-sjpxz"
Oct 30 20:48:52.326027 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:52.325768    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f543e9a0-a5a6-417a-8f00-b0627a5c1776-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-sjpxz\" (UID: \"f543e9a0-a5a6-417a-8f00-b0627a5c1776\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-sjpxz"
Oct 30 20:48:52.326027 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:52.325899    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f543e9a0-a5a6-417a-8f00-b0627a5c1776-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-sjpxz\" (UID: \"f543e9a0-a5a6-417a-8f00-b0627a5c1776\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-sjpxz"
Oct 30 20:48:52.327007 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:52.326979    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f321ee6-f63d-43b5-a1e9-983985a789ff-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-7jfsh\" (UID: \"2f321ee6-f63d-43b5-a1e9-983985a789ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7jfsh"
Oct 30 20:48:52.327104 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:52.326981    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f543e9a0-a5a6-417a-8f00-b0627a5c1776-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-sjpxz\" (UID: \"f543e9a0-a5a6-417a-8f00-b0627a5c1776\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-sjpxz"
Oct 30 20:48:52.327104 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:52.326984    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f321ee6-f63d-43b5-a1e9-983985a789ff-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-7jfsh\" (UID: \"2f321ee6-f63d-43b5-a1e9-983985a789ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7jfsh"
Oct 30 20:48:52.327174 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:52.327112    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f543e9a0-a5a6-417a-8f00-b0627a5c1776-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-sjpxz\" (UID: \"f543e9a0-a5a6-417a-8f00-b0627a5c1776\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-sjpxz"
Oct 30 20:48:52.383135 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:52.383102    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ddbn\" (UniqueName: \"kubernetes.io/projected/f543e9a0-a5a6-417a-8f00-b0627a5c1776-kube-api-access-4ddbn\") pod \"maas-default-gateway-openshift-default-769df7846c-sjpxz\" (UID: \"f543e9a0-a5a6-417a-8f00-b0627a5c1776\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-sjpxz"
Oct 30 20:48:52.385209 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:52.385187    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f321ee6-f63d-43b5-a1e9-983985a789ff-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-7jfsh\" (UID: \"2f321ee6-f63d-43b5-a1e9-983985a789ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7jfsh"
Oct 30 20:48:52.386942 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:52.386919    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f543e9a0-a5a6-417a-8f00-b0627a5c1776-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-sjpxz\" (UID: \"f543e9a0-a5a6-417a-8f00-b0627a5c1776\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-sjpxz"
Oct 30 20:48:52.387342 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:52.387325    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p88qr\" (UniqueName: \"kubernetes.io/projected/2f321ee6-f63d-43b5-a1e9-983985a789ff-kube-api-access-p88qr\") pod \"maas-default-gateway-openshift-default-769df7846c-7jfsh\" (UID: \"2f321ee6-f63d-43b5-a1e9-983985a789ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7jfsh"
Oct 30 20:48:52.407339 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:52.407318    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7jfsh"
Oct 30 20:48:52.658252 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:52.658164    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-sjpxz"
Oct 30 20:48:52.839132 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:52.839103    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7jfsh"]
Oct 30 20:48:52.842210 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:48:52.842180    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f321ee6_f63d_43b5_a1e9_983985a789ff.slice/crio-58729d3db1ce9773115663c2b59f96491cf842c96081813c83aa6789184400fd WatchSource:0}: Error finding container 58729d3db1ce9773115663c2b59f96491cf842c96081813c83aa6789184400fd: Status 404 returned error can't find the container with id 58729d3db1ce9773115663c2b59f96491cf842c96081813c83aa6789184400fd
Oct 30 20:48:52.844511 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:52.844482    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 20:48:52.844581 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:52.844555    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 20:48:52.844615 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:52.844587    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 20:48:53.057946 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:48:53.057910    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf543e9a0_a5a6_417a_8f00_b0627a5c1776.slice/crio-0d6a015e059c74dd84dc452a6009893c90eb22deb024b97e409784d8ca459e58 WatchSource:0}: Error finding container 0d6a015e059c74dd84dc452a6009893c90eb22deb024b97e409784d8ca459e58: Status 404 returned error can't find the container with id 0d6a015e059c74dd84dc452a6009893c90eb22deb024b97e409784d8ca459e58
Oct 30 20:48:53.060982 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:53.060892    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 20:48:53.061120 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:53.061100    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 20:48:53.061213 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:53.061193    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 20:48:53.064311 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:53.064274    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-sjpxz"]
Oct 30 20:48:53.099173 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:53.099146    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-sjpxz" event={"ID":"f543e9a0-a5a6-417a-8f00-b0627a5c1776","Type":"ContainerStarted","Data":"0d6a015e059c74dd84dc452a6009893c90eb22deb024b97e409784d8ca459e58"}
Oct 30 20:48:53.101141 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:53.101108    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7jfsh" event={"ID":"2f321ee6-f63d-43b5-a1e9-983985a789ff","Type":"ContainerStarted","Data":"2ee967897bdc9567d9005aa1b3818cc5ea9b5d72a1dd38998a2675edb05dbf77"}
Oct 30 20:48:53.101255 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:53.101221    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7jfsh" event={"ID":"2f321ee6-f63d-43b5-a1e9-983985a789ff","Type":"ContainerStarted","Data":"58729d3db1ce9773115663c2b59f96491cf842c96081813c83aa6789184400fd"}
Oct 30 20:48:53.237355 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:53.237294    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7jfsh" podStartSLOduration=1.237274154 podStartE2EDuration="1.237274154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:48:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:48:53.23290447 +0000 UTC m=+616.297680868" watchObservedRunningTime="2026-10-30 20:48:53.237274154 +0000 UTC m=+616.302050553"
Oct 30 20:48:53.407993 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:53.407903    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7jfsh"
Oct 30 20:48:53.409481 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:53.409447    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-7jfsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 30 20:48:53.409615 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:53.409501    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7jfsh" podUID="2f321ee6-f63d-43b5-a1e9-983985a789ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 30 20:48:54.108290 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:54.108251    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-sjpxz" event={"ID":"f543e9a0-a5a6-417a-8f00-b0627a5c1776","Type":"ContainerStarted","Data":"639cde1a6ec57812a7185a4df695ef7d15ae78553fa1c7389e2c179598726ec7"}
Oct 30 20:48:54.237813 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:54.234219    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-sjpxz" podStartSLOduration=3.234200088 podStartE2EDuration="3.234200088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:48:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:48:54.233553624 +0000 UTC m=+617.298330023" watchObservedRunningTime="2026-10-30 20:48:54.234200088 +0000 UTC m=+617.298976486"
Oct 30 20:48:54.407776 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:54.407690    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-7jfsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 30 20:48:54.407776 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:54.407747    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7jfsh" podUID="2f321ee6-f63d-43b5-a1e9-983985a789ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 30 20:48:54.506436 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:48:54.506381    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 20:48:54.663874 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:54.662966    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-sjpxz"
Oct 30 20:48:54.668816 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:54.665959    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-sjpxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Oct 30 20:48:54.668816 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:54.666039    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-sjpxz" podUID="f543e9a0-a5a6-417a-8f00-b0627a5c1776" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Oct 30 20:48:55.408517 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:55.408485    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-7jfsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 30 20:48:55.408953 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:55.408548    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7jfsh" podUID="2f321ee6-f63d-43b5-a1e9-983985a789ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 30 20:48:55.662308 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:55.662206    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-sjpxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Oct 30 20:48:55.662308 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:55.662274    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-sjpxz" podUID="f543e9a0-a5a6-417a-8f00-b0627a5c1776" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Oct 30 20:48:56.408459 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:56.408427    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-7jfsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 30 20:48:56.408648 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:56.408490    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7jfsh" podUID="2f321ee6-f63d-43b5-a1e9-983985a789ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 30 20:48:56.662873 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:56.662233    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-sjpxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Oct 30 20:48:56.662873 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:56.662299    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-sjpxz" podUID="f543e9a0-a5a6-417a-8f00-b0627a5c1776" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Oct 30 20:48:57.408355 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:57.408319    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-7jfsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 30 20:48:57.408620 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:57.408369    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7jfsh" podUID="2f321ee6-f63d-43b5-a1e9-983985a789ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 30 20:48:57.664868 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:57.664246    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-sjpxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Oct 30 20:48:57.664868 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:57.664313    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-sjpxz" podUID="f543e9a0-a5a6-417a-8f00-b0627a5c1776" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Oct 30 20:48:58.408002 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:58.407963    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-7jfsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 30 20:48:58.408199 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:58.408035    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7jfsh" podUID="2f321ee6-f63d-43b5-a1e9-983985a789ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 30 20:48:58.659871 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:58.659201    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-sjpxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Oct 30 20:48:58.659871 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:58.659298    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-sjpxz" podUID="f543e9a0-a5a6-417a-8f00-b0627a5c1776" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Oct 30 20:48:59.408493 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:59.408442    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-7jfsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 30 20:48:59.409808 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:59.409739    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7jfsh" podUID="2f321ee6-f63d-43b5-a1e9-983985a789ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 30 20:48:59.659355 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:59.659271    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-sjpxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Oct 30 20:48:59.659355 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:48:59.659340    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-sjpxz" podUID="f543e9a0-a5a6-417a-8f00-b0627a5c1776" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Oct 30 20:49:00.408650 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:00.408609    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-7jfsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 30 20:49:00.409071 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:00.408719    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7jfsh" podUID="2f321ee6-f63d-43b5-a1e9-983985a789ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 30 20:49:00.658714 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:00.658628    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-sjpxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Oct 30 20:49:00.658714 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:00.658698    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-sjpxz" podUID="f543e9a0-a5a6-417a-8f00-b0627a5c1776" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Oct 30 20:49:01.411726 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:01.411694    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7jfsh"
Oct 30 20:49:01.412158 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:01.412133    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7jfsh"
Oct 30 20:49:01.412921 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:01.412900    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7jfsh"
Oct 30 20:49:01.663036 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:01.662967    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-sjpxz"
Oct 30 20:49:01.663269 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:01.663250    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-sjpxz"
Oct 30 20:49:01.663864 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:01.663844    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-sjpxz"
Oct 30 20:49:24.458106 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:49:24.458068    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 20:49:54.458046 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:49:54.458005    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 20:49:56.644553 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:56.644526    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs"]
Oct 30 20:49:56.648198 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:56.648180    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs"
Oct 30 20:49:56.702502 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:56.702479    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-8fa2a7-openshift-default-dockercfg-gf577\""
Oct 30 20:49:56.752229 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:56.752197    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs"]
Oct 30 20:49:56.815472 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:56.815438    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7907175f-e439-48d7-8754-04591bde77cd-credential-socket\") pod \"e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs\" (UID: \"7907175f-e439-48d7-8754-04591bde77cd\") " pod="openshift-ingress/e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs"
Oct 30 20:49:56.815642 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:56.815562    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7907175f-e439-48d7-8754-04591bde77cd-istiod-ca-cert\") pod \"e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs\" (UID: \"7907175f-e439-48d7-8754-04591bde77cd\") " pod="openshift-ingress/e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs"
Oct 30 20:49:56.815642 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:56.815607    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7907175f-e439-48d7-8754-04591bde77cd-istio-data\") pod \"e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs\" (UID: \"7907175f-e439-48d7-8754-04591bde77cd\") " pod="openshift-ingress/e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs"
Oct 30 20:49:56.815642 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:56.815641    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7907175f-e439-48d7-8754-04591bde77cd-workload-certs\") pod \"e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs\" (UID: \"7907175f-e439-48d7-8754-04591bde77cd\") " pod="openshift-ingress/e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs"
Oct 30 20:49:56.815743 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:56.815668    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7907175f-e439-48d7-8754-04591bde77cd-istio-token\") pod \"e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs\" (UID: \"7907175f-e439-48d7-8754-04591bde77cd\") " pod="openshift-ingress/e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs"
Oct 30 20:49:56.815743 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:56.815713    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7907175f-e439-48d7-8754-04591bde77cd-workload-socket\") pod \"e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs\" (UID: \"7907175f-e439-48d7-8754-04591bde77cd\") " pod="openshift-ingress/e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs"
Oct 30 20:49:56.815822 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:56.815741    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2st6r\" (UniqueName: \"kubernetes.io/projected/7907175f-e439-48d7-8754-04591bde77cd-kube-api-access-2st6r\") pod \"e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs\" (UID: \"7907175f-e439-48d7-8754-04591bde77cd\") " pod="openshift-ingress/e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs"
Oct 30 20:49:56.815822 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:56.815813    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7907175f-e439-48d7-8754-04591bde77cd-istio-envoy\") pod \"e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs\" (UID: \"7907175f-e439-48d7-8754-04591bde77cd\") " pod="openshift-ingress/e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs"
Oct 30 20:49:56.815885 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:56.815835    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7907175f-e439-48d7-8754-04591bde77cd-istio-podinfo\") pod \"e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs\" (UID: \"7907175f-e439-48d7-8754-04591bde77cd\") " pod="openshift-ingress/e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs"
Oct 30 20:49:56.916845 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:56.916755    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7907175f-e439-48d7-8754-04591bde77cd-istiod-ca-cert\") pod \"e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs\" (UID: \"7907175f-e439-48d7-8754-04591bde77cd\") " pod="openshift-ingress/e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs"
Oct 30 20:49:56.916845 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:56.916823    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7907175f-e439-48d7-8754-04591bde77cd-istio-data\") pod \"e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs\" (UID: \"7907175f-e439-48d7-8754-04591bde77cd\") " pod="openshift-ingress/e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs"
Oct 30 20:49:56.917067 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:56.916853    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7907175f-e439-48d7-8754-04591bde77cd-workload-certs\") pod \"e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs\" (UID: \"7907175f-e439-48d7-8754-04591bde77cd\") " pod="openshift-ingress/e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs"
Oct 30 20:49:56.917067 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:56.916877    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7907175f-e439-48d7-8754-04591bde77cd-istio-token\") pod \"e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs\" (UID: \"7907175f-e439-48d7-8754-04591bde77cd\") " pod="openshift-ingress/e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs"
Oct 30 20:49:56.917067 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:56.916897    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7907175f-e439-48d7-8754-04591bde77cd-workload-socket\") pod \"e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs\" (UID: \"7907175f-e439-48d7-8754-04591bde77cd\") " pod="openshift-ingress/e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs"
Oct 30 20:49:56.917067 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:56.916923    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2st6r\" (UniqueName: \"kubernetes.io/projected/7907175f-e439-48d7-8754-04591bde77cd-kube-api-access-2st6r\") pod \"e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs\" (UID: \"7907175f-e439-48d7-8754-04591bde77cd\") " pod="openshift-ingress/e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs"
Oct 30 20:49:56.917067 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:56.916952    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7907175f-e439-48d7-8754-04591bde77cd-istio-envoy\") pod \"e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs\" (UID: \"7907175f-e439-48d7-8754-04591bde77cd\") " pod="openshift-ingress/e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs"
Oct 30 20:49:56.917067 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:56.916974    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7907175f-e439-48d7-8754-04591bde77cd-istio-podinfo\") pod \"e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs\" (UID: \"7907175f-e439-48d7-8754-04591bde77cd\") " pod="openshift-ingress/e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs"
Oct 30 20:49:56.917067 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:56.916997    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7907175f-e439-48d7-8754-04591bde77cd-credential-socket\") pod \"e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs\" (UID: \"7907175f-e439-48d7-8754-04591bde77cd\") " pod="openshift-ingress/e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs"
Oct 30 20:49:56.917386 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:56.917290    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7907175f-e439-48d7-8754-04591bde77cd-workload-certs\") pod \"e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs\" (UID: \"7907175f-e439-48d7-8754-04591bde77cd\") " pod="openshift-ingress/e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs"
Oct 30 20:49:56.917386 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:56.917353    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7907175f-e439-48d7-8754-04591bde77cd-workload-socket\") pod \"e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs\" (UID: \"7907175f-e439-48d7-8754-04591bde77cd\") " pod="openshift-ingress/e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs"
Oct 30 20:49:56.917517 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:56.917498    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7907175f-e439-48d7-8754-04591bde77cd-istio-data\") pod \"e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs\" (UID: \"7907175f-e439-48d7-8754-04591bde77cd\") " pod="openshift-ingress/e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs"
Oct 30 20:49:56.917554 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:56.917506    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7907175f-e439-48d7-8754-04591bde77cd-credential-socket\") pod \"e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs\" (UID: \"7907175f-e439-48d7-8754-04591bde77cd\") " pod="openshift-ingress/e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs"
Oct 30 20:49:56.917744 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:56.917723    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7907175f-e439-48d7-8754-04591bde77cd-istiod-ca-cert\") pod \"e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs\" (UID: \"7907175f-e439-48d7-8754-04591bde77cd\") " pod="openshift-ingress/e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs"
Oct 30 20:49:56.919432 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:56.919409    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7907175f-e439-48d7-8754-04591bde77cd-istio-envoy\") pod \"e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs\" (UID: \"7907175f-e439-48d7-8754-04591bde77cd\") " pod="openshift-ingress/e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs"
Oct 30 20:49:56.919665 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:56.919648    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7907175f-e439-48d7-8754-04591bde77cd-istio-podinfo\") pod \"e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs\" (UID: \"7907175f-e439-48d7-8754-04591bde77cd\") " pod="openshift-ingress/e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs"
Oct 30 20:49:56.982209 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:56.982177    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7907175f-e439-48d7-8754-04591bde77cd-istio-token\") pod \"e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs\" (UID: \"7907175f-e439-48d7-8754-04591bde77cd\") " pod="openshift-ingress/e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs"
Oct 30 20:49:56.991650 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:56.991622    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2st6r\" (UniqueName: \"kubernetes.io/projected/7907175f-e439-48d7-8754-04591bde77cd-kube-api-access-2st6r\") pod \"e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs\" (UID: \"7907175f-e439-48d7-8754-04591bde77cd\") " pod="openshift-ingress/e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs"
Oct 30 20:49:57.260610 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:57.260572    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs"
Oct 30 20:49:57.341298 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:57.340998    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq"]
Oct 30 20:49:57.345262 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:57.344986    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq"
Oct 30 20:49:57.399017 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:57.398979    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-2b425d-openshift-default-dockercfg-mnjbj\""
Oct 30 20:49:57.407233 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:57.407196    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq"]
Oct 30 20:49:57.422593 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:57.422561    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63ace536-4242-41d2-b1a0-3b10efad3333-istio-podinfo\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq\" (UID: \"63ace536-4242-41d2-b1a0-3b10efad3333\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq"
Oct 30 20:49:57.422593 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:57.422597    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2nwl\" (UniqueName: \"kubernetes.io/projected/63ace536-4242-41d2-b1a0-3b10efad3333-kube-api-access-f2nwl\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq\" (UID: \"63ace536-4242-41d2-b1a0-3b10efad3333\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq"
Oct 30 20:49:57.422834 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:57.422625    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63ace536-4242-41d2-b1a0-3b10efad3333-workload-certs\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq\" (UID: \"63ace536-4242-41d2-b1a0-3b10efad3333\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq"
Oct 30 20:49:57.422902 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:57.422831    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63ace536-4242-41d2-b1a0-3b10efad3333-istio-token\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq\" (UID: \"63ace536-4242-41d2-b1a0-3b10efad3333\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq"
Oct 30 20:49:57.422902 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:57.422890    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63ace536-4242-41d2-b1a0-3b10efad3333-istiod-ca-cert\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq\" (UID: \"63ace536-4242-41d2-b1a0-3b10efad3333\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq"
Oct 30 20:49:57.423016 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:57.422970    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63ace536-4242-41d2-b1a0-3b10efad3333-istio-envoy\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq\" (UID: \"63ace536-4242-41d2-b1a0-3b10efad3333\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq"
Oct 30 20:49:57.423016 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:57.422992    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63ace536-4242-41d2-b1a0-3b10efad3333-istio-data\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq\" (UID: \"63ace536-4242-41d2-b1a0-3b10efad3333\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq"
Oct 30 20:49:57.423146 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:57.423018    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63ace536-4242-41d2-b1a0-3b10efad3333-credential-socket\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq\" (UID: \"63ace536-4242-41d2-b1a0-3b10efad3333\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq"
Oct 30 20:49:57.423146 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:57.423037    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63ace536-4242-41d2-b1a0-3b10efad3333-workload-socket\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq\" (UID: \"63ace536-4242-41d2-b1a0-3b10efad3333\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq"
Oct 30 20:49:57.524134 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:57.524058    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63ace536-4242-41d2-b1a0-3b10efad3333-workload-certs\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq\" (UID: \"63ace536-4242-41d2-b1a0-3b10efad3333\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq"
Oct 30 20:49:57.524273 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:57.524146    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63ace536-4242-41d2-b1a0-3b10efad3333-istio-token\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq\" (UID: \"63ace536-4242-41d2-b1a0-3b10efad3333\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq"
Oct 30 20:49:57.524273 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:57.524193    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63ace536-4242-41d2-b1a0-3b10efad3333-istiod-ca-cert\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq\" (UID: \"63ace536-4242-41d2-b1a0-3b10efad3333\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq"
Oct 30 20:49:57.524273 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:57.524229    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63ace536-4242-41d2-b1a0-3b10efad3333-istio-envoy\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq\" (UID: \"63ace536-4242-41d2-b1a0-3b10efad3333\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq"
Oct 30 20:49:57.524273 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:57.524252    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63ace536-4242-41d2-b1a0-3b10efad3333-istio-data\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq\" (UID: \"63ace536-4242-41d2-b1a0-3b10efad3333\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq"
Oct 30 20:49:57.524552 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:57.524282    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63ace536-4242-41d2-b1a0-3b10efad3333-credential-socket\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq\" (UID: \"63ace536-4242-41d2-b1a0-3b10efad3333\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq"
Oct 30 20:49:57.524552 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:57.524312    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63ace536-4242-41d2-b1a0-3b10efad3333-workload-socket\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq\" (UID: \"63ace536-4242-41d2-b1a0-3b10efad3333\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq"
Oct 30 20:49:57.524552 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:57.524389    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63ace536-4242-41d2-b1a0-3b10efad3333-istio-podinfo\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq\" (UID: \"63ace536-4242-41d2-b1a0-3b10efad3333\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq"
Oct 30 20:49:57.524552 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:57.524419    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2nwl\" (UniqueName: \"kubernetes.io/projected/63ace536-4242-41d2-b1a0-3b10efad3333-kube-api-access-f2nwl\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq\" (UID: \"63ace536-4242-41d2-b1a0-3b10efad3333\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq"
Oct 30 20:49:57.524755 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:57.524567    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63ace536-4242-41d2-b1a0-3b10efad3333-workload-certs\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq\" (UID: \"63ace536-4242-41d2-b1a0-3b10efad3333\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq"
Oct 30 20:49:57.524755 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:57.524606    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63ace536-4242-41d2-b1a0-3b10efad3333-credential-socket\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq\" (UID: \"63ace536-4242-41d2-b1a0-3b10efad3333\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq"
Oct 30 20:49:57.524755 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:57.524695    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63ace536-4242-41d2-b1a0-3b10efad3333-istio-data\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq\" (UID: \"63ace536-4242-41d2-b1a0-3b10efad3333\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq"
Oct 30 20:49:57.524900 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:57.524775    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63ace536-4242-41d2-b1a0-3b10efad3333-workload-socket\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq\" (UID: \"63ace536-4242-41d2-b1a0-3b10efad3333\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq"
Oct 30 20:49:57.525222 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:57.525196    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63ace536-4242-41d2-b1a0-3b10efad3333-istiod-ca-cert\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq\" (UID: \"63ace536-4242-41d2-b1a0-3b10efad3333\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq"
Oct 30 20:49:57.527504 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:57.527481    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63ace536-4242-41d2-b1a0-3b10efad3333-istio-envoy\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq\" (UID: \"63ace536-4242-41d2-b1a0-3b10efad3333\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq"
Oct 30 20:49:57.527593 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:57.527576    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63ace536-4242-41d2-b1a0-3b10efad3333-istio-podinfo\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq\" (UID: \"63ace536-4242-41d2-b1a0-3b10efad3333\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq"
Oct 30 20:49:57.587506 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:57.587470    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63ace536-4242-41d2-b1a0-3b10efad3333-istio-token\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq\" (UID: \"63ace536-4242-41d2-b1a0-3b10efad3333\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq"
Oct 30 20:49:57.588528 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:57.588505    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2nwl\" (UniqueName: \"kubernetes.io/projected/63ace536-4242-41d2-b1a0-3b10efad3333-kube-api-access-f2nwl\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq\" (UID: \"63ace536-4242-41d2-b1a0-3b10efad3333\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq"
Oct 30 20:49:57.660999 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:57.660962    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq"
Oct 30 20:49:57.716429 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:57.715301    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs"]
Oct 30 20:49:57.720908 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:49:57.720872    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7907175f_e439_48d7_8754_04591bde77cd.slice/crio-daa0282ee07ef17c6bbf349ea0f28f1a29e691536b5a97556530366d50b648bb WatchSource:0}: Error finding container daa0282ee07ef17c6bbf349ea0f28f1a29e691536b5a97556530366d50b648bb: Status 404 returned error can't find the container with id daa0282ee07ef17c6bbf349ea0f28f1a29e691536b5a97556530366d50b648bb
Oct 30 20:49:57.726471 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:57.725694    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 20:49:57.726471 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:57.725895    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 20:49:57.726471 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:57.725955    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 20:49:58.051408 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:58.051372    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq"]
Oct 30 20:49:58.053561 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:49:58.053527    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63ace536_4242_41d2_b1a0_3b10efad3333.slice/crio-f9a194e253aab9210f3266c1d97a16be2231fec122d0f0ed3b2cc69813c31e38 WatchSource:0}: Error finding container f9a194e253aab9210f3266c1d97a16be2231fec122d0f0ed3b2cc69813c31e38: Status 404 returned error can't find the container with id f9a194e253aab9210f3266c1d97a16be2231fec122d0f0ed3b2cc69813c31e38
Oct 30 20:49:58.056308 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:58.056280    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 20:49:58.056411 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:58.056340    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 20:49:58.056411 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:58.056372    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 20:49:58.387570 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:58.387482    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm"]
Oct 30 20:49:58.391188 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:58.391160    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm"
Oct 30 20:49:58.444463 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:58.444428    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-d061d0-openshift-default-dockercfg-cwlxq\""
Oct 30 20:49:58.447602 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:58.447572    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq" event={"ID":"63ace536-4242-41d2-b1a0-3b10efad3333","Type":"ContainerStarted","Data":"b1dd9a119d562d2f3f9f7ca2671288b163d12504b86500f6e47a34b420812bdf"}
Oct 30 20:49:58.447746 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:58.447627    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq" event={"ID":"63ace536-4242-41d2-b1a0-3b10efad3333","Type":"ContainerStarted","Data":"f9a194e253aab9210f3266c1d97a16be2231fec122d0f0ed3b2cc69813c31e38"}
Oct 30 20:49:58.449524 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:58.449476    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs" event={"ID":"7907175f-e439-48d7-8754-04591bde77cd","Type":"ContainerStarted","Data":"819041f4dd6f4abf2ee3ffddd4269f5574b61ea26270d5292d338e4bbf72ce51"}
Oct 30 20:49:58.449626 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:58.449528    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs" event={"ID":"7907175f-e439-48d7-8754-04591bde77cd","Type":"ContainerStarted","Data":"daa0282ee07ef17c6bbf349ea0f28f1a29e691536b5a97556530366d50b648bb"}
Oct 30 20:49:58.462847 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:58.462812    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm"]
Oct 30 20:49:58.536931 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:58.535680    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4e643cf-8167-42e3-a104-03ac5f7bd133-istio-data\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm\" (UID: \"e4e643cf-8167-42e3-a104-03ac5f7bd133\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm"
Oct 30 20:49:58.536931 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:58.535889    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e643cf-8167-42e3-a104-03ac5f7bd133-credential-socket\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm\" (UID: \"e4e643cf-8167-42e3-a104-03ac5f7bd133\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm"
Oct 30 20:49:58.536931 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:58.535982    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4e643cf-8167-42e3-a104-03ac5f7bd133-istio-envoy\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm\" (UID: \"e4e643cf-8167-42e3-a104-03ac5f7bd133\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm"
Oct 30 20:49:58.536931 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:58.536095    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vsq9\" (UniqueName: \"kubernetes.io/projected/e4e643cf-8167-42e3-a104-03ac5f7bd133-kube-api-access-8vsq9\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm\" (UID: \"e4e643cf-8167-42e3-a104-03ac5f7bd133\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm"
Oct 30 20:49:58.536931 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:58.536160    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4e643cf-8167-42e3-a104-03ac5f7bd133-istio-token\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm\" (UID: \"e4e643cf-8167-42e3-a104-03ac5f7bd133\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm"
Oct 30 20:49:58.536931 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:58.536184    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4e643cf-8167-42e3-a104-03ac5f7bd133-workload-certs\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm\" (UID: \"e4e643cf-8167-42e3-a104-03ac5f7bd133\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm"
Oct 30 20:49:58.536931 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:58.536345    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e643cf-8167-42e3-a104-03ac5f7bd133-workload-socket\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm\" (UID: \"e4e643cf-8167-42e3-a104-03ac5f7bd133\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm"
Oct 30 20:49:58.536931 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:58.536476    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4e643cf-8167-42e3-a104-03ac5f7bd133-istiod-ca-cert\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm\" (UID: \"e4e643cf-8167-42e3-a104-03ac5f7bd133\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm"
Oct 30 20:49:58.536931 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:58.536517    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4e643cf-8167-42e3-a104-03ac5f7bd133-istio-podinfo\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm\" (UID: \"e4e643cf-8167-42e3-a104-03ac5f7bd133\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm"
Oct 30 20:49:58.633326 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:58.633270    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq" podStartSLOduration=1.633254092 podStartE2EDuration="1.633254092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:49:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:49:58.630562596 +0000 UTC m=+681.695338995" watchObservedRunningTime="2026-10-30 20:49:58.633254092 +0000 UTC m=+681.698030489"
Oct 30 20:49:58.637708 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:58.637654    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4e643cf-8167-42e3-a104-03ac5f7bd133-istio-envoy\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm\" (UID: \"e4e643cf-8167-42e3-a104-03ac5f7bd133\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm"
Oct 30 20:49:58.637708 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:58.637699    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vsq9\" (UniqueName: \"kubernetes.io/projected/e4e643cf-8167-42e3-a104-03ac5f7bd133-kube-api-access-8vsq9\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm\" (UID: \"e4e643cf-8167-42e3-a104-03ac5f7bd133\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm"
Oct 30 20:49:58.637855 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:58.637728    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4e643cf-8167-42e3-a104-03ac5f7bd133-istio-token\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm\" (UID: \"e4e643cf-8167-42e3-a104-03ac5f7bd133\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm"
Oct 30 20:49:58.637855 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:58.637746    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4e643cf-8167-42e3-a104-03ac5f7bd133-workload-certs\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm\" (UID: \"e4e643cf-8167-42e3-a104-03ac5f7bd133\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm"
Oct 30 20:49:58.637855 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:58.637772    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e643cf-8167-42e3-a104-03ac5f7bd133-workload-socket\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm\" (UID: \"e4e643cf-8167-42e3-a104-03ac5f7bd133\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm"
Oct 30 20:49:58.637855 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:58.637844    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4e643cf-8167-42e3-a104-03ac5f7bd133-istiod-ca-cert\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm\" (UID: \"e4e643cf-8167-42e3-a104-03ac5f7bd133\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm"
Oct 30 20:49:58.638009 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:58.637876    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4e643cf-8167-42e3-a104-03ac5f7bd133-istio-podinfo\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm\" (UID: \"e4e643cf-8167-42e3-a104-03ac5f7bd133\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm"
Oct 30 20:49:58.638009 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:58.637937    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4e643cf-8167-42e3-a104-03ac5f7bd133-istio-data\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm\" (UID: \"e4e643cf-8167-42e3-a104-03ac5f7bd133\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm"
Oct 30 20:49:58.638009 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:58.637984    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e643cf-8167-42e3-a104-03ac5f7bd133-credential-socket\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm\" (UID: \"e4e643cf-8167-42e3-a104-03ac5f7bd133\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm"
Oct 30 20:49:58.638262 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:58.638242    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e643cf-8167-42e3-a104-03ac5f7bd133-workload-socket\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm\" (UID: \"e4e643cf-8167-42e3-a104-03ac5f7bd133\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm"
Oct 30 20:49:58.638433 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:58.638332    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4e643cf-8167-42e3-a104-03ac5f7bd133-workload-certs\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm\" (UID: \"e4e643cf-8167-42e3-a104-03ac5f7bd133\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm"
Oct 30 20:49:58.638433 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:58.638352    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e643cf-8167-42e3-a104-03ac5f7bd133-credential-socket\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm\" (UID: \"e4e643cf-8167-42e3-a104-03ac5f7bd133\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm"
Oct 30 20:49:58.638558 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:58.638504    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4e643cf-8167-42e3-a104-03ac5f7bd133-istio-data\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm\" (UID: \"e4e643cf-8167-42e3-a104-03ac5f7bd133\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm"
Oct 30 20:49:58.638970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:58.638944    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4e643cf-8167-42e3-a104-03ac5f7bd133-istiod-ca-cert\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm\" (UID: \"e4e643cf-8167-42e3-a104-03ac5f7bd133\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm"
Oct 30 20:49:58.640276 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:58.640255    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4e643cf-8167-42e3-a104-03ac5f7bd133-istio-envoy\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm\" (UID: \"e4e643cf-8167-42e3-a104-03ac5f7bd133\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm"
Oct 30 20:49:58.640492 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:58.640474    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4e643cf-8167-42e3-a104-03ac5f7bd133-istio-podinfo\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm\" (UID: \"e4e643cf-8167-42e3-a104-03ac5f7bd133\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm"
Oct 30 20:49:58.661456 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:58.661425    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq"
Oct 30 20:49:58.696086 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:58.696060    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vsq9\" (UniqueName: \"kubernetes.io/projected/e4e643cf-8167-42e3-a104-03ac5f7bd133-kube-api-access-8vsq9\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm\" (UID: \"e4e643cf-8167-42e3-a104-03ac5f7bd133\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm"
Oct 30 20:49:58.696233 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:58.696177    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4e643cf-8167-42e3-a104-03ac5f7bd133-istio-token\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm\" (UID: \"e4e643cf-8167-42e3-a104-03ac5f7bd133\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm"
Oct 30 20:49:58.708386 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:58.708363    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm"
Oct 30 20:49:58.799329 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:58.799277    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs" podStartSLOduration=2.799263061 podStartE2EDuration="2.799263061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:49:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:49:58.798128082 +0000 UTC m=+681.862904481" watchObservedRunningTime="2026-10-30 20:49:58.799263061 +0000 UTC m=+681.864039459"
Oct 30 20:49:59.127468 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:59.127432    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm"]
Oct 30 20:49:59.154941 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:49:59.154909    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4e643cf_8167_42e3_a104_03ac5f7bd133.slice/crio-9e3b401e1ac0a5bcc5e63972e1e222545a39ae2279c53af8fc2089e92f1b0a41 WatchSource:0}: Error finding container 9e3b401e1ac0a5bcc5e63972e1e222545a39ae2279c53af8fc2089e92f1b0a41: Status 404 returned error can't find the container with id 9e3b401e1ac0a5bcc5e63972e1e222545a39ae2279c53af8fc2089e92f1b0a41
Oct 30 20:49:59.157612 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:59.157578    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 20:49:59.157705 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:59.157669    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 20:49:59.157762 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:59.157716    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 20:49:59.260925 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:59.260880    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs"
Oct 30 20:49:59.266574 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:59.266549    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs"
Oct 30 20:49:59.461682 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:59.461647    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm" event={"ID":"e4e643cf-8167-42e3-a104-03ac5f7bd133","Type":"ContainerStarted","Data":"e885c7f726a81161a04012f02368d308828df70aa2ed6e28ab7870842505ce8a"}
Oct 30 20:49:59.461898 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:59.461707    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm" event={"ID":"e4e643cf-8167-42e3-a104-03ac5f7bd133","Type":"ContainerStarted","Data":"9e3b401e1ac0a5bcc5e63972e1e222545a39ae2279c53af8fc2089e92f1b0a41"}
Oct 30 20:49:59.462128 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:59.462107    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs"
Oct 30 20:49:59.462962 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:59.462945    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs"
Oct 30 20:49:59.597053 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:59.597000    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm" podStartSLOduration=1.596987567 podStartE2EDuration="1.596987567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:49:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:49:59.594062231 +0000 UTC m=+682.658838631" watchObservedRunningTime="2026-10-30 20:49:59.596987567 +0000 UTC m=+682.661763964"
Oct 30 20:49:59.666743 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:59.666713    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq"
Oct 30 20:49:59.708705 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:49:59.708670    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm"
Oct 30 20:50:00.467720 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:00.467694    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq"
Oct 30 20:50:00.468699 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:00.468681    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq"
Oct 30 20:50:00.714417 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:00.714386    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm"
Oct 30 20:50:01.473507 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:01.473469    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm"
Oct 30 20:50:01.474560 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:01.474540    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm"
Oct 30 20:50:10.678458 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:10.678422    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-682999-5489c955b7-dzmt9"]
Oct 30 20:50:10.683373 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:10.683356    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-682999-5489c955b7-dzmt9"
Oct 30 20:50:10.743062 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:10.743027    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-682999-5489c955b7-dzmt9"]
Oct 30 20:50:10.789934 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:10.789903    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-682999\""
Oct 30 20:50:10.790141 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:10.789955    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-682999-dockercfg-lphff\""
Oct 30 20:50:10.881313 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:10.881278    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ckjv\" (UniqueName: \"kubernetes.io/projected/230f6842-289d-43fd-b1b7-849c27c5f97b-kube-api-access-9ckjv\") pod \"payload-pre-processing-e2e-worker-w3-682999-5489c955b7-dzmt9\" (UID: \"230f6842-289d-43fd-b1b7-849c27c5f97b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-682999-5489c955b7-dzmt9"
Oct 30 20:50:10.881514 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:10.881447    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/230f6842-289d-43fd-b1b7-849c27c5f97b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-682999-5489c955b7-dzmt9\" (UID: \"230f6842-289d-43fd-b1b7-849c27c5f97b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-682999-5489c955b7-dzmt9"
Oct 30 20:50:10.982042 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:10.982013    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/230f6842-289d-43fd-b1b7-849c27c5f97b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-682999-5489c955b7-dzmt9\" (UID: \"230f6842-289d-43fd-b1b7-849c27c5f97b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-682999-5489c955b7-dzmt9"
Oct 30 20:50:10.982232 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:10.982066    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ckjv\" (UniqueName: \"kubernetes.io/projected/230f6842-289d-43fd-b1b7-849c27c5f97b-kube-api-access-9ckjv\") pod \"payload-pre-processing-e2e-worker-w3-682999-5489c955b7-dzmt9\" (UID: \"230f6842-289d-43fd-b1b7-849c27c5f97b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-682999-5489c955b7-dzmt9"
Oct 30 20:50:10.982671 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:10.982652    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/230f6842-289d-43fd-b1b7-849c27c5f97b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-682999-5489c955b7-dzmt9\" (UID: \"230f6842-289d-43fd-b1b7-849c27c5f97b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-682999-5489c955b7-dzmt9"
Oct 30 20:50:11.043541 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:11.043514    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ckjv\" (UniqueName: \"kubernetes.io/projected/230f6842-289d-43fd-b1b7-849c27c5f97b-kube-api-access-9ckjv\") pod \"payload-pre-processing-e2e-worker-w3-682999-5489c955b7-dzmt9\" (UID: \"230f6842-289d-43fd-b1b7-849c27c5f97b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-682999-5489c955b7-dzmt9"
Oct 30 20:50:11.297910 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:11.297826    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-682999-5489c955b7-dzmt9"
Oct 30 20:50:11.673292 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:11.673269    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-682999-5489c955b7-dzmt9"]
Oct 30 20:50:11.674569 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:50:11.674542    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod230f6842_289d_43fd_b1b7_849c27c5f97b.slice/crio-a4c6f32d21873699db7942bab67d56310fac306376aa3f171828c65c46c3aabe WatchSource:0}: Error finding container a4c6f32d21873699db7942bab67d56310fac306376aa3f171828c65c46c3aabe: Status 404 returned error can't find the container with id a4c6f32d21873699db7942bab67d56310fac306376aa3f171828c65c46c3aabe
Oct 30 20:50:12.531685 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:12.531650    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-682999-5489c955b7-dzmt9" event={"ID":"230f6842-289d-43fd-b1b7-849c27c5f97b","Type":"ContainerStarted","Data":"a4c6f32d21873699db7942bab67d56310fac306376aa3f171828c65c46c3aabe"}
Oct 30 20:50:14.545117 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:14.545031    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-682999-5489c955b7-dzmt9" event={"ID":"230f6842-289d-43fd-b1b7-849c27c5f97b","Type":"ContainerStarted","Data":"3e751404069b40b372c7beb6b585a4373c732f633e03ae0dea87b9a88fe1a275"}
Oct 30 20:50:14.545117 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:14.545089    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-682999-5489c955b7-dzmt9"
Oct 30 20:50:14.675216 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:14.675162    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-682999-5489c955b7-dzmt9" podStartSLOduration=2.117724939 podStartE2EDuration="4.675148357s" totalImagesPullingTime="2.557423418s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:50:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:14.672423125 +0000 UTC m=+697.737199523" watchObservedRunningTime="2026-10-30 20:50:14.675148357 +0000 UTC m=+697.739924802"
Oct 30 20:50:18.204531 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:18.204446    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-781058-6d867c875-285pt"]
Oct 30 20:50:18.208133 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:18.208112    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-781058-6d867c875-285pt"
Oct 30 20:50:18.257437 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:18.257383    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e725a39a-4b17-4c3c-9166-e82091523ae5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-781058-6d867c875-285pt\" (UID: \"e725a39a-4b17-4c3c-9166-e82091523ae5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-781058-6d867c875-285pt"
Oct 30 20:50:18.257626 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:18.257605    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47vl5\" (UniqueName: \"kubernetes.io/projected/e725a39a-4b17-4c3c-9166-e82091523ae5-kube-api-access-47vl5\") pod \"payload-pre-processing-e2e-worker-w0-781058-6d867c875-285pt\" (UID: \"e725a39a-4b17-4c3c-9166-e82091523ae5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-781058-6d867c875-285pt"
Oct 30 20:50:18.314105 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:18.314071    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-781058\""
Oct 30 20:50:18.314321 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:18.314194    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-781058-dockercfg-84vjz\""
Oct 30 20:50:18.317306 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:18.317281    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-781058-6d867c875-285pt"]
Oct 30 20:50:18.358923 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:18.358883    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47vl5\" (UniqueName: \"kubernetes.io/projected/e725a39a-4b17-4c3c-9166-e82091523ae5-kube-api-access-47vl5\") pod \"payload-pre-processing-e2e-worker-w0-781058-6d867c875-285pt\" (UID: \"e725a39a-4b17-4c3c-9166-e82091523ae5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-781058-6d867c875-285pt"
Oct 30 20:50:18.359121 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:18.358943    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e725a39a-4b17-4c3c-9166-e82091523ae5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-781058-6d867c875-285pt\" (UID: \"e725a39a-4b17-4c3c-9166-e82091523ae5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-781058-6d867c875-285pt"
Oct 30 20:50:18.359673 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:18.359647    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e725a39a-4b17-4c3c-9166-e82091523ae5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-781058-6d867c875-285pt\" (UID: \"e725a39a-4b17-4c3c-9166-e82091523ae5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-781058-6d867c875-285pt"
Oct 30 20:50:18.416680 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:18.416652    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47vl5\" (UniqueName: \"kubernetes.io/projected/e725a39a-4b17-4c3c-9166-e82091523ae5-kube-api-access-47vl5\") pod \"payload-pre-processing-e2e-worker-w0-781058-6d867c875-285pt\" (UID: \"e725a39a-4b17-4c3c-9166-e82091523ae5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-781058-6d867c875-285pt"
Oct 30 20:50:18.521484 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:18.521390    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-781058-6d867c875-285pt"
Oct 30 20:50:18.901690 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:18.901663    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-781058-6d867c875-285pt"]
Oct 30 20:50:18.903366 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:50:18.903339    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode725a39a_4b17_4c3c_9166_e82091523ae5.slice/crio-3d25cd85d76da174d487463a3abab28bcd806c23b6f6b3df7f2ed8e9bcee791f WatchSource:0}: Error finding container 3d25cd85d76da174d487463a3abab28bcd806c23b6f6b3df7f2ed8e9bcee791f: Status 404 returned error can't find the container with id 3d25cd85d76da174d487463a3abab28bcd806c23b6f6b3df7f2ed8e9bcee791f
Oct 30 20:50:19.572458 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:19.572416    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-781058-6d867c875-285pt" event={"ID":"e725a39a-4b17-4c3c-9166-e82091523ae5","Type":"ContainerStarted","Data":"22165e39e06ba7f0d34899a19dee4963c66747c782ba94213f5f9b922f724d34"}
Oct 30 20:50:19.572909 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:19.572482    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-781058-6d867c875-285pt" event={"ID":"e725a39a-4b17-4c3c-9166-e82091523ae5","Type":"ContainerStarted","Data":"3d25cd85d76da174d487463a3abab28bcd806c23b6f6b3df7f2ed8e9bcee791f"}
Oct 30 20:50:19.572909 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:19.572523    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-781058-6d867c875-285pt"
Oct 30 20:50:19.688825 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:19.688756    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-781058-6d867c875-285pt" podStartSLOduration=1.688740417 podStartE2EDuration="1.688740417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:50:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:19.687935603 +0000 UTC m=+702.752712002" watchObservedRunningTime="2026-10-30 20:50:19.688740417 +0000 UTC m=+702.753516814"
Oct 30 20:50:20.813306 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:20.813266    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-8fa2a7-977ff4795-mgd5h"]
Oct 30 20:50:20.817011 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:20.816993    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-8fa2a7-977ff4795-mgd5h"
Oct 30 20:50:20.872299 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:20.872267    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-8fa2a7\""
Oct 30 20:50:20.886005 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:20.885965    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9h2j\" (UniqueName: \"kubernetes.io/projected/0fa5be36-c8de-4cdc-87d9-ebf11115f0c3-kube-api-access-h9h2j\") pod \"payload-pre-processing-e2e-worker-w6-8fa2a7-977ff4795-mgd5h\" (UID: \"0fa5be36-c8de-4cdc-87d9-ebf11115f0c3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-8fa2a7-977ff4795-mgd5h"
Oct 30 20:50:20.886191 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:20.886115    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fa5be36-c8de-4cdc-87d9-ebf11115f0c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-8fa2a7-977ff4795-mgd5h\" (UID: \"0fa5be36-c8de-4cdc-87d9-ebf11115f0c3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-8fa2a7-977ff4795-mgd5h"
Oct 30 20:50:20.922601 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:20.922568    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-8fa2a7-dockercfg-tfglb\""
Oct 30 20:50:20.936773 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:20.926545    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-8fa2a7-977ff4795-mgd5h"]
Oct 30 20:50:20.987167 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:20.987131    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9h2j\" (UniqueName: \"kubernetes.io/projected/0fa5be36-c8de-4cdc-87d9-ebf11115f0c3-kube-api-access-h9h2j\") pod \"payload-pre-processing-e2e-worker-w6-8fa2a7-977ff4795-mgd5h\" (UID: \"0fa5be36-c8de-4cdc-87d9-ebf11115f0c3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-8fa2a7-977ff4795-mgd5h"
Oct 30 20:50:20.987358 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:20.987235    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fa5be36-c8de-4cdc-87d9-ebf11115f0c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-8fa2a7-977ff4795-mgd5h\" (UID: \"0fa5be36-c8de-4cdc-87d9-ebf11115f0c3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-8fa2a7-977ff4795-mgd5h"
Oct 30 20:50:20.987927 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:20.987909    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fa5be36-c8de-4cdc-87d9-ebf11115f0c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-8fa2a7-977ff4795-mgd5h\" (UID: \"0fa5be36-c8de-4cdc-87d9-ebf11115f0c3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-8fa2a7-977ff4795-mgd5h"
Oct 30 20:50:21.045305 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:21.045276    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9h2j\" (UniqueName: \"kubernetes.io/projected/0fa5be36-c8de-4cdc-87d9-ebf11115f0c3-kube-api-access-h9h2j\") pod \"payload-pre-processing-e2e-worker-w6-8fa2a7-977ff4795-mgd5h\" (UID: \"0fa5be36-c8de-4cdc-87d9-ebf11115f0c3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-8fa2a7-977ff4795-mgd5h"
Oct 30 20:50:21.130452 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:21.130359    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-8fa2a7-977ff4795-mgd5h"
Oct 30 20:50:21.505132 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:21.505103    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-8fa2a7-977ff4795-mgd5h"]
Oct 30 20:50:21.507317 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:50:21.507275    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fa5be36_c8de_4cdc_87d9_ebf11115f0c3.slice/crio-c64919b63d4c86ae9639fc914d7004712633e653bd0aa3caf70cec217a181ca2 WatchSource:0}: Error finding container c64919b63d4c86ae9639fc914d7004712633e653bd0aa3caf70cec217a181ca2: Status 404 returned error can't find the container with id c64919b63d4c86ae9639fc914d7004712633e653bd0aa3caf70cec217a181ca2
Oct 30 20:50:21.584851 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:21.584821    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-8fa2a7-977ff4795-mgd5h" event={"ID":"0fa5be36-c8de-4cdc-87d9-ebf11115f0c3","Type":"ContainerStarted","Data":"c8860b676c5420e7791c6545f94fc26e9a5ebcd3b4cad569dcfc5d92bd9b6d4d"}
Oct 30 20:50:21.584985 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:21.584878    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-8fa2a7-977ff4795-mgd5h" event={"ID":"0fa5be36-c8de-4cdc-87d9-ebf11115f0c3","Type":"ContainerStarted","Data":"c64919b63d4c86ae9639fc914d7004712633e653bd0aa3caf70cec217a181ca2"}
Oct 30 20:50:21.584985 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:21.584950    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-8fa2a7-977ff4795-mgd5h"
Oct 30 20:50:21.724509 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:21.724455    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-8fa2a7-977ff4795-mgd5h" podStartSLOduration=1.724439123 podStartE2EDuration="1.724439123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:50:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:21.722209313 +0000 UTC m=+704.786985711" watchObservedRunningTime="2026-10-30 20:50:21.724439123 +0000 UTC m=+704.789215521"
Oct 30 20:50:24.482849 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:50:24.482816    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 20:50:25.557462 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:25.556447    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-682999-5489c955b7-dzmt9"
Oct 30 20:50:28.177081 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:28.177051    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-1fdaa7-bb8847cd7-kq2ml"]
Oct 30 20:50:28.183461 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:28.183437    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1fdaa7-bb8847cd7-kq2ml"
Oct 30 20:50:28.265815 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:28.262966    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-1fdaa7-bb8847cd7-kq2ml"]
Oct 30 20:50:28.284600 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:28.284554    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70aef1ff-8862-40b8-b3b7-c8878df1619f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-1fdaa7-bb8847cd7-kq2ml\" (UID: \"70aef1ff-8862-40b8-b3b7-c8878df1619f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1fdaa7-bb8847cd7-kq2ml"
Oct 30 20:50:28.285016 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:28.284993    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph4xw\" (UniqueName: \"kubernetes.io/projected/70aef1ff-8862-40b8-b3b7-c8878df1619f-kube-api-access-ph4xw\") pod \"payload-pre-processing-e2e-worker-w4-1fdaa7-bb8847cd7-kq2ml\" (UID: \"70aef1ff-8862-40b8-b3b7-c8878df1619f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1fdaa7-bb8847cd7-kq2ml"
Oct 30 20:50:28.298569 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:28.298543    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-1fdaa7\""
Oct 30 20:50:28.298834 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:28.298632    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-1fdaa7-dockercfg-kr5c7\""
Oct 30 20:50:28.358909 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:28.358873    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-1fdaa7-6fcb4879fd-bcz9f"]
Oct 30 20:50:28.363020 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:28.362997    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-1fdaa7-6fcb4879fd-bcz9f"
Oct 30 20:50:28.386196 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:28.386169    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph4xw\" (UniqueName: \"kubernetes.io/projected/70aef1ff-8862-40b8-b3b7-c8878df1619f-kube-api-access-ph4xw\") pod \"payload-pre-processing-e2e-worker-w4-1fdaa7-bb8847cd7-kq2ml\" (UID: \"70aef1ff-8862-40b8-b3b7-c8878df1619f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1fdaa7-bb8847cd7-kq2ml"
Oct 30 20:50:28.386325 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:28.386308    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70aef1ff-8862-40b8-b3b7-c8878df1619f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-1fdaa7-bb8847cd7-kq2ml\" (UID: \"70aef1ff-8862-40b8-b3b7-c8878df1619f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1fdaa7-bb8847cd7-kq2ml"
Oct 30 20:50:28.386973 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:28.386956    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70aef1ff-8862-40b8-b3b7-c8878df1619f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-1fdaa7-bb8847cd7-kq2ml\" (UID: \"70aef1ff-8862-40b8-b3b7-c8878df1619f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1fdaa7-bb8847cd7-kq2ml"
Oct 30 20:50:28.401856 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:28.401831    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-1fdaa7-6fcb4879fd-bcz9f"]
Oct 30 20:50:28.461268 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:28.461239    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph4xw\" (UniqueName: \"kubernetes.io/projected/70aef1ff-8862-40b8-b3b7-c8878df1619f-kube-api-access-ph4xw\") pod \"payload-pre-processing-e2e-worker-w4-1fdaa7-bb8847cd7-kq2ml\" (UID: \"70aef1ff-8862-40b8-b3b7-c8878df1619f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1fdaa7-bb8847cd7-kq2ml"
Oct 30 20:50:28.487641 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:28.487609    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msdps\" (UniqueName: \"kubernetes.io/projected/5304445a-544a-4cb7-b79a-53502f94fe48-kube-api-access-msdps\") pod \"payload-processing-e2e-worker-w4-1fdaa7-6fcb4879fd-bcz9f\" (UID: \"5304445a-544a-4cb7-b79a-53502f94fe48\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-1fdaa7-6fcb4879fd-bcz9f"
Oct 30 20:50:28.487813 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:28.487666    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5304445a-544a-4cb7-b79a-53502f94fe48-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-1fdaa7-6fcb4879fd-bcz9f\" (UID: \"5304445a-544a-4cb7-b79a-53502f94fe48\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-1fdaa7-6fcb4879fd-bcz9f"
Oct 30 20:50:28.497309 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:28.497284    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1fdaa7-bb8847cd7-kq2ml"
Oct 30 20:50:28.588737 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:28.588671    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msdps\" (UniqueName: \"kubernetes.io/projected/5304445a-544a-4cb7-b79a-53502f94fe48-kube-api-access-msdps\") pod \"payload-processing-e2e-worker-w4-1fdaa7-6fcb4879fd-bcz9f\" (UID: \"5304445a-544a-4cb7-b79a-53502f94fe48\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-1fdaa7-6fcb4879fd-bcz9f"
Oct 30 20:50:28.588925 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:28.588740    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5304445a-544a-4cb7-b79a-53502f94fe48-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-1fdaa7-6fcb4879fd-bcz9f\" (UID: \"5304445a-544a-4cb7-b79a-53502f94fe48\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-1fdaa7-6fcb4879fd-bcz9f"
Oct 30 20:50:28.589760 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:28.589735    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5304445a-544a-4cb7-b79a-53502f94fe48-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-1fdaa7-6fcb4879fd-bcz9f\" (UID: \"5304445a-544a-4cb7-b79a-53502f94fe48\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-1fdaa7-6fcb4879fd-bcz9f"
Oct 30 20:50:28.661644 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:28.661609    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msdps\" (UniqueName: \"kubernetes.io/projected/5304445a-544a-4cb7-b79a-53502f94fe48-kube-api-access-msdps\") pod \"payload-processing-e2e-worker-w4-1fdaa7-6fcb4879fd-bcz9f\" (UID: \"5304445a-544a-4cb7-b79a-53502f94fe48\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-1fdaa7-6fcb4879fd-bcz9f"
Oct 30 20:50:28.676204 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:28.676165    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-1fdaa7-6fcb4879fd-bcz9f"
Oct 30 20:50:28.905503 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:28.905478    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-1fdaa7-bb8847cd7-kq2ml"]
Oct 30 20:50:28.907367 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:50:28.907338    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70aef1ff_8862_40b8_b3b7_c8878df1619f.slice/crio-459a351f71410b9142c4404d4a69248a2c2b831178a53897d0183bb6bc7f6353 WatchSource:0}: Error finding container 459a351f71410b9142c4404d4a69248a2c2b831178a53897d0183bb6bc7f6353: Status 404 returned error can't find the container with id 459a351f71410b9142c4404d4a69248a2c2b831178a53897d0183bb6bc7f6353
Oct 30 20:50:29.075427 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:29.075405    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-1fdaa7-6fcb4879fd-bcz9f"]
Oct 30 20:50:29.078671 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:50:29.077940    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5304445a_544a_4cb7_b79a_53502f94fe48.slice/crio-0f7df3d9f7bc13f1ef94b83bc06a0a9dc901a25c95326f4838cc1e8ccfdb985f WatchSource:0}: Error finding container 0f7df3d9f7bc13f1ef94b83bc06a0a9dc901a25c95326f4838cc1e8ccfdb985f: Status 404 returned error can't find the container with id 0f7df3d9f7bc13f1ef94b83bc06a0a9dc901a25c95326f4838cc1e8ccfdb985f
Oct 30 20:50:29.630849 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:29.630768    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-1fdaa7-6fcb4879fd-bcz9f" event={"ID":"5304445a-544a-4cb7-b79a-53502f94fe48","Type":"ContainerStarted","Data":"beca861051be812f67a97f823851e310d809f206f4bf724e896593cd93e3a955"}
Oct 30 20:50:29.631298 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:29.630880    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-1fdaa7-6fcb4879fd-bcz9f" event={"ID":"5304445a-544a-4cb7-b79a-53502f94fe48","Type":"ContainerStarted","Data":"0f7df3d9f7bc13f1ef94b83bc06a0a9dc901a25c95326f4838cc1e8ccfdb985f"}
Oct 30 20:50:29.631298 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:29.630912    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-1fdaa7-6fcb4879fd-bcz9f"
Oct 30 20:50:29.632449 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:29.632423    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1fdaa7-bb8847cd7-kq2ml" event={"ID":"70aef1ff-8862-40b8-b3b7-c8878df1619f","Type":"ContainerStarted","Data":"2f46b08c16b08a4e8cb84d4301eab1832cb7bee4fbf766401131eab9983ac404"}
Oct 30 20:50:29.632586 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:29.632467    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1fdaa7-bb8847cd7-kq2ml" event={"ID":"70aef1ff-8862-40b8-b3b7-c8878df1619f","Type":"ContainerStarted","Data":"459a351f71410b9142c4404d4a69248a2c2b831178a53897d0183bb6bc7f6353"}
Oct 30 20:50:29.632586 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:29.632508    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1fdaa7-bb8847cd7-kq2ml"
Oct 30 20:50:29.749138 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:29.749088    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-1fdaa7-6fcb4879fd-bcz9f" podStartSLOduration=1.749074046 podStartE2EDuration="1.749074046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:50:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:29.746679283 +0000 UTC m=+712.811455682" watchObservedRunningTime="2026-10-30 20:50:29.749074046 +0000 UTC m=+712.813850488"
Oct 30 20:50:29.865571 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:29.865519    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1fdaa7-bb8847cd7-kq2ml" podStartSLOduration=1.865504665 podStartE2EDuration="1.865504665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:50:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:29.863427203 +0000 UTC m=+712.928203614" watchObservedRunningTime="2026-10-30 20:50:29.865504665 +0000 UTC m=+712.930281062"
Oct 30 20:50:30.581929 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:30.581898    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-781058-6d867c875-285pt"
Oct 30 20:50:30.882071 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:30.881981    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-d061d0-9888979ff-kclfx"]
Oct 30 20:50:30.886063 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:30.886038    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d061d0-9888979ff-kclfx"
Oct 30 20:50:30.942373 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:30.942296    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-d061d0\""
Oct 30 20:50:30.942599 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:30.942568    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-d061d0-dockercfg-nlffj\""
Oct 30 20:50:30.944314 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:30.944290    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-d061d0-9888979ff-kclfx"]
Oct 30 20:50:31.022039 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:31.021999    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-589r8\" (UniqueName: \"kubernetes.io/projected/b47eb722-c69a-4565-b0ae-1b46f99090df-kube-api-access-589r8\") pod \"payload-pre-processing-e2e-worker-w2-d061d0-9888979ff-kclfx\" (UID: \"b47eb722-c69a-4565-b0ae-1b46f99090df\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d061d0-9888979ff-kclfx"
Oct 30 20:50:31.022234 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:31.022194    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b47eb722-c69a-4565-b0ae-1b46f99090df-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-d061d0-9888979ff-kclfx\" (UID: \"b47eb722-c69a-4565-b0ae-1b46f99090df\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d061d0-9888979ff-kclfx"
Oct 30 20:50:31.034524 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:31.034493    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-d061d0-644b5ff884-cj67l"]
Oct 30 20:50:31.039325 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:31.039293    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-d061d0-644b5ff884-cj67l"
Oct 30 20:50:31.097568 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:31.097527    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-d061d0-644b5ff884-cj67l"]
Oct 30 20:50:31.122964 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:31.122915    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b47eb722-c69a-4565-b0ae-1b46f99090df-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-d061d0-9888979ff-kclfx\" (UID: \"b47eb722-c69a-4565-b0ae-1b46f99090df\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d061d0-9888979ff-kclfx"
Oct 30 20:50:31.123167 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:31.123004    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-589r8\" (UniqueName: \"kubernetes.io/projected/b47eb722-c69a-4565-b0ae-1b46f99090df-kube-api-access-589r8\") pod \"payload-pre-processing-e2e-worker-w2-d061d0-9888979ff-kclfx\" (UID: \"b47eb722-c69a-4565-b0ae-1b46f99090df\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d061d0-9888979ff-kclfx"
Oct 30 20:50:31.123613 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:31.123592    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b47eb722-c69a-4565-b0ae-1b46f99090df-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-d061d0-9888979ff-kclfx\" (UID: \"b47eb722-c69a-4565-b0ae-1b46f99090df\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d061d0-9888979ff-kclfx"
Oct 30 20:50:31.180699 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:31.180669    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-589r8\" (UniqueName: \"kubernetes.io/projected/b47eb722-c69a-4565-b0ae-1b46f99090df-kube-api-access-589r8\") pod \"payload-pre-processing-e2e-worker-w2-d061d0-9888979ff-kclfx\" (UID: \"b47eb722-c69a-4565-b0ae-1b46f99090df\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d061d0-9888979ff-kclfx"
Oct 30 20:50:31.200862 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:31.200824    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d061d0-9888979ff-kclfx"
Oct 30 20:50:31.224519 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:31.224486    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a08925fb-aee2-425a-84da-c407fb8cfb7f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-d061d0-644b5ff884-cj67l\" (UID: \"a08925fb-aee2-425a-84da-c407fb8cfb7f\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-d061d0-644b5ff884-cj67l"
Oct 30 20:50:31.224629 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:31.224615    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t89rm\" (UniqueName: \"kubernetes.io/projected/a08925fb-aee2-425a-84da-c407fb8cfb7f-kube-api-access-t89rm\") pod \"payload-processing-e2e-worker-w2-d061d0-644b5ff884-cj67l\" (UID: \"a08925fb-aee2-425a-84da-c407fb8cfb7f\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-d061d0-644b5ff884-cj67l"
Oct 30 20:50:31.326324 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:31.326281    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a08925fb-aee2-425a-84da-c407fb8cfb7f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-d061d0-644b5ff884-cj67l\" (UID: \"a08925fb-aee2-425a-84da-c407fb8cfb7f\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-d061d0-644b5ff884-cj67l"
Oct 30 20:50:31.326521 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:31.326343    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t89rm\" (UniqueName: \"kubernetes.io/projected/a08925fb-aee2-425a-84da-c407fb8cfb7f-kube-api-access-t89rm\") pod \"payload-processing-e2e-worker-w2-d061d0-644b5ff884-cj67l\" (UID: \"a08925fb-aee2-425a-84da-c407fb8cfb7f\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-d061d0-644b5ff884-cj67l"
Oct 30 20:50:31.327041 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:31.327018    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a08925fb-aee2-425a-84da-c407fb8cfb7f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-d061d0-644b5ff884-cj67l\" (UID: \"a08925fb-aee2-425a-84da-c407fb8cfb7f\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-d061d0-644b5ff884-cj67l"
Oct 30 20:50:31.383808 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:31.383763    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t89rm\" (UniqueName: \"kubernetes.io/projected/a08925fb-aee2-425a-84da-c407fb8cfb7f-kube-api-access-t89rm\") pod \"payload-processing-e2e-worker-w2-d061d0-644b5ff884-cj67l\" (UID: \"a08925fb-aee2-425a-84da-c407fb8cfb7f\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-d061d0-644b5ff884-cj67l"
Oct 30 20:50:31.510048 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:31.509948    5713 patch_prober.go:29] interesting pod/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:50:31.510048 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:31.510004    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq" podUID="63ace536-4242-41d2-b1a0-3b10efad3333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:50:31.583039 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:31.583005    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-d061d0-9888979ff-kclfx"]
Oct 30 20:50:31.584953 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:50:31.584920    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb47eb722_c69a_4565_b0ae_1b46f99090df.slice/crio-c94b3f7ffe04da83cd245a72ec4250da65071d8b2fb9514be6673d598b29a3b1 WatchSource:0}: Error finding container c94b3f7ffe04da83cd245a72ec4250da65071d8b2fb9514be6673d598b29a3b1: Status 404 returned error can't find the container with id c94b3f7ffe04da83cd245a72ec4250da65071d8b2fb9514be6673d598b29a3b1
Oct 30 20:50:31.647233 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:31.647196    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d061d0-9888979ff-kclfx" event={"ID":"b47eb722-c69a-4565-b0ae-1b46f99090df","Type":"ContainerStarted","Data":"c94b3f7ffe04da83cd245a72ec4250da65071d8b2fb9514be6673d598b29a3b1"}
Oct 30 20:50:31.656464 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:31.656438    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-d061d0-644b5ff884-cj67l"
Oct 30 20:50:32.038654 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:32.038622    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-d061d0-644b5ff884-cj67l"]
Oct 30 20:50:32.039179 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:50:32.039156    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda08925fb_aee2_425a_84da_c407fb8cfb7f.slice/crio-339d1f349064852d0124c70abb184796cafbfda1e72b8d5e460932ddd2cc8211 WatchSource:0}: Error finding container 339d1f349064852d0124c70abb184796cafbfda1e72b8d5e460932ddd2cc8211: Status 404 returned error can't find the container with id 339d1f349064852d0124c70abb184796cafbfda1e72b8d5e460932ddd2cc8211
Oct 30 20:50:32.595010 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:32.594980    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-8fa2a7-977ff4795-mgd5h"
Oct 30 20:50:32.654256 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:32.654213    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-d061d0-644b5ff884-cj67l" event={"ID":"a08925fb-aee2-425a-84da-c407fb8cfb7f","Type":"ContainerStarted","Data":"c815c95c65e21b95193016db19fc0cfce93d0c3cde714312f9408b5de525fb0f"}
Oct 30 20:50:32.654451 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:32.654280    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-d061d0-644b5ff884-cj67l" event={"ID":"a08925fb-aee2-425a-84da-c407fb8cfb7f","Type":"ContainerStarted","Data":"339d1f349064852d0124c70abb184796cafbfda1e72b8d5e460932ddd2cc8211"}
Oct 30 20:50:32.654451 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:32.654311    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-d061d0-644b5ff884-cj67l"
Oct 30 20:50:32.655948 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:32.655921    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d061d0-9888979ff-kclfx" event={"ID":"b47eb722-c69a-4565-b0ae-1b46f99090df","Type":"ContainerStarted","Data":"11d2d335bebea202947eed0e482f1b44ea4f97286b2a3f48fbb9d4c0aea3ac3c"}
Oct 30 20:50:32.656091 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:32.656078    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d061d0-9888979ff-kclfx"
Oct 30 20:50:32.832156 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:32.832092    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d061d0-9888979ff-kclfx" podStartSLOduration=2.832079269 podStartE2EDuration="2.832079269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:50:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:32.831244186 +0000 UTC m=+715.896020585" watchObservedRunningTime="2026-10-30 20:50:32.832079269 +0000 UTC m=+715.896855667"
Oct 30 20:50:32.950421 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:32.950353    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-d061d0-644b5ff884-cj67l" podStartSLOduration=2.9503320459999998 podStartE2EDuration="2.950332046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:50:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:32.948336918 +0000 UTC m=+716.013113315" watchObservedRunningTime="2026-10-30 20:50:32.950332046 +0000 UTC m=+716.015108444"
Oct 30 20:50:33.662881 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:33.662841    5713 generic.go:356] "Generic (PLEG): container finished" podID="63ace536-4242-41d2-b1a0-3b10efad3333" containerID="b1dd9a119d562d2f3f9f7ca2671288b163d12504b86500f6e47a34b420812bdf" exitCode=0
Oct 30 20:50:33.663282 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:33.662913    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq" event={"ID":"63ace536-4242-41d2-b1a0-3b10efad3333","Type":"ContainerDied","Data":"b1dd9a119d562d2f3f9f7ca2671288b163d12504b86500f6e47a34b420812bdf"}
Oct 30 20:50:33.663282 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:33.663276    5713 scope.go:122] "RemoveContainer" containerID="b1dd9a119d562d2f3f9f7ca2671288b163d12504b86500f6e47a34b420812bdf"
Oct 30 20:50:33.664700 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:33.664676    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 20:50:33.664764 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:33.664740    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 20:50:33.664837 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:33.664775    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 20:50:34.670264 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:34.670225    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq" event={"ID":"63ace536-4242-41d2-b1a0-3b10efad3333","Type":"ContainerStarted","Data":"983ffde06d9cc0123227140305489c5db6ae4b16afed321ad2f93946fa8b52a3"}
Oct 30 20:50:35.662201 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:35.662161    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq"
Oct 30 20:50:35.662487 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:35.662461    5713 patch_prober.go:29] interesting pod/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 30 20:50:35.662546 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:35.662522    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq" podUID="63ace536-4242-41d2-b1a0-3b10efad3333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 30 20:50:36.662401 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:36.662364    5713 patch_prober.go:29] interesting pod/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 30 20:50:36.662837 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:36.662421    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq" podUID="63ace536-4242-41d2-b1a0-3b10efad3333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 30 20:50:37.662379 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:37.662340    5713 patch_prober.go:29] interesting pod/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 30 20:50:37.662565 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:37.662403    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq" podUID="63ace536-4242-41d2-b1a0-3b10efad3333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 30 20:50:38.667814 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:38.664176    5713 patch_prober.go:29] interesting pod/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 30 20:50:38.667814 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:38.664241    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq" podUID="63ace536-4242-41d2-b1a0-3b10efad3333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 30 20:50:39.662402 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:39.662363    5713 patch_prober.go:29] interesting pod/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 30 20:50:39.662623 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:39.662421    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq" podUID="63ace536-4242-41d2-b1a0-3b10efad3333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 30 20:50:40.660203 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:40.660167    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-1fdaa7-6fcb4879fd-bcz9f"
Oct 30 20:50:40.660750 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:40.660364    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1fdaa7-bb8847cd7-kq2ml"
Oct 30 20:50:40.665144 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:40.665117    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq"
Oct 30 20:50:40.665399 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:40.665385    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq"
Oct 30 20:50:40.666073 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:40.666058    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq"
Oct 30 20:50:43.665622 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:43.665585    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-d061d0-644b5ff884-cj67l"
Oct 30 20:50:43.666128 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:43.665648    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d061d0-9888979ff-kclfx"
Oct 30 20:50:45.503978 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:45.503935    5713 patch_prober.go:29] interesting pod/e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:50:45.504589 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:45.504550    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs" podUID="7907175f-e439-48d7-8754-04591bde77cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:50:47.014028 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:47.013946    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-682999/simulator-w3-682999-kserve-57cfcb6f8f-6mgrg"]
Oct 30 20:50:47.017411 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:47.017390    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-682999/simulator-w3-682999-kserve-57cfcb6f8f-6mgrg"
Oct 30 20:50:47.071619 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:47.071594    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-682999\"/\"openshift-service-ca.crt\""
Oct 30 20:50:47.086115 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:47.086087    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-682999\"/\"simulator-w3-682999-kserve-self-signed-certs\""
Oct 30 20:50:47.104232 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:47.104201    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53296e7f-c6bd-45f2-ac31-004feddc63a7-home\") pod \"simulator-w3-682999-kserve-57cfcb6f8f-6mgrg\" (UID: \"53296e7f-c6bd-45f2-ac31-004feddc63a7\") " pod="e2e-models-e2e-worker-w3-682999/simulator-w3-682999-kserve-57cfcb6f8f-6mgrg"
Oct 30 20:50:47.104232 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:47.104243    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53296e7f-c6bd-45f2-ac31-004feddc63a7-tmp-dir\") pod \"simulator-w3-682999-kserve-57cfcb6f8f-6mgrg\" (UID: \"53296e7f-c6bd-45f2-ac31-004feddc63a7\") " pod="e2e-models-e2e-worker-w3-682999/simulator-w3-682999-kserve-57cfcb6f8f-6mgrg"
Oct 30 20:50:47.104451 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:47.104273    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d4bb\" (UniqueName: \"kubernetes.io/projected/53296e7f-c6bd-45f2-ac31-004feddc63a7-kube-api-access-2d4bb\") pod \"simulator-w3-682999-kserve-57cfcb6f8f-6mgrg\" (UID: \"53296e7f-c6bd-45f2-ac31-004feddc63a7\") " pod="e2e-models-e2e-worker-w3-682999/simulator-w3-682999-kserve-57cfcb6f8f-6mgrg"
Oct 30 20:50:47.104451 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:47.104357    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53296e7f-c6bd-45f2-ac31-004feddc63a7-model-cache\") pod \"simulator-w3-682999-kserve-57cfcb6f8f-6mgrg\" (UID: \"53296e7f-c6bd-45f2-ac31-004feddc63a7\") " pod="e2e-models-e2e-worker-w3-682999/simulator-w3-682999-kserve-57cfcb6f8f-6mgrg"
Oct 30 20:50:47.104451 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:47.104401    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53296e7f-c6bd-45f2-ac31-004feddc63a7-tls-certs\") pod \"simulator-w3-682999-kserve-57cfcb6f8f-6mgrg\" (UID: \"53296e7f-c6bd-45f2-ac31-004feddc63a7\") " pod="e2e-models-e2e-worker-w3-682999/simulator-w3-682999-kserve-57cfcb6f8f-6mgrg"
Oct 30 20:50:47.104552 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:47.104488    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53296e7f-c6bd-45f2-ac31-004feddc63a7-dshm\") pod \"simulator-w3-682999-kserve-57cfcb6f8f-6mgrg\" (UID: \"53296e7f-c6bd-45f2-ac31-004feddc63a7\") " pod="e2e-models-e2e-worker-w3-682999/simulator-w3-682999-kserve-57cfcb6f8f-6mgrg"
Oct 30 20:50:47.121429 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:47.121385    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-682999\"/\"kube-root-ca.crt\""
Oct 30 20:50:47.121589 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:47.121538    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-682999\"/\"default-dockercfg-6hppw\""
Oct 30 20:50:47.123779 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:47.123751    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-682999/simulator-w3-682999-kserve-57cfcb6f8f-6mgrg"]
Oct 30 20:50:47.206077 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:47.206040    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53296e7f-c6bd-45f2-ac31-004feddc63a7-dshm\") pod \"simulator-w3-682999-kserve-57cfcb6f8f-6mgrg\" (UID: \"53296e7f-c6bd-45f2-ac31-004feddc63a7\") " pod="e2e-models-e2e-worker-w3-682999/simulator-w3-682999-kserve-57cfcb6f8f-6mgrg"
Oct 30 20:50:47.206249 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:47.206179    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53296e7f-c6bd-45f2-ac31-004feddc63a7-home\") pod \"simulator-w3-682999-kserve-57cfcb6f8f-6mgrg\" (UID: \"53296e7f-c6bd-45f2-ac31-004feddc63a7\") " pod="e2e-models-e2e-worker-w3-682999/simulator-w3-682999-kserve-57cfcb6f8f-6mgrg"
Oct 30 20:50:47.206249 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:47.206230    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53296e7f-c6bd-45f2-ac31-004feddc63a7-tmp-dir\") pod \"simulator-w3-682999-kserve-57cfcb6f8f-6mgrg\" (UID: \"53296e7f-c6bd-45f2-ac31-004feddc63a7\") " pod="e2e-models-e2e-worker-w3-682999/simulator-w3-682999-kserve-57cfcb6f8f-6mgrg"
Oct 30 20:50:47.206324 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:47.206275    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d4bb\" (UniqueName: \"kubernetes.io/projected/53296e7f-c6bd-45f2-ac31-004feddc63a7-kube-api-access-2d4bb\") pod \"simulator-w3-682999-kserve-57cfcb6f8f-6mgrg\" (UID: \"53296e7f-c6bd-45f2-ac31-004feddc63a7\") " pod="e2e-models-e2e-worker-w3-682999/simulator-w3-682999-kserve-57cfcb6f8f-6mgrg"
Oct 30 20:50:47.206324 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:47.206307    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53296e7f-c6bd-45f2-ac31-004feddc63a7-model-cache\") pod \"simulator-w3-682999-kserve-57cfcb6f8f-6mgrg\" (UID: \"53296e7f-c6bd-45f2-ac31-004feddc63a7\") " pod="e2e-models-e2e-worker-w3-682999/simulator-w3-682999-kserve-57cfcb6f8f-6mgrg"
Oct 30 20:50:47.206413 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:47.206362    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53296e7f-c6bd-45f2-ac31-004feddc63a7-tls-certs\") pod \"simulator-w3-682999-kserve-57cfcb6f8f-6mgrg\" (UID: \"53296e7f-c6bd-45f2-ac31-004feddc63a7\") " pod="e2e-models-e2e-worker-w3-682999/simulator-w3-682999-kserve-57cfcb6f8f-6mgrg"
Oct 30 20:50:47.206648 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:47.206619    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53296e7f-c6bd-45f2-ac31-004feddc63a7-home\") pod \"simulator-w3-682999-kserve-57cfcb6f8f-6mgrg\" (UID: \"53296e7f-c6bd-45f2-ac31-004feddc63a7\") " pod="e2e-models-e2e-worker-w3-682999/simulator-w3-682999-kserve-57cfcb6f8f-6mgrg"
Oct 30 20:50:47.206808 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:47.206634    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53296e7f-c6bd-45f2-ac31-004feddc63a7-model-cache\") pod \"simulator-w3-682999-kserve-57cfcb6f8f-6mgrg\" (UID: \"53296e7f-c6bd-45f2-ac31-004feddc63a7\") " pod="e2e-models-e2e-worker-w3-682999/simulator-w3-682999-kserve-57cfcb6f8f-6mgrg"
Oct 30 20:50:47.206808 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:47.206726    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53296e7f-c6bd-45f2-ac31-004feddc63a7-tmp-dir\") pod \"simulator-w3-682999-kserve-57cfcb6f8f-6mgrg\" (UID: \"53296e7f-c6bd-45f2-ac31-004feddc63a7\") " pod="e2e-models-e2e-worker-w3-682999/simulator-w3-682999-kserve-57cfcb6f8f-6mgrg"
Oct 30 20:50:47.208546 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:47.208519    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53296e7f-c6bd-45f2-ac31-004feddc63a7-dshm\") pod \"simulator-w3-682999-kserve-57cfcb6f8f-6mgrg\" (UID: \"53296e7f-c6bd-45f2-ac31-004feddc63a7\") " pod="e2e-models-e2e-worker-w3-682999/simulator-w3-682999-kserve-57cfcb6f8f-6mgrg"
Oct 30 20:50:47.208941 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:47.208925    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53296e7f-c6bd-45f2-ac31-004feddc63a7-tls-certs\") pod \"simulator-w3-682999-kserve-57cfcb6f8f-6mgrg\" (UID: \"53296e7f-c6bd-45f2-ac31-004feddc63a7\") " pod="e2e-models-e2e-worker-w3-682999/simulator-w3-682999-kserve-57cfcb6f8f-6mgrg"
Oct 30 20:50:47.264327 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:47.264247    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d4bb\" (UniqueName: \"kubernetes.io/projected/53296e7f-c6bd-45f2-ac31-004feddc63a7-kube-api-access-2d4bb\") pod \"simulator-w3-682999-kserve-57cfcb6f8f-6mgrg\" (UID: \"53296e7f-c6bd-45f2-ac31-004feddc63a7\") " pod="e2e-models-e2e-worker-w3-682999/simulator-w3-682999-kserve-57cfcb6f8f-6mgrg"
Oct 30 20:50:47.332134 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:47.332091    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-682999/simulator-w3-682999-kserve-57cfcb6f8f-6mgrg"
Oct 30 20:50:47.718637 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:47.718612    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-682999/simulator-w3-682999-kserve-57cfcb6f8f-6mgrg"]
Oct 30 20:50:47.720773 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:50:47.720744    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod53296e7f_c6bd_45f2_ac31_004feddc63a7.slice/crio-53b656b9968d2435a783f75a9bfb35f98e8d71e8d47b9eebda422ff927d06919 WatchSource:0}: Error finding container 53b656b9968d2435a783f75a9bfb35f98e8d71e8d47b9eebda422ff927d06919: Status 404 returned error can't find the container with id 53b656b9968d2435a783f75a9bfb35f98e8d71e8d47b9eebda422ff927d06919
Oct 30 20:50:47.752150 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:47.752106    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-682999/simulator-w3-682999-kserve-57cfcb6f8f-6mgrg" event={"ID":"53296e7f-c6bd-45f2-ac31-004feddc63a7","Type":"ContainerStarted","Data":"53b656b9968d2435a783f75a9bfb35f98e8d71e8d47b9eebda422ff927d06919"}
Oct 30 20:50:48.759477 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:48.759443    5713 generic.go:356] "Generic (PLEG): container finished" podID="7907175f-e439-48d7-8754-04591bde77cd" containerID="819041f4dd6f4abf2ee3ffddd4269f5574b61ea26270d5292d338e4bbf72ce51" exitCode=0
Oct 30 20:50:48.760043 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:48.759522    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs" event={"ID":"7907175f-e439-48d7-8754-04591bde77cd","Type":"ContainerDied","Data":"819041f4dd6f4abf2ee3ffddd4269f5574b61ea26270d5292d338e4bbf72ce51"}
Oct 30 20:50:48.760114 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:48.760069    5713 scope.go:122] "RemoveContainer" containerID="819041f4dd6f4abf2ee3ffddd4269f5574b61ea26270d5292d338e4bbf72ce51"
Oct 30 20:50:48.761232 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:48.761210    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-682999/simulator-w3-682999-kserve-57cfcb6f8f-6mgrg" event={"ID":"53296e7f-c6bd-45f2-ac31-004feddc63a7","Type":"ContainerStarted","Data":"d7ee6f5080fa7876e24f0680c9b284c34d7aee714b48cb2ac58d36d8348e318f"}
Oct 30 20:50:48.761507 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:48.761484    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 20:50:48.761566 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:48.761549    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 20:50:48.761602 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:48.761580    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 20:50:49.332862 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:49.332829    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-682999/simulator-w3-682999-kserve-57cfcb6f8f-6mgrg"
Oct 30 20:50:49.769474 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:49.769439    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs" event={"ID":"7907175f-e439-48d7-8754-04591bde77cd","Type":"ContainerStarted","Data":"0e42e9d485f3930f8a16e3310171c6c31f47bbcdfb4268da004df78e284e2b98"}
Oct 30 20:50:49.948813 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:49.940779    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-682999/simulator-w3-682999-kserve-57cfcb6f8f-6mgrg" podStartSLOduration=3.940755889 podStartE2EDuration="3.940755889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:50:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:49.148856764 +0000 UTC m=+732.213633162" watchObservedRunningTime="2026-10-30 20:50:49.940755889 +0000 UTC m=+733.005532287"
Oct 30 20:50:50.261194 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:50.261150    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs"
Oct 30 20:50:50.261415 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:50.261372    5713 patch_prober.go:29] interesting pod/e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 30 20:50:50.261497 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:50.261424    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs" podUID="7907175f-e439-48d7-8754-04591bde77cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 30 20:50:51.261691 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:51.261637    5713 patch_prober.go:29] interesting pod/e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 30 20:50:51.262191 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:51.261735    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs" podUID="7907175f-e439-48d7-8754-04591bde77cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 30 20:50:52.261390 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:52.261344    5713 patch_prober.go:29] interesting pod/e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 30 20:50:52.261599 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:52.261415    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs" podUID="7907175f-e439-48d7-8754-04591bde77cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 30 20:50:53.261591 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:53.261556    5713 patch_prober.go:29] interesting pod/e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 30 20:50:53.262052 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:53.261614    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs" podUID="7907175f-e439-48d7-8754-04591bde77cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 30 20:50:54.261710 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:54.261661    5713 patch_prober.go:29] interesting pod/e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 30 20:50:54.262289 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:54.261724    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs" podUID="7907175f-e439-48d7-8754-04591bde77cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 30 20:50:54.483422 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:50:54.483386    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 20:50:55.265135 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:55.265099    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs"
Oct 30 20:50:55.265590 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:55.265408    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs"
Oct 30 20:50:55.266098 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:55.266082    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs"
Oct 30 20:50:57.333160 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:57.333129    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-682999/simulator-w3-682999-kserve-57cfcb6f8f-6mgrg"
Oct 30 20:50:57.345463 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:57.345435    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-682999/simulator-w3-682999-kserve-57cfcb6f8f-6mgrg"
Oct 30 20:50:57.824572 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:50:57.824547    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-682999/simulator-w3-682999-kserve-57cfcb6f8f-6mgrg"
Oct 30 20:51:09.907817 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:09.907706    5713 generic.go:356] "Generic (PLEG): container finished" podID="e4e643cf-8167-42e3-a104-03ac5f7bd133" containerID="e885c7f726a81161a04012f02368d308828df70aa2ed6e28ab7870842505ce8a" exitCode=0
Oct 30 20:51:09.907817 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:09.907773    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm" event={"ID":"e4e643cf-8167-42e3-a104-03ac5f7bd133","Type":"ContainerDied","Data":"e885c7f726a81161a04012f02368d308828df70aa2ed6e28ab7870842505ce8a"}
Oct 30 20:51:09.908269 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:09.908175    5713 scope.go:122] "RemoveContainer" containerID="e885c7f726a81161a04012f02368d308828df70aa2ed6e28ab7870842505ce8a"
Oct 30 20:51:09.909614 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:09.909584    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 20:51:09.909679 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:09.909654    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 20:51:09.909714 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:09.909691    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 20:51:10.915370 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:10.915338    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm" event={"ID":"e4e643cf-8167-42e3-a104-03ac5f7bd133","Type":"ContainerStarted","Data":"06cf657bc4c28717d86d3c09fd43b27935bceb0b8e9c9d563fa4a76e3ab76cd1"}
Oct 30 20:51:11.708869 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:11.708831    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm"
Oct 30 20:51:11.709903 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:11.709878    5713 patch_prober.go:29] interesting pod/e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 30 20:51:11.710026 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:11.709926    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm" podUID="e4e643cf-8167-42e3-a104-03ac5f7bd133" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 30 20:51:12.709100 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:12.709065    5713 patch_prober.go:29] interesting pod/e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 30 20:51:12.709539 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:12.709130    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm" podUID="e4e643cf-8167-42e3-a104-03ac5f7bd133" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 30 20:51:12.722305 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:12.722267    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-2b425d/simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk"]
Oct 30 20:51:12.725238 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:12.725219    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2b425d/simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk"
Oct 30 20:51:12.801568 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:12.801530    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8108824-03ff-4c1d-9f5b-07be3a3a92d6-model-cache\") pod \"simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk\" (UID: \"e8108824-03ff-4c1d-9f5b-07be3a3a92d6\") " pod="e2e-models-e2e-worker-w5-2b425d/simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk"
Oct 30 20:51:12.801750 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:12.801581    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8108824-03ff-4c1d-9f5b-07be3a3a92d6-tls-certs\") pod \"simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk\" (UID: \"e8108824-03ff-4c1d-9f5b-07be3a3a92d6\") " pod="e2e-models-e2e-worker-w5-2b425d/simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk"
Oct 30 20:51:12.801750 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:12.801671    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkzkj\" (UniqueName: \"kubernetes.io/projected/e8108824-03ff-4c1d-9f5b-07be3a3a92d6-kube-api-access-kkzkj\") pod \"simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk\" (UID: \"e8108824-03ff-4c1d-9f5b-07be3a3a92d6\") " pod="e2e-models-e2e-worker-w5-2b425d/simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk"
Oct 30 20:51:12.801750 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:12.801745    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8108824-03ff-4c1d-9f5b-07be3a3a92d6-home\") pod \"simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk\" (UID: \"e8108824-03ff-4c1d-9f5b-07be3a3a92d6\") " pod="e2e-models-e2e-worker-w5-2b425d/simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk"
Oct 30 20:51:12.801905 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:12.801886    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8108824-03ff-4c1d-9f5b-07be3a3a92d6-tmp-dir\") pod \"simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk\" (UID: \"e8108824-03ff-4c1d-9f5b-07be3a3a92d6\") " pod="e2e-models-e2e-worker-w5-2b425d/simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk"
Oct 30 20:51:12.801940 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:12.801909    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8108824-03ff-4c1d-9f5b-07be3a3a92d6-dshm\") pod \"simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk\" (UID: \"e8108824-03ff-4c1d-9f5b-07be3a3a92d6\") " pod="e2e-models-e2e-worker-w5-2b425d/simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk"
Oct 30 20:51:12.818689 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:12.818660    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-2b425d\"/\"kube-root-ca.crt\""
Oct 30 20:51:12.818689 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:12.818683    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-2b425d\"/\"openshift-service-ca.crt\""
Oct 30 20:51:12.818906 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:12.818883    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-2b425d\"/\"default-dockercfg-h4z8l\""
Oct 30 20:51:12.828169 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:12.828147    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-2b425d\"/\"simulator-w5-2b425d-kserve-self-signed-certs\""
Oct 30 20:51:12.831081 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:12.831059    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-2b425d/simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk"]
Oct 30 20:51:12.903558 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:12.903518    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8108824-03ff-4c1d-9f5b-07be3a3a92d6-home\") pod \"simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk\" (UID: \"e8108824-03ff-4c1d-9f5b-07be3a3a92d6\") " pod="e2e-models-e2e-worker-w5-2b425d/simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk"
Oct 30 20:51:12.903836 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:12.903641    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8108824-03ff-4c1d-9f5b-07be3a3a92d6-tmp-dir\") pod \"simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk\" (UID: \"e8108824-03ff-4c1d-9f5b-07be3a3a92d6\") " pod="e2e-models-e2e-worker-w5-2b425d/simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk"
Oct 30 20:51:12.903836 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:12.903674    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8108824-03ff-4c1d-9f5b-07be3a3a92d6-dshm\") pod \"simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk\" (UID: \"e8108824-03ff-4c1d-9f5b-07be3a3a92d6\") " pod="e2e-models-e2e-worker-w5-2b425d/simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk"
Oct 30 20:51:12.903836 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:12.903735    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8108824-03ff-4c1d-9f5b-07be3a3a92d6-model-cache\") pod \"simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk\" (UID: \"e8108824-03ff-4c1d-9f5b-07be3a3a92d6\") " pod="e2e-models-e2e-worker-w5-2b425d/simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk"
Oct 30 20:51:12.903836 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:12.903765    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8108824-03ff-4c1d-9f5b-07be3a3a92d6-tls-certs\") pod \"simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk\" (UID: \"e8108824-03ff-4c1d-9f5b-07be3a3a92d6\") " pod="e2e-models-e2e-worker-w5-2b425d/simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk"
Oct 30 20:51:12.903836 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:12.903815    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkzkj\" (UniqueName: \"kubernetes.io/projected/e8108824-03ff-4c1d-9f5b-07be3a3a92d6-kube-api-access-kkzkj\") pod \"simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk\" (UID: \"e8108824-03ff-4c1d-9f5b-07be3a3a92d6\") " pod="e2e-models-e2e-worker-w5-2b425d/simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk"
Oct 30 20:51:12.904147 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:12.904014    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8108824-03ff-4c1d-9f5b-07be3a3a92d6-home\") pod \"simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk\" (UID: \"e8108824-03ff-4c1d-9f5b-07be3a3a92d6\") " pod="e2e-models-e2e-worker-w5-2b425d/simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk"
Oct 30 20:51:12.904480 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:12.904453    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8108824-03ff-4c1d-9f5b-07be3a3a92d6-model-cache\") pod \"simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk\" (UID: \"e8108824-03ff-4c1d-9f5b-07be3a3a92d6\") " pod="e2e-models-e2e-worker-w5-2b425d/simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk"
Oct 30 20:51:12.905397 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:12.904614    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8108824-03ff-4c1d-9f5b-07be3a3a92d6-tmp-dir\") pod \"simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk\" (UID: \"e8108824-03ff-4c1d-9f5b-07be3a3a92d6\") " pod="e2e-models-e2e-worker-w5-2b425d/simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk"
Oct 30 20:51:12.907906 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:12.907607    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8108824-03ff-4c1d-9f5b-07be3a3a92d6-dshm\") pod \"simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk\" (UID: \"e8108824-03ff-4c1d-9f5b-07be3a3a92d6\") " pod="e2e-models-e2e-worker-w5-2b425d/simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk"
Oct 30 20:51:12.908439 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:12.908417    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8108824-03ff-4c1d-9f5b-07be3a3a92d6-tls-certs\") pod \"simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk\" (UID: \"e8108824-03ff-4c1d-9f5b-07be3a3a92d6\") " pod="e2e-models-e2e-worker-w5-2b425d/simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk"
Oct 30 20:51:12.970342 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:12.970261    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkzkj\" (UniqueName: \"kubernetes.io/projected/e8108824-03ff-4c1d-9f5b-07be3a3a92d6-kube-api-access-kkzkj\") pod \"simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk\" (UID: \"e8108824-03ff-4c1d-9f5b-07be3a3a92d6\") " pod="e2e-models-e2e-worker-w5-2b425d/simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk"
Oct 30 20:51:13.049063 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:13.049027    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2b425d/simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk"
Oct 30 20:51:13.459931 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:13.459885    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-2b425d/simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk"]
Oct 30 20:51:13.462412 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:51:13.462380    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8108824_03ff_4c1d_9f5b_07be3a3a92d6.slice/crio-0dbe75693a989a7d5180176d8a3d9742cd0328525eb45943ed1c8366782cc0b5 WatchSource:0}: Error finding container 0dbe75693a989a7d5180176d8a3d9742cd0328525eb45943ed1c8366782cc0b5: Status 404 returned error can't find the container with id 0dbe75693a989a7d5180176d8a3d9742cd0328525eb45943ed1c8366782cc0b5
Oct 30 20:51:13.709721 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:13.709633    5713 patch_prober.go:29] interesting pod/e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 30 20:51:13.709721 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:13.709689    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm" podUID="e4e643cf-8167-42e3-a104-03ac5f7bd133" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 30 20:51:13.961184 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:13.961059    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2b425d/simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk" event={"ID":"e8108824-03ff-4c1d-9f5b-07be3a3a92d6","Type":"ContainerStarted","Data":"559e5d07ad88e9d0f729142d98d4be7401236036557c9dfec80618fffd659db5"}
Oct 30 20:51:13.961184 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:13.961131    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2b425d/simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk" event={"ID":"e8108824-03ff-4c1d-9f5b-07be3a3a92d6","Type":"ContainerStarted","Data":"0dbe75693a989a7d5180176d8a3d9742cd0328525eb45943ed1c8366782cc0b5"}
Oct 30 20:51:14.055515 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:14.055480    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2b425d/simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk"
Oct 30 20:51:14.078997 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:14.078940    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-2b425d/simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk" podStartSLOduration=2.07892578 podStartE2EDuration="2.07892578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:51:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:51:14.077972825 +0000 UTC m=+757.142749224" watchObservedRunningTime="2026-10-30 20:51:14.07892578 +0000 UTC m=+757.143702177"
Oct 30 20:51:14.709515 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:14.709470    5713 patch_prober.go:29] interesting pod/e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 30 20:51:14.709733 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:14.709533    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm" podUID="e4e643cf-8167-42e3-a104-03ac5f7bd133" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 30 20:51:15.709139 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:15.709095    5713 patch_prober.go:29] interesting pod/e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 30 20:51:15.709361 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:15.709156    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm" podUID="e4e643cf-8167-42e3-a104-03ac5f7bd133" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 30 20:51:16.473631 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:16.473594    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm"
Oct 30 20:51:16.713248 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:16.713221    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm"
Oct 30 20:51:16.986325 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:16.986301    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm"
Oct 30 20:51:23.050038 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:23.049998    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-2b425d/simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk"
Oct 30 20:51:23.062661 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:23.062634    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-2b425d/simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk"
Oct 30 20:51:24.038545 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:24.038513    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2b425d/simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk"
Oct 30 20:51:24.458868 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:51:24.458835    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 20:51:41.479950 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:41.479916    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-682999/premium-w3-682999-kserve-5f47cf9b8d-xqnnr"]
Oct 30 20:51:41.484760 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:41.484741    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-682999/premium-w3-682999-kserve-5f47cf9b8d-xqnnr"
Oct 30 20:51:41.532336 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:41.532296    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48ptx\" (UniqueName: \"kubernetes.io/projected/a54cd33d-feef-42bb-a78b-7480634cb58d-kube-api-access-48ptx\") pod \"premium-w3-682999-kserve-5f47cf9b8d-xqnnr\" (UID: \"a54cd33d-feef-42bb-a78b-7480634cb58d\") " pod="e2e-models-e2e-worker-w3-682999/premium-w3-682999-kserve-5f47cf9b8d-xqnnr"
Oct 30 20:51:41.532499 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:41.532342    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a54cd33d-feef-42bb-a78b-7480634cb58d-tls-certs\") pod \"premium-w3-682999-kserve-5f47cf9b8d-xqnnr\" (UID: \"a54cd33d-feef-42bb-a78b-7480634cb58d\") " pod="e2e-models-e2e-worker-w3-682999/premium-w3-682999-kserve-5f47cf9b8d-xqnnr"
Oct 30 20:51:41.532499 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:41.532458    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a54cd33d-feef-42bb-a78b-7480634cb58d-dshm\") pod \"premium-w3-682999-kserve-5f47cf9b8d-xqnnr\" (UID: \"a54cd33d-feef-42bb-a78b-7480634cb58d\") " pod="e2e-models-e2e-worker-w3-682999/premium-w3-682999-kserve-5f47cf9b8d-xqnnr"
Oct 30 20:51:41.532594 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:41.532511    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a54cd33d-feef-42bb-a78b-7480634cb58d-home\") pod \"premium-w3-682999-kserve-5f47cf9b8d-xqnnr\" (UID: \"a54cd33d-feef-42bb-a78b-7480634cb58d\") " pod="e2e-models-e2e-worker-w3-682999/premium-w3-682999-kserve-5f47cf9b8d-xqnnr"
Oct 30 20:51:41.532635 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:41.532616    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a54cd33d-feef-42bb-a78b-7480634cb58d-tmp-dir\") pod \"premium-w3-682999-kserve-5f47cf9b8d-xqnnr\" (UID: \"a54cd33d-feef-42bb-a78b-7480634cb58d\") " pod="e2e-models-e2e-worker-w3-682999/premium-w3-682999-kserve-5f47cf9b8d-xqnnr"
Oct 30 20:51:41.532687 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:41.532675    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a54cd33d-feef-42bb-a78b-7480634cb58d-model-cache\") pod \"premium-w3-682999-kserve-5f47cf9b8d-xqnnr\" (UID: \"a54cd33d-feef-42bb-a78b-7480634cb58d\") " pod="e2e-models-e2e-worker-w3-682999/premium-w3-682999-kserve-5f47cf9b8d-xqnnr"
Oct 30 20:51:41.537886 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:41.537860    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-682999\"/\"premium-w3-682999-kserve-self-signed-certs\""
Oct 30 20:51:41.590289 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:41.590255    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-682999/premium-w3-682999-kserve-5f47cf9b8d-xqnnr"]
Oct 30 20:51:41.633919 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:41.633858    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48ptx\" (UniqueName: \"kubernetes.io/projected/a54cd33d-feef-42bb-a78b-7480634cb58d-kube-api-access-48ptx\") pod \"premium-w3-682999-kserve-5f47cf9b8d-xqnnr\" (UID: \"a54cd33d-feef-42bb-a78b-7480634cb58d\") " pod="e2e-models-e2e-worker-w3-682999/premium-w3-682999-kserve-5f47cf9b8d-xqnnr"
Oct 30 20:51:41.633919 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:41.633906    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a54cd33d-feef-42bb-a78b-7480634cb58d-tls-certs\") pod \"premium-w3-682999-kserve-5f47cf9b8d-xqnnr\" (UID: \"a54cd33d-feef-42bb-a78b-7480634cb58d\") " pod="e2e-models-e2e-worker-w3-682999/premium-w3-682999-kserve-5f47cf9b8d-xqnnr"
Oct 30 20:51:41.634168 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:41.633953    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a54cd33d-feef-42bb-a78b-7480634cb58d-dshm\") pod \"premium-w3-682999-kserve-5f47cf9b8d-xqnnr\" (UID: \"a54cd33d-feef-42bb-a78b-7480634cb58d\") " pod="e2e-models-e2e-worker-w3-682999/premium-w3-682999-kserve-5f47cf9b8d-xqnnr"
Oct 30 20:51:41.634168 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:41.633996    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a54cd33d-feef-42bb-a78b-7480634cb58d-home\") pod \"premium-w3-682999-kserve-5f47cf9b8d-xqnnr\" (UID: \"a54cd33d-feef-42bb-a78b-7480634cb58d\") " pod="e2e-models-e2e-worker-w3-682999/premium-w3-682999-kserve-5f47cf9b8d-xqnnr"
Oct 30 20:51:41.634168 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:41.634067    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a54cd33d-feef-42bb-a78b-7480634cb58d-tmp-dir\") pod \"premium-w3-682999-kserve-5f47cf9b8d-xqnnr\" (UID: \"a54cd33d-feef-42bb-a78b-7480634cb58d\") " pod="e2e-models-e2e-worker-w3-682999/premium-w3-682999-kserve-5f47cf9b8d-xqnnr"
Oct 30 20:51:41.634168 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:41.634137    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a54cd33d-feef-42bb-a78b-7480634cb58d-model-cache\") pod \"premium-w3-682999-kserve-5f47cf9b8d-xqnnr\" (UID: \"a54cd33d-feef-42bb-a78b-7480634cb58d\") " pod="e2e-models-e2e-worker-w3-682999/premium-w3-682999-kserve-5f47cf9b8d-xqnnr"
Oct 30 20:51:41.634589 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:41.634565    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a54cd33d-feef-42bb-a78b-7480634cb58d-model-cache\") pod \"premium-w3-682999-kserve-5f47cf9b8d-xqnnr\" (UID: \"a54cd33d-feef-42bb-a78b-7480634cb58d\") " pod="e2e-models-e2e-worker-w3-682999/premium-w3-682999-kserve-5f47cf9b8d-xqnnr"
Oct 30 20:51:41.634863 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:41.634826    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a54cd33d-feef-42bb-a78b-7480634cb58d-home\") pod \"premium-w3-682999-kserve-5f47cf9b8d-xqnnr\" (UID: \"a54cd33d-feef-42bb-a78b-7480634cb58d\") " pod="e2e-models-e2e-worker-w3-682999/premium-w3-682999-kserve-5f47cf9b8d-xqnnr"
Oct 30 20:51:41.635603 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:41.635568    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a54cd33d-feef-42bb-a78b-7480634cb58d-tmp-dir\") pod \"premium-w3-682999-kserve-5f47cf9b8d-xqnnr\" (UID: \"a54cd33d-feef-42bb-a78b-7480634cb58d\") " pod="e2e-models-e2e-worker-w3-682999/premium-w3-682999-kserve-5f47cf9b8d-xqnnr"
Oct 30 20:51:41.638489 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:41.638465    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a54cd33d-feef-42bb-a78b-7480634cb58d-tls-certs\") pod \"premium-w3-682999-kserve-5f47cf9b8d-xqnnr\" (UID: \"a54cd33d-feef-42bb-a78b-7480634cb58d\") " pod="e2e-models-e2e-worker-w3-682999/premium-w3-682999-kserve-5f47cf9b8d-xqnnr"
Oct 30 20:51:41.638653 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:41.638555    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a54cd33d-feef-42bb-a78b-7480634cb58d-dshm\") pod \"premium-w3-682999-kserve-5f47cf9b8d-xqnnr\" (UID: \"a54cd33d-feef-42bb-a78b-7480634cb58d\") " pod="e2e-models-e2e-worker-w3-682999/premium-w3-682999-kserve-5f47cf9b8d-xqnnr"
Oct 30 20:51:41.691912 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:41.691881    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48ptx\" (UniqueName: \"kubernetes.io/projected/a54cd33d-feef-42bb-a78b-7480634cb58d-kube-api-access-48ptx\") pod \"premium-w3-682999-kserve-5f47cf9b8d-xqnnr\" (UID: \"a54cd33d-feef-42bb-a78b-7480634cb58d\") " pod="e2e-models-e2e-worker-w3-682999/premium-w3-682999-kserve-5f47cf9b8d-xqnnr"
Oct 30 20:51:41.796896 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:41.796775    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-682999/premium-w3-682999-kserve-5f47cf9b8d-xqnnr"
Oct 30 20:51:42.201978 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:42.201947    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-682999/premium-w3-682999-kserve-5f47cf9b8d-xqnnr"]
Oct 30 20:51:42.203531 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:51:42.203504    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda54cd33d_feef_42bb_a78b_7480634cb58d.slice/crio-4cde660a526396ed02a18238250340456e8c35a229fee6f1ccfa79b48c7de78f WatchSource:0}: Error finding container 4cde660a526396ed02a18238250340456e8c35a229fee6f1ccfa79b48c7de78f: Status 404 returned error can't find the container with id 4cde660a526396ed02a18238250340456e8c35a229fee6f1ccfa79b48c7de78f
Oct 30 20:51:43.150078 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:43.150041    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-682999/premium-w3-682999-kserve-5f47cf9b8d-xqnnr" event={"ID":"a54cd33d-feef-42bb-a78b-7480634cb58d","Type":"ContainerStarted","Data":"e3eee1e460f9157c14b58bcbd9c0e9f1f91a8d3395ca6267ddcbd0db5540458f"}
Oct 30 20:51:43.150484 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:43.150086    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-682999/premium-w3-682999-kserve-5f47cf9b8d-xqnnr" event={"ID":"a54cd33d-feef-42bb-a78b-7480634cb58d","Type":"ContainerStarted","Data":"4cde660a526396ed02a18238250340456e8c35a229fee6f1ccfa79b48c7de78f"}
Oct 30 20:51:43.268219 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:43.268160    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-682999/premium-w3-682999-kserve-5f47cf9b8d-xqnnr" podStartSLOduration=2.268145528 podStartE2EDuration="2.268145528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:51:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:51:43.265610915 +0000 UTC m=+786.330387314" watchObservedRunningTime="2026-10-30 20:51:43.268145528 +0000 UTC m=+786.332921976"
Oct 30 20:51:43.797384 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:43.797345    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-682999/premium-w3-682999-kserve-5f47cf9b8d-xqnnr"
Oct 30 20:51:51.797576 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:51.797540    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-682999/premium-w3-682999-kserve-5f47cf9b8d-xqnnr"
Oct 30 20:51:51.810234 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:51.810209    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-682999/premium-w3-682999-kserve-5f47cf9b8d-xqnnr"
Oct 30 20:51:52.215022 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:52.214993    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-682999/premium-w3-682999-kserve-5f47cf9b8d-xqnnr"
Oct 30 20:51:54.458235 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:51:54.458200    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 20:51:54.474431 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:54.474404    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-781058/simulator-w0-781058-kserve-55b954cb4b-4k7z6"]
Oct 30 20:51:54.478106 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:54.478088    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-781058/simulator-w0-781058-kserve-55b954cb4b-4k7z6"
Oct 30 20:51:54.578946 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:54.578910    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92c43343-2820-4097-b829-9aae96b670de-tmp-dir\") pod \"simulator-w0-781058-kserve-55b954cb4b-4k7z6\" (UID: \"92c43343-2820-4097-b829-9aae96b670de\") " pod="e2e-models-e2e-worker-w0-781058/simulator-w0-781058-kserve-55b954cb4b-4k7z6"
Oct 30 20:51:54.579115 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:54.578969    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92c43343-2820-4097-b829-9aae96b670de-model-cache\") pod \"simulator-w0-781058-kserve-55b954cb4b-4k7z6\" (UID: \"92c43343-2820-4097-b829-9aae96b670de\") " pod="e2e-models-e2e-worker-w0-781058/simulator-w0-781058-kserve-55b954cb4b-4k7z6"
Oct 30 20:51:54.579115 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:54.579041    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92c43343-2820-4097-b829-9aae96b670de-tls-certs\") pod \"simulator-w0-781058-kserve-55b954cb4b-4k7z6\" (UID: \"92c43343-2820-4097-b829-9aae96b670de\") " pod="e2e-models-e2e-worker-w0-781058/simulator-w0-781058-kserve-55b954cb4b-4k7z6"
Oct 30 20:51:54.579115 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:54.579071    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92c43343-2820-4097-b829-9aae96b670de-dshm\") pod \"simulator-w0-781058-kserve-55b954cb4b-4k7z6\" (UID: \"92c43343-2820-4097-b829-9aae96b670de\") " pod="e2e-models-e2e-worker-w0-781058/simulator-w0-781058-kserve-55b954cb4b-4k7z6"
Oct 30 20:51:54.579115 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:54.579087    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8kbk\" (UniqueName: \"kubernetes.io/projected/92c43343-2820-4097-b829-9aae96b670de-kube-api-access-h8kbk\") pod \"simulator-w0-781058-kserve-55b954cb4b-4k7z6\" (UID: \"92c43343-2820-4097-b829-9aae96b670de\") " pod="e2e-models-e2e-worker-w0-781058/simulator-w0-781058-kserve-55b954cb4b-4k7z6"
Oct 30 20:51:54.579115 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:54.579113    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92c43343-2820-4097-b829-9aae96b670de-home\") pod \"simulator-w0-781058-kserve-55b954cb4b-4k7z6\" (UID: \"92c43343-2820-4097-b829-9aae96b670de\") " pod="e2e-models-e2e-worker-w0-781058/simulator-w0-781058-kserve-55b954cb4b-4k7z6"
Oct 30 20:51:54.582343 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:54.582317    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-781058\"/\"openshift-service-ca.crt\""
Oct 30 20:51:54.582343 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:54.582318    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-781058\"/\"kube-root-ca.crt\""
Oct 30 20:51:54.582478 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:54.582350    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-781058\"/\"default-dockercfg-ws5l2\""
Oct 30 20:51:54.582523 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:54.582512    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-781058\"/\"simulator-w0-781058-kserve-self-signed-certs\""
Oct 30 20:51:54.584819 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:54.584782    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-781058/simulator-w0-781058-kserve-55b954cb4b-4k7z6"]
Oct 30 20:51:54.680714 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:54.680682    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92c43343-2820-4097-b829-9aae96b670de-model-cache\") pod \"simulator-w0-781058-kserve-55b954cb4b-4k7z6\" (UID: \"92c43343-2820-4097-b829-9aae96b670de\") " pod="e2e-models-e2e-worker-w0-781058/simulator-w0-781058-kserve-55b954cb4b-4k7z6"
Oct 30 20:51:54.680714 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:54.680721    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92c43343-2820-4097-b829-9aae96b670de-tls-certs\") pod \"simulator-w0-781058-kserve-55b954cb4b-4k7z6\" (UID: \"92c43343-2820-4097-b829-9aae96b670de\") " pod="e2e-models-e2e-worker-w0-781058/simulator-w0-781058-kserve-55b954cb4b-4k7z6"
Oct 30 20:51:54.681015 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:54.680738    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92c43343-2820-4097-b829-9aae96b670de-dshm\") pod \"simulator-w0-781058-kserve-55b954cb4b-4k7z6\" (UID: \"92c43343-2820-4097-b829-9aae96b670de\") " pod="e2e-models-e2e-worker-w0-781058/simulator-w0-781058-kserve-55b954cb4b-4k7z6"
Oct 30 20:51:54.681015 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:54.680754    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8kbk\" (UniqueName: \"kubernetes.io/projected/92c43343-2820-4097-b829-9aae96b670de-kube-api-access-h8kbk\") pod \"simulator-w0-781058-kserve-55b954cb4b-4k7z6\" (UID: \"92c43343-2820-4097-b829-9aae96b670de\") " pod="e2e-models-e2e-worker-w0-781058/simulator-w0-781058-kserve-55b954cb4b-4k7z6"
Oct 30 20:51:54.681015 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:54.680840    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92c43343-2820-4097-b829-9aae96b670de-home\") pod \"simulator-w0-781058-kserve-55b954cb4b-4k7z6\" (UID: \"92c43343-2820-4097-b829-9aae96b670de\") " pod="e2e-models-e2e-worker-w0-781058/simulator-w0-781058-kserve-55b954cb4b-4k7z6"
Oct 30 20:51:54.681152 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:54.681032    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92c43343-2820-4097-b829-9aae96b670de-tmp-dir\") pod \"simulator-w0-781058-kserve-55b954cb4b-4k7z6\" (UID: \"92c43343-2820-4097-b829-9aae96b670de\") " pod="e2e-models-e2e-worker-w0-781058/simulator-w0-781058-kserve-55b954cb4b-4k7z6"
Oct 30 20:51:54.681207 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:54.681183    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92c43343-2820-4097-b829-9aae96b670de-model-cache\") pod \"simulator-w0-781058-kserve-55b954cb4b-4k7z6\" (UID: \"92c43343-2820-4097-b829-9aae96b670de\") " pod="e2e-models-e2e-worker-w0-781058/simulator-w0-781058-kserve-55b954cb4b-4k7z6"
Oct 30 20:51:54.681276 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:54.681251    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92c43343-2820-4097-b829-9aae96b670de-home\") pod \"simulator-w0-781058-kserve-55b954cb4b-4k7z6\" (UID: \"92c43343-2820-4097-b829-9aae96b670de\") " pod="e2e-models-e2e-worker-w0-781058/simulator-w0-781058-kserve-55b954cb4b-4k7z6"
Oct 30 20:51:54.681385 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:54.681366    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92c43343-2820-4097-b829-9aae96b670de-tmp-dir\") pod \"simulator-w0-781058-kserve-55b954cb4b-4k7z6\" (UID: \"92c43343-2820-4097-b829-9aae96b670de\") " pod="e2e-models-e2e-worker-w0-781058/simulator-w0-781058-kserve-55b954cb4b-4k7z6"
Oct 30 20:51:54.683289 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:54.683269    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92c43343-2820-4097-b829-9aae96b670de-dshm\") pod \"simulator-w0-781058-kserve-55b954cb4b-4k7z6\" (UID: \"92c43343-2820-4097-b829-9aae96b670de\") " pod="e2e-models-e2e-worker-w0-781058/simulator-w0-781058-kserve-55b954cb4b-4k7z6"
Oct 30 20:51:54.683497 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:54.683483    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92c43343-2820-4097-b829-9aae96b670de-tls-certs\") pod \"simulator-w0-781058-kserve-55b954cb4b-4k7z6\" (UID: \"92c43343-2820-4097-b829-9aae96b670de\") " pod="e2e-models-e2e-worker-w0-781058/simulator-w0-781058-kserve-55b954cb4b-4k7z6"
Oct 30 20:51:54.739719 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:54.739643    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8kbk\" (UniqueName: \"kubernetes.io/projected/92c43343-2820-4097-b829-9aae96b670de-kube-api-access-h8kbk\") pod \"simulator-w0-781058-kserve-55b954cb4b-4k7z6\" (UID: \"92c43343-2820-4097-b829-9aae96b670de\") " pod="e2e-models-e2e-worker-w0-781058/simulator-w0-781058-kserve-55b954cb4b-4k7z6"
Oct 30 20:51:54.790474 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:54.790424    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-781058/simulator-w0-781058-kserve-55b954cb4b-4k7z6"
Oct 30 20:51:55.176455 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:55.176424    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-781058/simulator-w0-781058-kserve-55b954cb4b-4k7z6"]
Oct 30 20:51:55.178543 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:51:55.178516    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod92c43343_2820_4097_b829_9aae96b670de.slice/crio-19eacedf1b007cfbcce22b68742b9023b0f3bfca56abe034c6a377e6127a72ae WatchSource:0}: Error finding container 19eacedf1b007cfbcce22b68742b9023b0f3bfca56abe034c6a377e6127a72ae: Status 404 returned error can't find the container with id 19eacedf1b007cfbcce22b68742b9023b0f3bfca56abe034c6a377e6127a72ae
Oct 30 20:51:55.227646 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:55.227617    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-781058/simulator-w0-781058-kserve-55b954cb4b-4k7z6" event={"ID":"92c43343-2820-4097-b829-9aae96b670de","Type":"ContainerStarted","Data":"19eacedf1b007cfbcce22b68742b9023b0f3bfca56abe034c6a377e6127a72ae"}
Oct 30 20:51:56.236233 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:56.236197    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-781058/simulator-w0-781058-kserve-55b954cb4b-4k7z6" event={"ID":"92c43343-2820-4097-b829-9aae96b670de","Type":"ContainerStarted","Data":"50a55ff6235c501e4a11d2ce850bce47b190ac21bd7f8c8e0ddab4e97b439c6c"}
Oct 30 20:51:56.353641 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:56.353586    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-781058/simulator-w0-781058-kserve-55b954cb4b-4k7z6" podStartSLOduration=2.35357176 podStartE2EDuration="2.35357176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:51:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:51:56.352029523 +0000 UTC m=+799.416805924" watchObservedRunningTime="2026-10-30 20:51:56.35357176 +0000 UTC m=+799.418348158"
Oct 30 20:51:56.791296 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:51:56.791269    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-781058/simulator-w0-781058-kserve-55b954cb4b-4k7z6"
Oct 30 20:52:04.790954 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:52:04.790906    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-781058/simulator-w0-781058-kserve-55b954cb4b-4k7z6"
Oct 30 20:52:04.803662 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:52:04.803639    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-781058/simulator-w0-781058-kserve-55b954cb4b-4k7z6"
Oct 30 20:52:05.298552 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:52:05.298517    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-781058/simulator-w0-781058-kserve-55b954cb4b-4k7z6"
Oct 30 20:52:17.383251 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:52:17.383169    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-2b425d/premium-w5-2b425d-kserve-78756b6c5b-9x5xb"]
Oct 30 20:52:17.388721 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:52:17.388699    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2b425d/premium-w5-2b425d-kserve-78756b6c5b-9x5xb"
Oct 30 20:52:17.445728 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:52:17.445474    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-2b425d/premium-w5-2b425d-kserve-78756b6c5b-9x5xb"]
Oct 30 20:52:17.489569 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:52:17.489546    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-2b425d\"/\"premium-w5-2b425d-kserve-self-signed-certs\""
Oct 30 20:52:17.527176 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:52:17.527139    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abf7b512-50e5-4400-9ca3-9ffccaead3d7-home\") pod \"premium-w5-2b425d-kserve-78756b6c5b-9x5xb\" (UID: \"abf7b512-50e5-4400-9ca3-9ffccaead3d7\") " pod="e2e-models-e2e-worker-w5-2b425d/premium-w5-2b425d-kserve-78756b6c5b-9x5xb"
Oct 30 20:52:17.527176 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:52:17.527188    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abf7b512-50e5-4400-9ca3-9ffccaead3d7-dshm\") pod \"premium-w5-2b425d-kserve-78756b6c5b-9x5xb\" (UID: \"abf7b512-50e5-4400-9ca3-9ffccaead3d7\") " pod="e2e-models-e2e-worker-w5-2b425d/premium-w5-2b425d-kserve-78756b6c5b-9x5xb"
Oct 30 20:52:17.527438 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:52:17.527310    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nh49\" (UniqueName: \"kubernetes.io/projected/abf7b512-50e5-4400-9ca3-9ffccaead3d7-kube-api-access-8nh49\") pod \"premium-w5-2b425d-kserve-78756b6c5b-9x5xb\" (UID: \"abf7b512-50e5-4400-9ca3-9ffccaead3d7\") " pod="e2e-models-e2e-worker-w5-2b425d/premium-w5-2b425d-kserve-78756b6c5b-9x5xb"
Oct 30 20:52:17.527438 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:52:17.527392    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abf7b512-50e5-4400-9ca3-9ffccaead3d7-tls-certs\") pod \"premium-w5-2b425d-kserve-78756b6c5b-9x5xb\" (UID: \"abf7b512-50e5-4400-9ca3-9ffccaead3d7\") " pod="e2e-models-e2e-worker-w5-2b425d/premium-w5-2b425d-kserve-78756b6c5b-9x5xb"
Oct 30 20:52:17.527620 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:52:17.527568    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abf7b512-50e5-4400-9ca3-9ffccaead3d7-model-cache\") pod \"premium-w5-2b425d-kserve-78756b6c5b-9x5xb\" (UID: \"abf7b512-50e5-4400-9ca3-9ffccaead3d7\") " pod="e2e-models-e2e-worker-w5-2b425d/premium-w5-2b425d-kserve-78756b6c5b-9x5xb"
Oct 30 20:52:17.527620 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:52:17.527608    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abf7b512-50e5-4400-9ca3-9ffccaead3d7-tmp-dir\") pod \"premium-w5-2b425d-kserve-78756b6c5b-9x5xb\" (UID: \"abf7b512-50e5-4400-9ca3-9ffccaead3d7\") " pod="e2e-models-e2e-worker-w5-2b425d/premium-w5-2b425d-kserve-78756b6c5b-9x5xb"
Oct 30 20:52:17.628519 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:52:17.628481    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abf7b512-50e5-4400-9ca3-9ffccaead3d7-model-cache\") pod \"premium-w5-2b425d-kserve-78756b6c5b-9x5xb\" (UID: \"abf7b512-50e5-4400-9ca3-9ffccaead3d7\") " pod="e2e-models-e2e-worker-w5-2b425d/premium-w5-2b425d-kserve-78756b6c5b-9x5xb"
Oct 30 20:52:17.628519 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:52:17.628523    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abf7b512-50e5-4400-9ca3-9ffccaead3d7-tmp-dir\") pod \"premium-w5-2b425d-kserve-78756b6c5b-9x5xb\" (UID: \"abf7b512-50e5-4400-9ca3-9ffccaead3d7\") " pod="e2e-models-e2e-worker-w5-2b425d/premium-w5-2b425d-kserve-78756b6c5b-9x5xb"
Oct 30 20:52:17.628772 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:52:17.628570    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abf7b512-50e5-4400-9ca3-9ffccaead3d7-home\") pod \"premium-w5-2b425d-kserve-78756b6c5b-9x5xb\" (UID: \"abf7b512-50e5-4400-9ca3-9ffccaead3d7\") " pod="e2e-models-e2e-worker-w5-2b425d/premium-w5-2b425d-kserve-78756b6c5b-9x5xb"
Oct 30 20:52:17.628772 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:52:17.628588    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abf7b512-50e5-4400-9ca3-9ffccaead3d7-dshm\") pod \"premium-w5-2b425d-kserve-78756b6c5b-9x5xb\" (UID: \"abf7b512-50e5-4400-9ca3-9ffccaead3d7\") " pod="e2e-models-e2e-worker-w5-2b425d/premium-w5-2b425d-kserve-78756b6c5b-9x5xb"
Oct 30 20:52:17.628772 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:52:17.628611    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nh49\" (UniqueName: \"kubernetes.io/projected/abf7b512-50e5-4400-9ca3-9ffccaead3d7-kube-api-access-8nh49\") pod \"premium-w5-2b425d-kserve-78756b6c5b-9x5xb\" (UID: \"abf7b512-50e5-4400-9ca3-9ffccaead3d7\") " pod="e2e-models-e2e-worker-w5-2b425d/premium-w5-2b425d-kserve-78756b6c5b-9x5xb"
Oct 30 20:52:17.628772 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:52:17.628668    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abf7b512-50e5-4400-9ca3-9ffccaead3d7-tls-certs\") pod \"premium-w5-2b425d-kserve-78756b6c5b-9x5xb\" (UID: \"abf7b512-50e5-4400-9ca3-9ffccaead3d7\") " pod="e2e-models-e2e-worker-w5-2b425d/premium-w5-2b425d-kserve-78756b6c5b-9x5xb"
Oct 30 20:52:17.629074 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:52:17.629044    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abf7b512-50e5-4400-9ca3-9ffccaead3d7-model-cache\") pod \"premium-w5-2b425d-kserve-78756b6c5b-9x5xb\" (UID: \"abf7b512-50e5-4400-9ca3-9ffccaead3d7\") " pod="e2e-models-e2e-worker-w5-2b425d/premium-w5-2b425d-kserve-78756b6c5b-9x5xb"
Oct 30 20:52:17.629138 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:52:17.629074    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abf7b512-50e5-4400-9ca3-9ffccaead3d7-home\") pod \"premium-w5-2b425d-kserve-78756b6c5b-9x5xb\" (UID: \"abf7b512-50e5-4400-9ca3-9ffccaead3d7\") " pod="e2e-models-e2e-worker-w5-2b425d/premium-w5-2b425d-kserve-78756b6c5b-9x5xb"
Oct 30 20:52:17.629138 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:52:17.629130    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abf7b512-50e5-4400-9ca3-9ffccaead3d7-tmp-dir\") pod \"premium-w5-2b425d-kserve-78756b6c5b-9x5xb\" (UID: \"abf7b512-50e5-4400-9ca3-9ffccaead3d7\") " pod="e2e-models-e2e-worker-w5-2b425d/premium-w5-2b425d-kserve-78756b6c5b-9x5xb"
Oct 30 20:52:17.631157 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:52:17.631135    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abf7b512-50e5-4400-9ca3-9ffccaead3d7-dshm\") pod \"premium-w5-2b425d-kserve-78756b6c5b-9x5xb\" (UID: \"abf7b512-50e5-4400-9ca3-9ffccaead3d7\") " pod="e2e-models-e2e-worker-w5-2b425d/premium-w5-2b425d-kserve-78756b6c5b-9x5xb"
Oct 30 20:52:17.631326 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:52:17.631310    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abf7b512-50e5-4400-9ca3-9ffccaead3d7-tls-certs\") pod \"premium-w5-2b425d-kserve-78756b6c5b-9x5xb\" (UID: \"abf7b512-50e5-4400-9ca3-9ffccaead3d7\") " pod="e2e-models-e2e-worker-w5-2b425d/premium-w5-2b425d-kserve-78756b6c5b-9x5xb"
Oct 30 20:52:17.686070 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:52:17.686045    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nh49\" (UniqueName: \"kubernetes.io/projected/abf7b512-50e5-4400-9ca3-9ffccaead3d7-kube-api-access-8nh49\") pod \"premium-w5-2b425d-kserve-78756b6c5b-9x5xb\" (UID: \"abf7b512-50e5-4400-9ca3-9ffccaead3d7\") " pod="e2e-models-e2e-worker-w5-2b425d/premium-w5-2b425d-kserve-78756b6c5b-9x5xb"
Oct 30 20:52:17.727689 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:52:17.727659    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2b425d/premium-w5-2b425d-kserve-78756b6c5b-9x5xb"
Oct 30 20:52:18.118570 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:52:18.118541    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-2b425d/premium-w5-2b425d-kserve-78756b6c5b-9x5xb"]
Oct 30 20:52:18.120344 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:52:18.120313    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podabf7b512_50e5_4400_9ca3_9ffccaead3d7.slice/crio-1571665678e05b987149fa7dd3b613cb6da99b457ea6c348d1fc6426f9ecc778 WatchSource:0}: Error finding container 1571665678e05b987149fa7dd3b613cb6da99b457ea6c348d1fc6426f9ecc778: Status 404 returned error can't find the container with id 1571665678e05b987149fa7dd3b613cb6da99b457ea6c348d1fc6426f9ecc778
Oct 30 20:52:18.364651 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:52:18.364563    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2b425d/premium-w5-2b425d-kserve-78756b6c5b-9x5xb" event={"ID":"abf7b512-50e5-4400-9ca3-9ffccaead3d7","Type":"ContainerStarted","Data":"a111f87edb57dccdaa4d3907e474c9625246c8f320d9a38a4446e6807dd1f967"}
Oct 30 20:52:18.364651 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:52:18.364610    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2b425d/premium-w5-2b425d-kserve-78756b6c5b-9x5xb" event={"ID":"abf7b512-50e5-4400-9ca3-9ffccaead3d7","Type":"ContainerStarted","Data":"1571665678e05b987149fa7dd3b613cb6da99b457ea6c348d1fc6426f9ecc778"}
Oct 30 20:52:18.481151 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:52:18.481098    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-2b425d/premium-w5-2b425d-kserve-78756b6c5b-9x5xb" podStartSLOduration=1.481082846 podStartE2EDuration="1.481082846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:52:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:52:18.478726102 +0000 UTC m=+821.543502500" watchObservedRunningTime="2026-10-30 20:52:18.481082846 +0000 UTC m=+821.545859244"
Oct 30 20:52:18.728849 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:52:18.728776    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2b425d/premium-w5-2b425d-kserve-78756b6c5b-9x5xb"
Oct 30 20:52:24.461194 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:52:24.461151    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 20:52:27.728781 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:52:27.728744    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-2b425d/premium-w5-2b425d-kserve-78756b6c5b-9x5xb"
Oct 30 20:52:27.741668 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:52:27.741638    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-2b425d/premium-w5-2b425d-kserve-78756b6c5b-9x5xb"
Oct 30 20:52:28.436359 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:52:28.436333    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2b425d/premium-w5-2b425d-kserve-78756b6c5b-9x5xb"
Oct 30 20:52:43.431650 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:52:43.431616    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-f9f277/premium-w1-f9f277-kserve-5dbfd4d496-72w4n"]
Oct 30 20:52:43.435732 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:52:43.435715    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f9f277/premium-w1-f9f277-kserve-5dbfd4d496-72w4n"
Oct 30 20:52:43.522000 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:52:43.521967    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cc1f811-c149-4457-80f1-1a55621d73b4-model-cache\") pod \"premium-w1-f9f277-kserve-5dbfd4d496-72w4n\" (UID: \"1cc1f811-c149-4457-80f1-1a55621d73b4\") " pod="e2e-models-e2e-worker-w1-f9f277/premium-w1-f9f277-kserve-5dbfd4d496-72w4n"
Oct 30 20:52:43.522000 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:52:43.522002    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cc1f811-c149-4457-80f1-1a55621d73b4-tls-certs\") pod \"premium-w1-f9f277-kserve-5dbfd4d496-72w4n\" (UID: \"1cc1f811-c149-4457-80f1-1a55621d73b4\") " pod="e2e-models-e2e-worker-w1-f9f277/premium-w1-f9f277-kserve-5dbfd4d496-72w4n"
Oct 30 20:52:43.522211 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:52:43.522113    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cc1f811-c149-4457-80f1-1a55621d73b4-home\") pod \"premium-w1-f9f277-kserve-5dbfd4d496-72w4n\" (UID: \"1cc1f811-c149-4457-80f1-1a55621d73b4\") " pod="e2e-models-e2e-worker-w1-f9f277/premium-w1-f9f277-kserve-5dbfd4d496-72w4n"
Oct 30 20:52:43.522211 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:52:43.522140    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cc1f811-c149-4457-80f1-1a55621d73b4-tmp-dir\") pod \"premium-w1-f9f277-kserve-5dbfd4d496-72w4n\" (UID: \"1cc1f811-c149-4457-80f1-1a55621d73b4\") " pod="e2e-models-e2e-worker-w1-f9f277/premium-w1-f9f277-kserve-5dbfd4d496-72w4n"
Oct 30 20:52:43.522211 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:52:43.522178    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cc1f811-c149-4457-80f1-1a55621d73b4-dshm\") pod \"premium-w1-f9f277-kserve-5dbfd4d496-72w4n\" (UID: \"1cc1f811-c149-4457-80f1-1a55621d73b4\") " pod="e2e-models-e2e-worker-w1-f9f277/premium-w1-f9f277-kserve-5dbfd4d496-72w4n"
Oct 30 20:52:43.522310 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:52:43.522265    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txhj7\" (UniqueName: \"kubernetes.io/projected/1cc1f811-c149-4457-80f1-1a55621d73b4-kube-api-access-txhj7\") pod \"premium-w1-f9f277-kserve-5dbfd4d496-72w4n\" (UID: \"1cc1f811-c149-4457-80f1-1a55621d73b4\") " pod="e2e-models-e2e-worker-w1-f9f277/premium-w1-f9f277-kserve-5dbfd4d496-72w4n"
Oct 30 20:52:43.538450 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:52:43.538425    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-f9f277\"/\"openshift-service-ca.crt\""
Oct 30 20:52:43.538450 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:52:43.538446    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-f9f277\"/\"kube-root-ca.crt\""
Oct 30 20:52:43.538631 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:52:43.538601    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-f9f277\"/\"default-dockercfg-xbx2p\""
Oct 30 20:52:43.538730 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:52:43.538710    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-f9f277\"/\"premium-w1-f9f277-kserve-self-signed-certs\""
Oct 30 20:52:43.541171 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:52:43.541148    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-f9f277/premium-w1-f9f277-kserve-5dbfd4d496-72w4n"]
Oct 30 20:52:43.623436 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:52:43.623401    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cc1f811-c149-4457-80f1-1a55621d73b4-model-cache\") pod \"premium-w1-f9f277-kserve-5dbfd4d496-72w4n\" (UID: \"1cc1f811-c149-4457-80f1-1a55621d73b4\") " pod="e2e-models-e2e-worker-w1-f9f277/premium-w1-f9f277-kserve-5dbfd4d496-72w4n"
Oct 30 20:52:43.623436 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:52:43.623440    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cc1f811-c149-4457-80f1-1a55621d73b4-tls-certs\") pod \"premium-w1-f9f277-kserve-5dbfd4d496-72w4n\" (UID: \"1cc1f811-c149-4457-80f1-1a55621d73b4\") " pod="e2e-models-e2e-worker-w1-f9f277/premium-w1-f9f277-kserve-5dbfd4d496-72w4n"
Oct 30 20:52:43.623692 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:52:43.623481    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cc1f811-c149-4457-80f1-1a55621d73b4-home\") pod \"premium-w1-f9f277-kserve-5dbfd4d496-72w4n\" (UID: \"1cc1f811-c149-4457-80f1-1a55621d73b4\") " pod="e2e-models-e2e-worker-w1-f9f277/premium-w1-f9f277-kserve-5dbfd4d496-72w4n"
Oct 30 20:52:43.623692 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:52:43.623497    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cc1f811-c149-4457-80f1-1a55621d73b4-tmp-dir\") pod \"premium-w1-f9f277-kserve-5dbfd4d496-72w4n\" (UID: \"1cc1f811-c149-4457-80f1-1a55621d73b4\") " pod="e2e-models-e2e-worker-w1-f9f277/premium-w1-f9f277-kserve-5dbfd4d496-72w4n"
Oct 30 20:52:43.623692 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:52:43.623529    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cc1f811-c149-4457-80f1-1a55621d73b4-dshm\") pod \"premium-w1-f9f277-kserve-5dbfd4d496-72w4n\" (UID: \"1cc1f811-c149-4457-80f1-1a55621d73b4\") " pod="e2e-models-e2e-worker-w1-f9f277/premium-w1-f9f277-kserve-5dbfd4d496-72w4n"
Oct 30 20:52:43.623692 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:52:43.623564    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txhj7\" (UniqueName: \"kubernetes.io/projected/1cc1f811-c149-4457-80f1-1a55621d73b4-kube-api-access-txhj7\") pod \"premium-w1-f9f277-kserve-5dbfd4d496-72w4n\" (UID: \"1cc1f811-c149-4457-80f1-1a55621d73b4\") " pod="e2e-models-e2e-worker-w1-f9f277/premium-w1-f9f277-kserve-5dbfd4d496-72w4n"
Oct 30 20:52:43.623919 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:52:43.623877    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cc1f811-c149-4457-80f1-1a55621d73b4-model-cache\") pod \"premium-w1-f9f277-kserve-5dbfd4d496-72w4n\" (UID: \"1cc1f811-c149-4457-80f1-1a55621d73b4\") " pod="e2e-models-e2e-worker-w1-f9f277/premium-w1-f9f277-kserve-5dbfd4d496-72w4n"
Oct 30 20:52:43.624025 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:52:43.624005    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cc1f811-c149-4457-80f1-1a55621d73b4-home\") pod \"premium-w1-f9f277-kserve-5dbfd4d496-72w4n\" (UID: \"1cc1f811-c149-4457-80f1-1a55621d73b4\") " pod="e2e-models-e2e-worker-w1-f9f277/premium-w1-f9f277-kserve-5dbfd4d496-72w4n"
Oct 30 20:52:43.624083 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:52:43.624030    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cc1f811-c149-4457-80f1-1a55621d73b4-tmp-dir\") pod \"premium-w1-f9f277-kserve-5dbfd4d496-72w4n\" (UID: \"1cc1f811-c149-4457-80f1-1a55621d73b4\") " pod="e2e-models-e2e-worker-w1-f9f277/premium-w1-f9f277-kserve-5dbfd4d496-72w4n"
Oct 30 20:52:43.625991 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:52:43.625970    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cc1f811-c149-4457-80f1-1a55621d73b4-dshm\") pod \"premium-w1-f9f277-kserve-5dbfd4d496-72w4n\" (UID: \"1cc1f811-c149-4457-80f1-1a55621d73b4\") " pod="e2e-models-e2e-worker-w1-f9f277/premium-w1-f9f277-kserve-5dbfd4d496-72w4n"
Oct 30 20:52:43.626151 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:52:43.626135    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cc1f811-c149-4457-80f1-1a55621d73b4-tls-certs\") pod \"premium-w1-f9f277-kserve-5dbfd4d496-72w4n\" (UID: \"1cc1f811-c149-4457-80f1-1a55621d73b4\") " pod="e2e-models-e2e-worker-w1-f9f277/premium-w1-f9f277-kserve-5dbfd4d496-72w4n"
Oct 30 20:52:43.681464 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:52:43.681425    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txhj7\" (UniqueName: \"kubernetes.io/projected/1cc1f811-c149-4457-80f1-1a55621d73b4-kube-api-access-txhj7\") pod \"premium-w1-f9f277-kserve-5dbfd4d496-72w4n\" (UID: \"1cc1f811-c149-4457-80f1-1a55621d73b4\") " pod="e2e-models-e2e-worker-w1-f9f277/premium-w1-f9f277-kserve-5dbfd4d496-72w4n"
Oct 30 20:52:43.748011 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:52:43.747928    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f9f277/premium-w1-f9f277-kserve-5dbfd4d496-72w4n"
Oct 30 20:52:44.147045 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:52:44.147022    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-f9f277/premium-w1-f9f277-kserve-5dbfd4d496-72w4n"]
Oct 30 20:52:44.148624 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:52:44.148599    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1cc1f811_c149_4457_80f1_1a55621d73b4.slice/crio-87c2ef76541bea66006ea0c4c0146177cf1fb3cb0584816acea452283471d9dd WatchSource:0}: Error finding container 87c2ef76541bea66006ea0c4c0146177cf1fb3cb0584816acea452283471d9dd: Status 404 returned error can't find the container with id 87c2ef76541bea66006ea0c4c0146177cf1fb3cb0584816acea452283471d9dd
Oct 30 20:52:44.522194 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:52:44.522164    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f9f277/premium-w1-f9f277-kserve-5dbfd4d496-72w4n" event={"ID":"1cc1f811-c149-4457-80f1-1a55621d73b4","Type":"ContainerStarted","Data":"1f11d3a069df800c1ba3e46c988c04320df7c73816f7a56aadc02411254a1afc"}
Oct 30 20:52:44.522194 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:52:44.522202    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f9f277/premium-w1-f9f277-kserve-5dbfd4d496-72w4n" event={"ID":"1cc1f811-c149-4457-80f1-1a55621d73b4","Type":"ContainerStarted","Data":"87c2ef76541bea66006ea0c4c0146177cf1fb3cb0584816acea452283471d9dd"}
Oct 30 20:52:44.642377 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:52:44.642325    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-f9f277/premium-w1-f9f277-kserve-5dbfd4d496-72w4n" podStartSLOduration=1.642308058 podStartE2EDuration="1.642308058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:52:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:52:44.641858158 +0000 UTC m=+847.706634548" watchObservedRunningTime="2026-10-30 20:52:44.642308058 +0000 UTC m=+847.707084493"
Oct 30 20:52:44.748957 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:52:44.748926    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f9f277/premium-w1-f9f277-kserve-5dbfd4d496-72w4n"
Oct 30 20:52:53.749176 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:52:53.749144    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-f9f277/premium-w1-f9f277-kserve-5dbfd4d496-72w4n"
Oct 30 20:52:53.761901 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:52:53.761873    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-f9f277/premium-w1-f9f277-kserve-5dbfd4d496-72w4n"
Oct 30 20:52:54.458216 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:52:54.458180    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 20:52:54.592249 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:52:54.592216    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f9f277/premium-w1-f9f277-kserve-5dbfd4d496-72w4n"
Oct 30 20:53:23.371383 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:23.371350    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-781058/premium-w0-781058-kserve-7f5c8964c9-z5fmk"]
Oct 30 20:53:23.375228 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:23.375208    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-781058/premium-w0-781058-kserve-7f5c8964c9-z5fmk"
Oct 30 20:53:23.475344 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:23.475308    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57626f4f-844e-4ec9-ab89-359d015923af-home\") pod \"premium-w0-781058-kserve-7f5c8964c9-z5fmk\" (UID: \"57626f4f-844e-4ec9-ab89-359d015923af\") " pod="e2e-models-e2e-worker-w0-781058/premium-w0-781058-kserve-7f5c8964c9-z5fmk"
Oct 30 20:53:23.475344 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:23.475348    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57626f4f-844e-4ec9-ab89-359d015923af-model-cache\") pod \"premium-w0-781058-kserve-7f5c8964c9-z5fmk\" (UID: \"57626f4f-844e-4ec9-ab89-359d015923af\") " pod="e2e-models-e2e-worker-w0-781058/premium-w0-781058-kserve-7f5c8964c9-z5fmk"
Oct 30 20:53:23.475589 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:23.475368    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57626f4f-844e-4ec9-ab89-359d015923af-tls-certs\") pod \"premium-w0-781058-kserve-7f5c8964c9-z5fmk\" (UID: \"57626f4f-844e-4ec9-ab89-359d015923af\") " pod="e2e-models-e2e-worker-w0-781058/premium-w0-781058-kserve-7f5c8964c9-z5fmk"
Oct 30 20:53:23.475589 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:23.475541    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxdvh\" (UniqueName: \"kubernetes.io/projected/57626f4f-844e-4ec9-ab89-359d015923af-kube-api-access-rxdvh\") pod \"premium-w0-781058-kserve-7f5c8964c9-z5fmk\" (UID: \"57626f4f-844e-4ec9-ab89-359d015923af\") " pod="e2e-models-e2e-worker-w0-781058/premium-w0-781058-kserve-7f5c8964c9-z5fmk"
Oct 30 20:53:23.475740 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:23.475723    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57626f4f-844e-4ec9-ab89-359d015923af-tmp-dir\") pod \"premium-w0-781058-kserve-7f5c8964c9-z5fmk\" (UID: \"57626f4f-844e-4ec9-ab89-359d015923af\") " pod="e2e-models-e2e-worker-w0-781058/premium-w0-781058-kserve-7f5c8964c9-z5fmk"
Oct 30 20:53:23.475826 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:23.475757    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57626f4f-844e-4ec9-ab89-359d015923af-dshm\") pod \"premium-w0-781058-kserve-7f5c8964c9-z5fmk\" (UID: \"57626f4f-844e-4ec9-ab89-359d015923af\") " pod="e2e-models-e2e-worker-w0-781058/premium-w0-781058-kserve-7f5c8964c9-z5fmk"
Oct 30 20:53:23.481356 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:23.481326    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-781058\"/\"premium-w0-781058-kserve-self-signed-certs\""
Oct 30 20:53:23.483867 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:23.483772    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-781058/premium-w0-781058-kserve-7f5c8964c9-z5fmk"]
Oct 30 20:53:23.576984 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:23.576945    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57626f4f-844e-4ec9-ab89-359d015923af-tmp-dir\") pod \"premium-w0-781058-kserve-7f5c8964c9-z5fmk\" (UID: \"57626f4f-844e-4ec9-ab89-359d015923af\") " pod="e2e-models-e2e-worker-w0-781058/premium-w0-781058-kserve-7f5c8964c9-z5fmk"
Oct 30 20:53:23.577200 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:23.576996    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57626f4f-844e-4ec9-ab89-359d015923af-dshm\") pod \"premium-w0-781058-kserve-7f5c8964c9-z5fmk\" (UID: \"57626f4f-844e-4ec9-ab89-359d015923af\") " pod="e2e-models-e2e-worker-w0-781058/premium-w0-781058-kserve-7f5c8964c9-z5fmk"
Oct 30 20:53:23.577200 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:23.577068    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57626f4f-844e-4ec9-ab89-359d015923af-home\") pod \"premium-w0-781058-kserve-7f5c8964c9-z5fmk\" (UID: \"57626f4f-844e-4ec9-ab89-359d015923af\") " pod="e2e-models-e2e-worker-w0-781058/premium-w0-781058-kserve-7f5c8964c9-z5fmk"
Oct 30 20:53:23.577200 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:23.577088    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57626f4f-844e-4ec9-ab89-359d015923af-model-cache\") pod \"premium-w0-781058-kserve-7f5c8964c9-z5fmk\" (UID: \"57626f4f-844e-4ec9-ab89-359d015923af\") " pod="e2e-models-e2e-worker-w0-781058/premium-w0-781058-kserve-7f5c8964c9-z5fmk"
Oct 30 20:53:23.577200 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:23.577110    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57626f4f-844e-4ec9-ab89-359d015923af-tls-certs\") pod \"premium-w0-781058-kserve-7f5c8964c9-z5fmk\" (UID: \"57626f4f-844e-4ec9-ab89-359d015923af\") " pod="e2e-models-e2e-worker-w0-781058/premium-w0-781058-kserve-7f5c8964c9-z5fmk"
Oct 30 20:53:23.577200 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:23.577156    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxdvh\" (UniqueName: \"kubernetes.io/projected/57626f4f-844e-4ec9-ab89-359d015923af-kube-api-access-rxdvh\") pod \"premium-w0-781058-kserve-7f5c8964c9-z5fmk\" (UID: \"57626f4f-844e-4ec9-ab89-359d015923af\") " pod="e2e-models-e2e-worker-w0-781058/premium-w0-781058-kserve-7f5c8964c9-z5fmk"
Oct 30 20:53:23.577463 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:23.577435    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57626f4f-844e-4ec9-ab89-359d015923af-tmp-dir\") pod \"premium-w0-781058-kserve-7f5c8964c9-z5fmk\" (UID: \"57626f4f-844e-4ec9-ab89-359d015923af\") " pod="e2e-models-e2e-worker-w0-781058/premium-w0-781058-kserve-7f5c8964c9-z5fmk"
Oct 30 20:53:23.577463 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:23.577444    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57626f4f-844e-4ec9-ab89-359d015923af-model-cache\") pod \"premium-w0-781058-kserve-7f5c8964c9-z5fmk\" (UID: \"57626f4f-844e-4ec9-ab89-359d015923af\") " pod="e2e-models-e2e-worker-w0-781058/premium-w0-781058-kserve-7f5c8964c9-z5fmk"
Oct 30 20:53:23.577564 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:23.577501    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57626f4f-844e-4ec9-ab89-359d015923af-home\") pod \"premium-w0-781058-kserve-7f5c8964c9-z5fmk\" (UID: \"57626f4f-844e-4ec9-ab89-359d015923af\") " pod="e2e-models-e2e-worker-w0-781058/premium-w0-781058-kserve-7f5c8964c9-z5fmk"
Oct 30 20:53:23.579394 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:23.579376    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57626f4f-844e-4ec9-ab89-359d015923af-dshm\") pod \"premium-w0-781058-kserve-7f5c8964c9-z5fmk\" (UID: \"57626f4f-844e-4ec9-ab89-359d015923af\") " pod="e2e-models-e2e-worker-w0-781058/premium-w0-781058-kserve-7f5c8964c9-z5fmk"
Oct 30 20:53:23.579818 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:23.579779    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57626f4f-844e-4ec9-ab89-359d015923af-tls-certs\") pod \"premium-w0-781058-kserve-7f5c8964c9-z5fmk\" (UID: \"57626f4f-844e-4ec9-ab89-359d015923af\") " pod="e2e-models-e2e-worker-w0-781058/premium-w0-781058-kserve-7f5c8964c9-z5fmk"
Oct 30 20:53:23.635452 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:23.635376    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxdvh\" (UniqueName: \"kubernetes.io/projected/57626f4f-844e-4ec9-ab89-359d015923af-kube-api-access-rxdvh\") pod \"premium-w0-781058-kserve-7f5c8964c9-z5fmk\" (UID: \"57626f4f-844e-4ec9-ab89-359d015923af\") " pod="e2e-models-e2e-worker-w0-781058/premium-w0-781058-kserve-7f5c8964c9-z5fmk"
Oct 30 20:53:23.687937 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:23.687905    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-781058/premium-w0-781058-kserve-7f5c8964c9-z5fmk"
Oct 30 20:53:24.064539 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:24.064512    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-781058/premium-w0-781058-kserve-7f5c8964c9-z5fmk"]
Oct 30 20:53:24.065896 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:53:24.065866    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod57626f4f_844e_4ec9_ab89_359d015923af.slice/crio-6b16125500de29985750c17c820341af752b3999a467dc864fb84845849fcfeb WatchSource:0}: Error finding container 6b16125500de29985750c17c820341af752b3999a467dc864fb84845849fcfeb: Status 404 returned error can't find the container with id 6b16125500de29985750c17c820341af752b3999a467dc864fb84845849fcfeb
Oct 30 20:53:24.459072 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:53:24.459040    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 20:53:24.780978 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:24.780883    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-781058/premium-w0-781058-kserve-7f5c8964c9-z5fmk" event={"ID":"57626f4f-844e-4ec9-ab89-359d015923af","Type":"ContainerStarted","Data":"c0ef753417cc6374a5d6ff21837df58e88d0c1db900d40b9a68655d8266088b9"}
Oct 30 20:53:24.780978 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:24.780926    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-781058/premium-w0-781058-kserve-7f5c8964c9-z5fmk" event={"ID":"57626f4f-844e-4ec9-ab89-359d015923af","Type":"ContainerStarted","Data":"6b16125500de29985750c17c820341af752b3999a467dc864fb84845849fcfeb"}
Oct 30 20:53:24.894146 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:24.894095    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-781058/premium-w0-781058-kserve-7f5c8964c9-z5fmk" podStartSLOduration=1.894080526 podStartE2EDuration="1.894080526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:53:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:53:24.892676829 +0000 UTC m=+887.957453228" watchObservedRunningTime="2026-10-30 20:53:24.894080526 +0000 UTC m=+887.958856923"
Oct 30 20:53:25.688313 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:25.688281    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-781058/premium-w0-781058-kserve-7f5c8964c9-z5fmk"
Oct 30 20:53:28.261328 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:28.261299    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd0c8b7d-wcdtn"]
Oct 30 20:53:28.265086 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:28.265068    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0c8b7d-wcdtn"
Oct 30 20:53:28.368323 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:28.368296    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:53:28.368323 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:28.368297    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:53:28.368323 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:28.368297    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 20:53:28.368323 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:28.368309    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:53:28.370964 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:28.370944    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd0c8b7d-wcdtn"]
Oct 30 20:53:28.446990 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:28.446953    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed2d8620-1d01-48d1-9491-3bacfd9214a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd0c8b7d-wcdtn\" (UID: \"ed2d8620-1d01-48d1-9491-3bacfd9214a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0c8b7d-wcdtn"
Oct 30 20:53:28.447172 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:28.447007    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed2d8620-1d01-48d1-9491-3bacfd9214a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd0c8b7d-wcdtn\" (UID: \"ed2d8620-1d01-48d1-9491-3bacfd9214a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0c8b7d-wcdtn"
Oct 30 20:53:28.447172 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:28.447032    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnnjf\" (UniqueName: \"kubernetes.io/projected/ed2d8620-1d01-48d1-9491-3bacfd9214a3-kube-api-access-bnnjf\") pod \"maas-api-revoke-sub-cd0c8b7d-wcdtn\" (UID: \"ed2d8620-1d01-48d1-9491-3bacfd9214a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0c8b7d-wcdtn"
Oct 30 20:53:28.548726 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:28.548631    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed2d8620-1d01-48d1-9491-3bacfd9214a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd0c8b7d-wcdtn\" (UID: \"ed2d8620-1d01-48d1-9491-3bacfd9214a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0c8b7d-wcdtn"
Oct 30 20:53:28.548726 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:28.548690    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed2d8620-1d01-48d1-9491-3bacfd9214a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd0c8b7d-wcdtn\" (UID: \"ed2d8620-1d01-48d1-9491-3bacfd9214a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0c8b7d-wcdtn"
Oct 30 20:53:28.548726 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:28.548711    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnnjf\" (UniqueName: \"kubernetes.io/projected/ed2d8620-1d01-48d1-9491-3bacfd9214a3-kube-api-access-bnnjf\") pod \"maas-api-revoke-sub-cd0c8b7d-wcdtn\" (UID: \"ed2d8620-1d01-48d1-9491-3bacfd9214a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0c8b7d-wcdtn"
Oct 30 20:53:28.549483 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:28.549455    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed2d8620-1d01-48d1-9491-3bacfd9214a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd0c8b7d-wcdtn\" (UID: \"ed2d8620-1d01-48d1-9491-3bacfd9214a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0c8b7d-wcdtn"
Oct 30 20:53:28.551217 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:28.551193    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed2d8620-1d01-48d1-9491-3bacfd9214a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd0c8b7d-wcdtn\" (UID: \"ed2d8620-1d01-48d1-9491-3bacfd9214a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0c8b7d-wcdtn"
Oct 30 20:53:28.607553 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:28.607521    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnnjf\" (UniqueName: \"kubernetes.io/projected/ed2d8620-1d01-48d1-9491-3bacfd9214a3-kube-api-access-bnnjf\") pod \"maas-api-revoke-sub-cd0c8b7d-wcdtn\" (UID: \"ed2d8620-1d01-48d1-9491-3bacfd9214a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0c8b7d-wcdtn"
Oct 30 20:53:28.879355 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:28.879256    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0c8b7d-wcdtn"
Oct 30 20:53:29.257123 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:29.257087    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd0c8b7d-wcdtn"]
Oct 30 20:53:29.259061 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:53:29.259036    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded2d8620_1d01_48d1_9491_3bacfd9214a3.slice/crio-0c7807b2d4e6ff7b44eeaa42b025394138f7a81880e530d102c27befb4ce4497 WatchSource:0}: Error finding container 0c7807b2d4e6ff7b44eeaa42b025394138f7a81880e530d102c27befb4ce4497: Status 404 returned error can't find the container with id 0c7807b2d4e6ff7b44eeaa42b025394138f7a81880e530d102c27befb4ce4497
Oct 30 20:53:29.261007 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:29.260991    5713 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:53:29.812000 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:29.811958    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0c8b7d-wcdtn" event={"ID":"ed2d8620-1d01-48d1-9491-3bacfd9214a3","Type":"ContainerStarted","Data":"0c7807b2d4e6ff7b44eeaa42b025394138f7a81880e530d102c27befb4ce4497"}
Oct 30 20:53:33.688210 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:33.688173    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-781058/premium-w0-781058-kserve-7f5c8964c9-z5fmk"
Oct 30 20:53:33.701280 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:33.701250    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-781058/premium-w0-781058-kserve-7f5c8964c9-z5fmk"
Oct 30 20:53:33.842957 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:33.842817    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0c8b7d-wcdtn" event={"ID":"ed2d8620-1d01-48d1-9491-3bacfd9214a3","Type":"ContainerStarted","Data":"3a8d769a8a55f38b6fbaac74ccc8ef912347a6865c0e492d322d83f640de4c9a"}
Oct 30 20:53:33.856896 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:33.856864    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-781058/premium-w0-781058-kserve-7f5c8964c9-z5fmk"
Oct 30 20:53:33.959293 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:33.959239    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0c8b7d-wcdtn" podStartSLOduration=1.9665545789999999 podStartE2EDuration="5.959225128s" totalImagesPullingTime="3.992670549s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:53:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:53:33.957031682 +0000 UTC m=+897.021808079" watchObservedRunningTime="2026-10-30 20:53:33.959225128 +0000 UTC m=+897.024001525"
Oct 30 20:53:34.849748 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:34.849714    5713 generic.go:356] "Generic (PLEG): container finished" podID="ed2d8620-1d01-48d1-9491-3bacfd9214a3" containerID="3a8d769a8a55f38b6fbaac74ccc8ef912347a6865c0e492d322d83f640de4c9a" exitCode=0
Oct 30 20:53:34.850158 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:34.849830    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0c8b7d-wcdtn" event={"ID":"ed2d8620-1d01-48d1-9491-3bacfd9214a3","Type":"ContainerDied","Data":"3a8d769a8a55f38b6fbaac74ccc8ef912347a6865c0e492d322d83f640de4c9a"}
Oct 30 20:53:36.001473 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:36.001449    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0c8b7d-wcdtn"
Oct 30 20:53:36.033071 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:36.033043    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed2d8620-1d01-48d1-9491-3bacfd9214a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed2d8620-1d01-48d1-9491-3bacfd9214a3-maas-api-openshift-ca\") pod \"ed2d8620-1d01-48d1-9491-3bacfd9214a3\" (UID: \"ed2d8620-1d01-48d1-9491-3bacfd9214a3\") "
Oct 30 20:53:36.033203 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:36.033085    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed2d8620-1d01-48d1-9491-3bacfd9214a3-kube-api-access-bnnjf\" (UniqueName: \"kubernetes.io/projected/ed2d8620-1d01-48d1-9491-3bacfd9214a3-kube-api-access-bnnjf\") pod \"ed2d8620-1d01-48d1-9491-3bacfd9214a3\" (UID: \"ed2d8620-1d01-48d1-9491-3bacfd9214a3\") "
Oct 30 20:53:36.033203 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:36.033133    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed2d8620-1d01-48d1-9491-3bacfd9214a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed2d8620-1d01-48d1-9491-3bacfd9214a3-maas-api-xks-ca\") pod \"ed2d8620-1d01-48d1-9491-3bacfd9214a3\" (UID: \"ed2d8620-1d01-48d1-9491-3bacfd9214a3\") "
Oct 30 20:53:36.033468 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:36.033441    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed2d8620-1d01-48d1-9491-3bacfd9214a3-maas-api-openshift-ca" pod "ed2d8620-1d01-48d1-9491-3bacfd9214a3" (UID: "ed2d8620-1d01-48d1-9491-3bacfd9214a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:53:36.035480 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:36.035454    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed2d8620-1d01-48d1-9491-3bacfd9214a3-maas-api-xks-ca" pod "ed2d8620-1d01-48d1-9491-3bacfd9214a3" (UID: "ed2d8620-1d01-48d1-9491-3bacfd9214a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:53:36.035480 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:36.035462    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed2d8620-1d01-48d1-9491-3bacfd9214a3-kube-api-access-bnnjf" pod "ed2d8620-1d01-48d1-9491-3bacfd9214a3" (UID: "ed2d8620-1d01-48d1-9491-3bacfd9214a3"). InnerVolumeSpecName "kube-api-access-bnnjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:53:36.135066 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:36.134900    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed2d8620-1d01-48d1-9491-3bacfd9214a3-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:36.135066 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:36.134936    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnnjf\" (UniqueName: \"kubernetes.io/projected/ed2d8620-1d01-48d1-9491-3bacfd9214a3-kube-api-access-bnnjf\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:36.135066 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:36.134949    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed2d8620-1d01-48d1-9491-3bacfd9214a3-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:36.868598 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:36.868569    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0c8b7d-wcdtn"
Oct 30 20:53:36.868828 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:36.868567    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0c8b7d-wcdtn" event={"ID":"ed2d8620-1d01-48d1-9491-3bacfd9214a3","Type":"ContainerDied","Data":"0c7807b2d4e6ff7b44eeaa42b025394138f7a81880e530d102c27befb4ce4497"}
Oct 30 20:53:36.868828 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:36.868696    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c7807b2d4e6ff7b44eeaa42b025394138f7a81880e530d102c27befb4ce4497"
Oct 30 20:53:37.187758 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:37.187729    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d4597073-8ff0-4a4e-933c-c64e934fe977/prometheus/0.log"
Oct 30 20:53:37.190254 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:37.190235    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d4597073-8ff0-4a4e-933c-c64e934fe977/prometheus/0.log"
Oct 30 20:53:37.201996 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:37.201975    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r7lpf_764461d5-e0b2-4856-a97e-0c5d3d34a497/ovn-acl-logging/0.log"
Oct 30 20:53:37.204160 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:37.204137    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r7lpf_764461d5-e0b2-4856-a97e-0c5d3d34a497/ovn-acl-logging/0.log"
Oct 30 20:53:43.596447 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:43.596407    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-247596eb-6mdlp"]
Oct 30 20:53:43.597048 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:43.597031    5713 cpu_manager.go:414] "containerMap: removing container" podUID="ed2d8620-1d01-48d1-9491-3bacfd9214a3" containerName="revoke-keys"
Oct 30 20:53:43.597103 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:43.597051    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed2d8620-1d01-48d1-9491-3bacfd9214a3" containerName="revoke-keys"
Oct 30 20:53:43.597175 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:43.597164    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed2d8620-1d01-48d1-9491-3bacfd9214a3" containerName="revoke-keys"
Oct 30 20:53:43.603536 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:43.603521    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-247596eb-6mdlp"
Oct 30 20:53:43.656393 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:43.656358    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-247596eb-6mdlp"]
Oct 30 20:53:43.702648 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:43.702616    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:53:43.702648 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:43.702616    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:53:43.702887 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:43.702631    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:53:43.709017 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:43.708997    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 20:53:43.721593 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:43.721565    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prggc\" (UniqueName: \"kubernetes.io/projected/b4674926-5f7f-4869-b1bd-274bd996d3e7-kube-api-access-prggc\") pod \"maas-api-revoke-sub-247596eb-6mdlp\" (UID: \"b4674926-5f7f-4869-b1bd-274bd996d3e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-247596eb-6mdlp"
Oct 30 20:53:43.721717 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:43.721631    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4674926-5f7f-4869-b1bd-274bd996d3e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-247596eb-6mdlp\" (UID: \"b4674926-5f7f-4869-b1bd-274bd996d3e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-247596eb-6mdlp"
Oct 30 20:53:43.721769 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:43.721752    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4674926-5f7f-4869-b1bd-274bd996d3e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-247596eb-6mdlp\" (UID: \"b4674926-5f7f-4869-b1bd-274bd996d3e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-247596eb-6mdlp"
Oct 30 20:53:43.822942 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:43.822911    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prggc\" (UniqueName: \"kubernetes.io/projected/b4674926-5f7f-4869-b1bd-274bd996d3e7-kube-api-access-prggc\") pod \"maas-api-revoke-sub-247596eb-6mdlp\" (UID: \"b4674926-5f7f-4869-b1bd-274bd996d3e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-247596eb-6mdlp"
Oct 30 20:53:43.823155 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:43.822970    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4674926-5f7f-4869-b1bd-274bd996d3e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-247596eb-6mdlp\" (UID: \"b4674926-5f7f-4869-b1bd-274bd996d3e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-247596eb-6mdlp"
Oct 30 20:53:43.823155 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:43.823038    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4674926-5f7f-4869-b1bd-274bd996d3e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-247596eb-6mdlp\" (UID: \"b4674926-5f7f-4869-b1bd-274bd996d3e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-247596eb-6mdlp"
Oct 30 20:53:43.823779 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:43.823757    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4674926-5f7f-4869-b1bd-274bd996d3e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-247596eb-6mdlp\" (UID: \"b4674926-5f7f-4869-b1bd-274bd996d3e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-247596eb-6mdlp"
Oct 30 20:53:43.825742 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:43.825706    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4674926-5f7f-4869-b1bd-274bd996d3e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-247596eb-6mdlp\" (UID: \"b4674926-5f7f-4869-b1bd-274bd996d3e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-247596eb-6mdlp"
Oct 30 20:53:43.881328 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:43.881253    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prggc\" (UniqueName: \"kubernetes.io/projected/b4674926-5f7f-4869-b1bd-274bd996d3e7-kube-api-access-prggc\") pod \"maas-api-revoke-sub-247596eb-6mdlp\" (UID: \"b4674926-5f7f-4869-b1bd-274bd996d3e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-247596eb-6mdlp"
Oct 30 20:53:43.917260 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:43.917228    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-247596eb-6mdlp"
Oct 30 20:53:44.296379 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:44.296330    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-247596eb-6mdlp"]
Oct 30 20:53:44.299778 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:53:44.299749    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4674926_5f7f_4869_b1bd_274bd996d3e7.slice/crio-8a1e63107c85c0418bb5f32984af73d33874f4bd36d2938553f822b879de6b57 WatchSource:0}: Error finding container 8a1e63107c85c0418bb5f32984af73d33874f4bd36d2938553f822b879de6b57: Status 404 returned error can't find the container with id 8a1e63107c85c0418bb5f32984af73d33874f4bd36d2938553f822b879de6b57
Oct 30 20:53:44.918549 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:44.918515    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-247596eb-6mdlp" event={"ID":"b4674926-5f7f-4869-b1bd-274bd996d3e7","Type":"ContainerStarted","Data":"901ecb0ffb090ba3698ca6dd19bc5a7e9eff3c81db9284f372af08872771ff86"}
Oct 30 20:53:44.918993 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:44.918579    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-247596eb-6mdlp" event={"ID":"b4674926-5f7f-4869-b1bd-274bd996d3e7","Type":"ContainerStarted","Data":"8a1e63107c85c0418bb5f32984af73d33874f4bd36d2938553f822b879de6b57"}
Oct 30 20:53:45.032557 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:45.032501    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-247596eb-6mdlp" podStartSLOduration=2.032485476 podStartE2EDuration="2.032485476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:53:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:53:45.031705798 +0000 UTC m=+908.096482197" watchObservedRunningTime="2026-10-30 20:53:45.032485476 +0000 UTC m=+908.097261873"
Oct 30 20:53:45.926714 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:45.926679    5713 generic.go:356] "Generic (PLEG): container finished" podID="b4674926-5f7f-4869-b1bd-274bd996d3e7" containerID="901ecb0ffb090ba3698ca6dd19bc5a7e9eff3c81db9284f372af08872771ff86" exitCode=0
Oct 30 20:53:45.927201 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:45.926770    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-247596eb-6mdlp" event={"ID":"b4674926-5f7f-4869-b1bd-274bd996d3e7","Type":"ContainerDied","Data":"901ecb0ffb090ba3698ca6dd19bc5a7e9eff3c81db9284f372af08872771ff86"}
Oct 30 20:53:47.083682 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:47.083655    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-247596eb-6mdlp"
Oct 30 20:53:47.163042 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:47.163014    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4674926-5f7f-4869-b1bd-274bd996d3e7-kube-api-access-prggc\" (UniqueName: \"kubernetes.io/projected/b4674926-5f7f-4869-b1bd-274bd996d3e7-kube-api-access-prggc\") pod \"b4674926-5f7f-4869-b1bd-274bd996d3e7\" (UID: \"b4674926-5f7f-4869-b1bd-274bd996d3e7\") "
Oct 30 20:53:47.163201 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:47.163068    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4674926-5f7f-4869-b1bd-274bd996d3e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4674926-5f7f-4869-b1bd-274bd996d3e7-maas-api-xks-ca\") pod \"b4674926-5f7f-4869-b1bd-274bd996d3e7\" (UID: \"b4674926-5f7f-4869-b1bd-274bd996d3e7\") "
Oct 30 20:53:47.163201 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:47.163161    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4674926-5f7f-4869-b1bd-274bd996d3e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4674926-5f7f-4869-b1bd-274bd996d3e7-maas-api-openshift-ca\") pod \"b4674926-5f7f-4869-b1bd-274bd996d3e7\" (UID: \"b4674926-5f7f-4869-b1bd-274bd996d3e7\") "
Oct 30 20:53:47.163548 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:47.163522    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4674926-5f7f-4869-b1bd-274bd996d3e7-maas-api-openshift-ca" pod "b4674926-5f7f-4869-b1bd-274bd996d3e7" (UID: "b4674926-5f7f-4869-b1bd-274bd996d3e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:53:47.165500 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:47.165466    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4674926-5f7f-4869-b1bd-274bd996d3e7-kube-api-access-prggc" pod "b4674926-5f7f-4869-b1bd-274bd996d3e7" (UID: "b4674926-5f7f-4869-b1bd-274bd996d3e7"). InnerVolumeSpecName "kube-api-access-prggc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:53:47.165616 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:47.165497    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4674926-5f7f-4869-b1bd-274bd996d3e7-maas-api-xks-ca" pod "b4674926-5f7f-4869-b1bd-274bd996d3e7" (UID: "b4674926-5f7f-4869-b1bd-274bd996d3e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:53:47.264844 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:47.264728    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prggc\" (UniqueName: \"kubernetes.io/projected/b4674926-5f7f-4869-b1bd-274bd996d3e7-kube-api-access-prggc\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:47.264844 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:47.264762    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4674926-5f7f-4869-b1bd-274bd996d3e7-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:47.264844 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:47.264777    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4674926-5f7f-4869-b1bd-274bd996d3e7-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:53:47.940688 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:47.940657    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-247596eb-6mdlp" event={"ID":"b4674926-5f7f-4869-b1bd-274bd996d3e7","Type":"ContainerDied","Data":"8a1e63107c85c0418bb5f32984af73d33874f4bd36d2938553f822b879de6b57"}
Oct 30 20:53:47.940688 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:47.940689    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-247596eb-6mdlp"
Oct 30 20:53:47.940925 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:47.940705    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a1e63107c85c0418bb5f32984af73d33874f4bd36d2938553f822b879de6b57"
Oct 30 20:53:54.458773 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:53:54.458734    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 20:53:58.959920 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:58.959883    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2"]
Oct 30 20:53:58.960480 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:58.960464    5713 cpu_manager.go:414] "containerMap: removing container" podUID="b4674926-5f7f-4869-b1bd-274bd996d3e7" containerName="revoke-keys"
Oct 30 20:53:58.960530 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:58.960480    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4674926-5f7f-4869-b1bd-274bd996d3e7" containerName="revoke-keys"
Oct 30 20:53:58.960574 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:58.960564    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4674926-5f7f-4869-b1bd-274bd996d3e7" containerName="revoke-keys"
Oct 30 20:53:58.964122 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:58.964099    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2"
Oct 30 20:53:59.001634 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:59.001601    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3faef396-57dc-4ff5-af9f-c1f940cce913-istio-podinfo\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2\" (UID: \"3faef396-57dc-4ff5-af9f-c1f940cce913\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2"
Oct 30 20:53:59.001828 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:59.001657    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3faef396-57dc-4ff5-af9f-c1f940cce913-workload-certs\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2\" (UID: \"3faef396-57dc-4ff5-af9f-c1f940cce913\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2"
Oct 30 20:53:59.001828 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:59.001675    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3faef396-57dc-4ff5-af9f-c1f940cce913-credential-socket\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2\" (UID: \"3faef396-57dc-4ff5-af9f-c1f940cce913\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2"
Oct 30 20:53:59.001934 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:59.001817    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3faef396-57dc-4ff5-af9f-c1f940cce913-workload-socket\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2\" (UID: \"3faef396-57dc-4ff5-af9f-c1f940cce913\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2"
Oct 30 20:53:59.001934 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:59.001861    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht7zn\" (UniqueName: \"kubernetes.io/projected/3faef396-57dc-4ff5-af9f-c1f940cce913-kube-api-access-ht7zn\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2\" (UID: \"3faef396-57dc-4ff5-af9f-c1f940cce913\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2"
Oct 30 20:53:59.002017 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:59.002002    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3faef396-57dc-4ff5-af9f-c1f940cce913-istio-data\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2\" (UID: \"3faef396-57dc-4ff5-af9f-c1f940cce913\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2"
Oct 30 20:53:59.002065 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:59.002030    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3faef396-57dc-4ff5-af9f-c1f940cce913-istio-envoy\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2\" (UID: \"3faef396-57dc-4ff5-af9f-c1f940cce913\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2"
Oct 30 20:53:59.002113 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:59.002100    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3faef396-57dc-4ff5-af9f-c1f940cce913-istio-token\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2\" (UID: \"3faef396-57dc-4ff5-af9f-c1f940cce913\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2"
Oct 30 20:53:59.002155 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:59.002122    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3faef396-57dc-4ff5-af9f-c1f940cce913-istiod-ca-cert\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2\" (UID: \"3faef396-57dc-4ff5-af9f-c1f940cce913\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2"
Oct 30 20:53:59.016226 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:59.016187    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-682999-openshift-default-dockercfg-g27tl\""
Oct 30 20:53:59.069207 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:59.069158    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2"]
Oct 30 20:53:59.103133 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:59.103101    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3faef396-57dc-4ff5-af9f-c1f940cce913-istio-podinfo\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2\" (UID: \"3faef396-57dc-4ff5-af9f-c1f940cce913\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2"
Oct 30 20:53:59.103352 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:59.103224    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3faef396-57dc-4ff5-af9f-c1f940cce913-workload-certs\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2\" (UID: \"3faef396-57dc-4ff5-af9f-c1f940cce913\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2"
Oct 30 20:53:59.103352 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:59.103252    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3faef396-57dc-4ff5-af9f-c1f940cce913-credential-socket\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2\" (UID: \"3faef396-57dc-4ff5-af9f-c1f940cce913\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2"
Oct 30 20:53:59.103352 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:59.103289    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3faef396-57dc-4ff5-af9f-c1f940cce913-workload-socket\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2\" (UID: \"3faef396-57dc-4ff5-af9f-c1f940cce913\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2"
Oct 30 20:53:59.103352 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:59.103330    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht7zn\" (UniqueName: \"kubernetes.io/projected/3faef396-57dc-4ff5-af9f-c1f940cce913-kube-api-access-ht7zn\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2\" (UID: \"3faef396-57dc-4ff5-af9f-c1f940cce913\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2"
Oct 30 20:53:59.103570 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:59.103447    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3faef396-57dc-4ff5-af9f-c1f940cce913-istio-data\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2\" (UID: \"3faef396-57dc-4ff5-af9f-c1f940cce913\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2"
Oct 30 20:53:59.103570 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:59.103485    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3faef396-57dc-4ff5-af9f-c1f940cce913-istio-envoy\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2\" (UID: \"3faef396-57dc-4ff5-af9f-c1f940cce913\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2"
Oct 30 20:53:59.103671 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:59.103580    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3faef396-57dc-4ff5-af9f-c1f940cce913-istio-token\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2\" (UID: \"3faef396-57dc-4ff5-af9f-c1f940cce913\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2"
Oct 30 20:53:59.103671 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:59.103620    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3faef396-57dc-4ff5-af9f-c1f940cce913-istiod-ca-cert\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2\" (UID: \"3faef396-57dc-4ff5-af9f-c1f940cce913\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2"
Oct 30 20:53:59.103777 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:59.103736    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3faef396-57dc-4ff5-af9f-c1f940cce913-credential-socket\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2\" (UID: \"3faef396-57dc-4ff5-af9f-c1f940cce913\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2"
Oct 30 20:53:59.103880 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:59.103842    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3faef396-57dc-4ff5-af9f-c1f940cce913-workload-socket\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2\" (UID: \"3faef396-57dc-4ff5-af9f-c1f940cce913\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2"
Oct 30 20:53:59.103880 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:59.103851    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3faef396-57dc-4ff5-af9f-c1f940cce913-workload-certs\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2\" (UID: \"3faef396-57dc-4ff5-af9f-c1f940cce913\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2"
Oct 30 20:53:59.104155 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:59.104128    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3faef396-57dc-4ff5-af9f-c1f940cce913-istio-data\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2\" (UID: \"3faef396-57dc-4ff5-af9f-c1f940cce913\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2"
Oct 30 20:53:59.104600 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:59.104579    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3faef396-57dc-4ff5-af9f-c1f940cce913-istiod-ca-cert\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2\" (UID: \"3faef396-57dc-4ff5-af9f-c1f940cce913\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2"
Oct 30 20:53:59.106163 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:59.106139    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3faef396-57dc-4ff5-af9f-c1f940cce913-istio-envoy\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2\" (UID: \"3faef396-57dc-4ff5-af9f-c1f940cce913\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2"
Oct 30 20:53:59.106258 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:59.106194    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3faef396-57dc-4ff5-af9f-c1f940cce913-istio-podinfo\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2\" (UID: \"3faef396-57dc-4ff5-af9f-c1f940cce913\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2"
Oct 30 20:53:59.160358 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:59.160323    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3faef396-57dc-4ff5-af9f-c1f940cce913-istio-token\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2\" (UID: \"3faef396-57dc-4ff5-af9f-c1f940cce913\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2"
Oct 30 20:53:59.160553 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:59.160527    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht7zn\" (UniqueName: \"kubernetes.io/projected/3faef396-57dc-4ff5-af9f-c1f940cce913-kube-api-access-ht7zn\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2\" (UID: \"3faef396-57dc-4ff5-af9f-c1f940cce913\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2"
Oct 30 20:53:59.264341 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:59.264256    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a561575-s9frn"]
Oct 30 20:53:59.269325 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:59.269308    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a561575-s9frn"
Oct 30 20:53:59.277678 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:59.277659    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2"
Oct 30 20:53:59.306077 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:59.306034    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b76f110a-b354-400d-b8ff-637690977e28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a561575-s9frn\" (UID: \"b76f110a-b354-400d-b8ff-637690977e28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a561575-s9frn"
Oct 30 20:53:59.306267 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:59.306243    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsxgb\" (UniqueName: \"kubernetes.io/projected/b76f110a-b354-400d-b8ff-637690977e28-kube-api-access-qsxgb\") pod \"maas-api-revoke-sub-5a561575-s9frn\" (UID: \"b76f110a-b354-400d-b8ff-637690977e28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a561575-s9frn"
Oct 30 20:53:59.306374 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:59.306356    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b76f110a-b354-400d-b8ff-637690977e28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a561575-s9frn\" (UID: \"b76f110a-b354-400d-b8ff-637690977e28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a561575-s9frn"
Oct 30 20:53:59.322319 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:59.322284    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:53:59.322486 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:59.322298    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:53:59.322486 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:59.322425    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 20:53:59.322486 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:59.322439    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:53:59.326112 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:59.326074    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a561575-s9frn"]
Oct 30 20:53:59.407639 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:59.407604    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsxgb\" (UniqueName: \"kubernetes.io/projected/b76f110a-b354-400d-b8ff-637690977e28-kube-api-access-qsxgb\") pod \"maas-api-revoke-sub-5a561575-s9frn\" (UID: \"b76f110a-b354-400d-b8ff-637690977e28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a561575-s9frn"
Oct 30 20:53:59.407850 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:59.407696    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b76f110a-b354-400d-b8ff-637690977e28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a561575-s9frn\" (UID: \"b76f110a-b354-400d-b8ff-637690977e28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a561575-s9frn"
Oct 30 20:53:59.407850 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:59.407747    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b76f110a-b354-400d-b8ff-637690977e28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a561575-s9frn\" (UID: \"b76f110a-b354-400d-b8ff-637690977e28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a561575-s9frn"
Oct 30 20:53:59.408454 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:59.408426    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b76f110a-b354-400d-b8ff-637690977e28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a561575-s9frn\" (UID: \"b76f110a-b354-400d-b8ff-637690977e28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a561575-s9frn"
Oct 30 20:53:59.410176 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:59.410149    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b76f110a-b354-400d-b8ff-637690977e28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a561575-s9frn\" (UID: \"b76f110a-b354-400d-b8ff-637690977e28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a561575-s9frn"
Oct 30 20:53:59.468266 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:59.468238    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsxgb\" (UniqueName: \"kubernetes.io/projected/b76f110a-b354-400d-b8ff-637690977e28-kube-api-access-qsxgb\") pod \"maas-api-revoke-sub-5a561575-s9frn\" (UID: \"b76f110a-b354-400d-b8ff-637690977e28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a561575-s9frn"
Oct 30 20:53:59.584275 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:59.584190    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a561575-s9frn"
Oct 30 20:53:59.713054 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:59.713025    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2"]
Oct 30 20:53:59.714297 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:53:59.714274    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3faef396_57dc_4ff5_af9f_c1f940cce913.slice/crio-44befe62a1c3aabdf6a96543f00f139f50911c9443099a8220cfd44c0384c4ed WatchSource:0}: Error finding container 44befe62a1c3aabdf6a96543f00f139f50911c9443099a8220cfd44c0384c4ed: Status 404 returned error can't find the container with id 44befe62a1c3aabdf6a96543f00f139f50911c9443099a8220cfd44c0384c4ed
Oct 30 20:53:59.716688 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:59.716658    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 20:53:59.716811 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:59.716736    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 20:53:59.716811 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:59.716781    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 20:53:59.970505 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:53:59.970477    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a561575-s9frn"]
Oct 30 20:53:59.971769 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:53:59.971733    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb76f110a_b354_400d_b8ff_637690977e28.slice/crio-375d9f918319dfc51bac95dfda7e5229fd34e792046609dc88af8fe3b5de4c88 WatchSource:0}: Error finding container 375d9f918319dfc51bac95dfda7e5229fd34e792046609dc88af8fe3b5de4c88: Status 404 returned error can't find the container with id 375d9f918319dfc51bac95dfda7e5229fd34e792046609dc88af8fe3b5de4c88
Oct 30 20:54:00.019304 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:00.019263    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2" event={"ID":"3faef396-57dc-4ff5-af9f-c1f940cce913","Type":"ContainerStarted","Data":"6c3f052bb8ec591c4c5dff555fbca6d71e69e33f79582ba339fde1b02474b8dc"}
Oct 30 20:54:00.019468 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:00.019325    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2" event={"ID":"3faef396-57dc-4ff5-af9f-c1f940cce913","Type":"ContainerStarted","Data":"44befe62a1c3aabdf6a96543f00f139f50911c9443099a8220cfd44c0384c4ed"}
Oct 30 20:54:00.020745 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:00.020719    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a561575-s9frn" event={"ID":"b76f110a-b354-400d-b8ff-637690977e28","Type":"ContainerStarted","Data":"375d9f918319dfc51bac95dfda7e5229fd34e792046609dc88af8fe3b5de4c88"}
Oct 30 20:54:00.138329 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:00.138270    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2" podStartSLOduration=2.138254966 podStartE2EDuration="2.138254966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:53:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:54:00.135081159 +0000 UTC m=+923.199857558" watchObservedRunningTime="2026-10-30 20:54:00.138254966 +0000 UTC m=+923.203031363"
Oct 30 20:54:00.278625 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:00.278537    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2"
Oct 30 20:54:00.280381 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:00.280358    5713 patch_prober.go:29] interesting pod/e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Oct 30 20:54:00.280459 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:00.280407    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2" podUID="3faef396-57dc-4ff5-af9f-c1f940cce913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Oct 30 20:54:01.028061 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:01.028027    5713 generic.go:356] "Generic (PLEG): container finished" podID="b76f110a-b354-400d-b8ff-637690977e28" containerID="987fc6c83b8aea11ef470282c8678aad0b35c2dbc15507e408b1f56779ae3c1d" exitCode=0
Oct 30 20:54:01.028498 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:01.028117    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a561575-s9frn" event={"ID":"b76f110a-b354-400d-b8ff-637690977e28","Type":"ContainerDied","Data":"987fc6c83b8aea11ef470282c8678aad0b35c2dbc15507e408b1f56779ae3c1d"}
Oct 30 20:54:01.278685 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:01.278596    5713 patch_prober.go:29] interesting pod/e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Oct 30 20:54:01.278685 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:01.278668    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2" podUID="3faef396-57dc-4ff5-af9f-c1f940cce913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Oct 30 20:54:02.224322 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:02.224299    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a561575-s9frn"
Oct 30 20:54:02.265844 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:02.265775    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b76f110a-b354-400d-b8ff-637690977e28-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b76f110a-b354-400d-b8ff-637690977e28-maas-api-openshift-ca\") pod \"b76f110a-b354-400d-b8ff-637690977e28\" (UID: \"b76f110a-b354-400d-b8ff-637690977e28\") "
Oct 30 20:54:02.266042 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:02.265967    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b76f110a-b354-400d-b8ff-637690977e28-kube-api-access-qsxgb\" (UniqueName: \"kubernetes.io/projected/b76f110a-b354-400d-b8ff-637690977e28-kube-api-access-qsxgb\") pod \"b76f110a-b354-400d-b8ff-637690977e28\" (UID: \"b76f110a-b354-400d-b8ff-637690977e28\") "
Oct 30 20:54:02.266042 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:02.266026    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b76f110a-b354-400d-b8ff-637690977e28-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b76f110a-b354-400d-b8ff-637690977e28-maas-api-xks-ca\") pod \"b76f110a-b354-400d-b8ff-637690977e28\" (UID: \"b76f110a-b354-400d-b8ff-637690977e28\") "
Oct 30 20:54:02.266165 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:02.266137    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b76f110a-b354-400d-b8ff-637690977e28-maas-api-openshift-ca" pod "b76f110a-b354-400d-b8ff-637690977e28" (UID: "b76f110a-b354-400d-b8ff-637690977e28"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:54:02.266405 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:02.266387    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b76f110a-b354-400d-b8ff-637690977e28-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:02.268235 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:02.268204    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b76f110a-b354-400d-b8ff-637690977e28-maas-api-xks-ca" pod "b76f110a-b354-400d-b8ff-637690977e28" (UID: "b76f110a-b354-400d-b8ff-637690977e28"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:54:02.268345 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:02.268252    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b76f110a-b354-400d-b8ff-637690977e28-kube-api-access-qsxgb" pod "b76f110a-b354-400d-b8ff-637690977e28" (UID: "b76f110a-b354-400d-b8ff-637690977e28"). InnerVolumeSpecName "kube-api-access-qsxgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:02.279006 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:02.278980    5713 patch_prober.go:29] interesting pod/e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Oct 30 20:54:02.279094 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:02.279036    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2" podUID="3faef396-57dc-4ff5-af9f-c1f940cce913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Oct 30 20:54:02.367820 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:02.367724    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsxgb\" (UniqueName: \"kubernetes.io/projected/b76f110a-b354-400d-b8ff-637690977e28-kube-api-access-qsxgb\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:02.367820 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:02.367752    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b76f110a-b354-400d-b8ff-637690977e28-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:03.066216 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:03.066189    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a561575-s9frn"
Oct 30 20:54:03.070186 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:03.070157    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a561575-s9frn" event={"ID":"b76f110a-b354-400d-b8ff-637690977e28","Type":"ContainerDied","Data":"375d9f918319dfc51bac95dfda7e5229fd34e792046609dc88af8fe3b5de4c88"}
Oct 30 20:54:03.070313 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:03.070208    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="375d9f918319dfc51bac95dfda7e5229fd34e792046609dc88af8fe3b5de4c88"
Oct 30 20:54:03.278101 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:03.278066    5713 patch_prober.go:29] interesting pod/e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Oct 30 20:54:03.278591 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:03.278130    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2" podUID="3faef396-57dc-4ff5-af9f-c1f940cce913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Oct 30 20:54:04.279082 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:04.279037    5713 patch_prober.go:29] interesting pod/e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Oct 30 20:54:04.279587 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:04.279110    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2" podUID="3faef396-57dc-4ff5-af9f-c1f940cce913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Oct 30 20:54:05.278617 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:05.278583    5713 patch_prober.go:29] interesting pod/e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Oct 30 20:54:05.278838 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:05.278643    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2" podUID="3faef396-57dc-4ff5-af9f-c1f940cce913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Oct 30 20:54:06.462087 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:06.462041    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2"
Oct 30 20:54:06.462554 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:06.462479    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2"
Oct 30 20:54:06.463349 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:06.463325    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2"
Oct 30 20:54:10.219997 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:10.219956    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95279ef2-6mc22"]
Oct 30 20:54:10.220483 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:10.220469    5713 cpu_manager.go:414] "containerMap: removing container" podUID="b76f110a-b354-400d-b8ff-637690977e28" containerName="revoke-keys"
Oct 30 20:54:10.220523 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:10.220488    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b76f110a-b354-400d-b8ff-637690977e28" containerName="revoke-keys"
Oct 30 20:54:10.220582 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:10.220573    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="b76f110a-b354-400d-b8ff-637690977e28" containerName="revoke-keys"
Oct 30 20:54:10.228209 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:10.228190    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95279ef2-6mc22"
Oct 30 20:54:10.282818 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:10.282765    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95279ef2-6mc22"]
Oct 30 20:54:10.327885 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:10.327847    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:54:10.328056 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:10.327887    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:54:10.328056 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:10.327920    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:54:10.330037 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:10.330005    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 20:54:10.369108 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:10.369072    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57g9v\" (UniqueName: \"kubernetes.io/projected/f9dae508-b784-4f7d-b6a1-7146e62d51b2-kube-api-access-57g9v\") pod \"maas-api-revoke-sub-95279ef2-6mc22\" (UID: \"f9dae508-b784-4f7d-b6a1-7146e62d51b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95279ef2-6mc22"
Oct 30 20:54:10.369299 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:10.369210    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9dae508-b784-4f7d-b6a1-7146e62d51b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95279ef2-6mc22\" (UID: \"f9dae508-b784-4f7d-b6a1-7146e62d51b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95279ef2-6mc22"
Oct 30 20:54:10.369299 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:10.369274    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9dae508-b784-4f7d-b6a1-7146e62d51b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95279ef2-6mc22\" (UID: \"f9dae508-b784-4f7d-b6a1-7146e62d51b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95279ef2-6mc22"
Oct 30 20:54:10.470339 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:10.470234    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57g9v\" (UniqueName: \"kubernetes.io/projected/f9dae508-b784-4f7d-b6a1-7146e62d51b2-kube-api-access-57g9v\") pod \"maas-api-revoke-sub-95279ef2-6mc22\" (UID: \"f9dae508-b784-4f7d-b6a1-7146e62d51b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95279ef2-6mc22"
Oct 30 20:54:10.470518 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:10.470346    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9dae508-b784-4f7d-b6a1-7146e62d51b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95279ef2-6mc22\" (UID: \"f9dae508-b784-4f7d-b6a1-7146e62d51b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95279ef2-6mc22"
Oct 30 20:54:10.470518 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:10.470431    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9dae508-b784-4f7d-b6a1-7146e62d51b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95279ef2-6mc22\" (UID: \"f9dae508-b784-4f7d-b6a1-7146e62d51b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95279ef2-6mc22"
Oct 30 20:54:10.471005 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:10.470988    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9dae508-b784-4f7d-b6a1-7146e62d51b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95279ef2-6mc22\" (UID: \"f9dae508-b784-4f7d-b6a1-7146e62d51b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95279ef2-6mc22"
Oct 30 20:54:10.472879 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:10.472859    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9dae508-b784-4f7d-b6a1-7146e62d51b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95279ef2-6mc22\" (UID: \"f9dae508-b784-4f7d-b6a1-7146e62d51b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95279ef2-6mc22"
Oct 30 20:54:10.532660 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:10.532629    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57g9v\" (UniqueName: \"kubernetes.io/projected/f9dae508-b784-4f7d-b6a1-7146e62d51b2-kube-api-access-57g9v\") pod \"maas-api-revoke-sub-95279ef2-6mc22\" (UID: \"f9dae508-b784-4f7d-b6a1-7146e62d51b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95279ef2-6mc22"
Oct 30 20:54:10.542499 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:10.542478    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95279ef2-6mc22"
Oct 30 20:54:10.919620 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:10.919538    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95279ef2-6mc22"]
Oct 30 20:54:10.923302 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:54:10.923265    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9dae508_b784_4f7d_b6a1_7146e62d51b2.slice/crio-ba20a55fa481ae984741d6623c4bea68b70828b4a4986362b3b7c7d366cea158 WatchSource:0}: Error finding container ba20a55fa481ae984741d6623c4bea68b70828b4a4986362b3b7c7d366cea158: Status 404 returned error can't find the container with id ba20a55fa481ae984741d6623c4bea68b70828b4a4986362b3b7c7d366cea158
Oct 30 20:54:11.119169 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:11.119135    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95279ef2-6mc22" event={"ID":"f9dae508-b784-4f7d-b6a1-7146e62d51b2","Type":"ContainerStarted","Data":"ba20a55fa481ae984741d6623c4bea68b70828b4a4986362b3b7c7d366cea158"}
Oct 30 20:54:12.133892 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:12.133768    5713 generic.go:356] "Generic (PLEG): container finished" podID="f9dae508-b784-4f7d-b6a1-7146e62d51b2" containerID="1327fe84166cc430aac285f9c6d38e2d759c69de43450f717c2902db94d6149b" exitCode=0
Oct 30 20:54:12.133892 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:12.133814    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95279ef2-6mc22" event={"ID":"f9dae508-b784-4f7d-b6a1-7146e62d51b2","Type":"ContainerDied","Data":"1327fe84166cc430aac285f9c6d38e2d759c69de43450f717c2902db94d6149b"}
Oct 30 20:54:13.291177 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:13.291153    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95279ef2-6mc22"
Oct 30 20:54:13.401975 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:13.401888    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9dae508-b784-4f7d-b6a1-7146e62d51b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9dae508-b784-4f7d-b6a1-7146e62d51b2-maas-api-openshift-ca\") pod \"f9dae508-b784-4f7d-b6a1-7146e62d51b2\" (UID: \"f9dae508-b784-4f7d-b6a1-7146e62d51b2\") "
Oct 30 20:54:13.401975 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:13.401932    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9dae508-b784-4f7d-b6a1-7146e62d51b2-kube-api-access-57g9v\" (UniqueName: \"kubernetes.io/projected/f9dae508-b784-4f7d-b6a1-7146e62d51b2-kube-api-access-57g9v\") pod \"f9dae508-b784-4f7d-b6a1-7146e62d51b2\" (UID: \"f9dae508-b784-4f7d-b6a1-7146e62d51b2\") "
Oct 30 20:54:13.401975 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:13.401974    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9dae508-b784-4f7d-b6a1-7146e62d51b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9dae508-b784-4f7d-b6a1-7146e62d51b2-maas-api-xks-ca\") pod \"f9dae508-b784-4f7d-b6a1-7146e62d51b2\" (UID: \"f9dae508-b784-4f7d-b6a1-7146e62d51b2\") "
Oct 30 20:54:13.402335 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:13.402311    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9dae508-b784-4f7d-b6a1-7146e62d51b2-maas-api-openshift-ca" pod "f9dae508-b784-4f7d-b6a1-7146e62d51b2" (UID: "f9dae508-b784-4f7d-b6a1-7146e62d51b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:54:13.404286 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:13.404254    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9dae508-b784-4f7d-b6a1-7146e62d51b2-maas-api-xks-ca" pod "f9dae508-b784-4f7d-b6a1-7146e62d51b2" (UID: "f9dae508-b784-4f7d-b6a1-7146e62d51b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:54:13.404409 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:13.404389    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9dae508-b784-4f7d-b6a1-7146e62d51b2-kube-api-access-57g9v" pod "f9dae508-b784-4f7d-b6a1-7146e62d51b2" (UID: "f9dae508-b784-4f7d-b6a1-7146e62d51b2"). InnerVolumeSpecName "kube-api-access-57g9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:13.503435 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:13.503403    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9dae508-b784-4f7d-b6a1-7146e62d51b2-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:13.503435 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:13.503433    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57g9v\" (UniqueName: \"kubernetes.io/projected/f9dae508-b784-4f7d-b6a1-7146e62d51b2-kube-api-access-57g9v\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:13.503629 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:13.503444    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9dae508-b784-4f7d-b6a1-7146e62d51b2-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:14.147802 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:14.147760    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95279ef2-6mc22" event={"ID":"f9dae508-b784-4f7d-b6a1-7146e62d51b2","Type":"ContainerDied","Data":"ba20a55fa481ae984741d6623c4bea68b70828b4a4986362b3b7c7d366cea158"}
Oct 30 20:54:14.147981 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:14.147815    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95279ef2-6mc22"
Oct 30 20:54:14.147981 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:14.147852    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba20a55fa481ae984741d6623c4bea68b70828b4a4986362b3b7c7d366cea158"
Oct 30 20:54:14.792759 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:14.792720    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb"]
Oct 30 20:54:14.793315 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:14.793298    5713 cpu_manager.go:414] "containerMap: removing container" podUID="f9dae508-b784-4f7d-b6a1-7146e62d51b2" containerName="revoke-keys"
Oct 30 20:54:14.793374 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:14.793316    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9dae508-b784-4f7d-b6a1-7146e62d51b2" containerName="revoke-keys"
Oct 30 20:54:14.793417 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:14.793403    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9dae508-b784-4f7d-b6a1-7146e62d51b2" containerName="revoke-keys"
Oct 30 20:54:14.796963 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:14.796949    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb"
Oct 30 20:54:14.900988 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:14.900955    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-1fdaa7-openshift-default-dockercfg-57h8w\""
Oct 30 20:54:14.903775 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:14.903752    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb"]
Oct 30 20:54:14.919454 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:14.919421    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c7e4408-b25d-44fa-9587-3a1288a8fc87-istiod-ca-cert\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb\" (UID: \"2c7e4408-b25d-44fa-9587-3a1288a8fc87\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb"
Oct 30 20:54:14.919595 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:14.919466    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c7e4408-b25d-44fa-9587-3a1288a8fc87-workload-socket\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb\" (UID: \"2c7e4408-b25d-44fa-9587-3a1288a8fc87\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb"
Oct 30 20:54:14.919595 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:14.919544    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c7e4408-b25d-44fa-9587-3a1288a8fc87-istio-data\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb\" (UID: \"2c7e4408-b25d-44fa-9587-3a1288a8fc87\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb"
Oct 30 20:54:14.919595 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:14.919569    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c7e4408-b25d-44fa-9587-3a1288a8fc87-istio-podinfo\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb\" (UID: \"2c7e4408-b25d-44fa-9587-3a1288a8fc87\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb"
Oct 30 20:54:14.919716 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:14.919609    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c7e4408-b25d-44fa-9587-3a1288a8fc87-workload-certs\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb\" (UID: \"2c7e4408-b25d-44fa-9587-3a1288a8fc87\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb"
Oct 30 20:54:14.919716 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:14.919662    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c7e4408-b25d-44fa-9587-3a1288a8fc87-istio-envoy\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb\" (UID: \"2c7e4408-b25d-44fa-9587-3a1288a8fc87\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb"
Oct 30 20:54:14.919716 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:14.919695    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c7e4408-b25d-44fa-9587-3a1288a8fc87-istio-token\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb\" (UID: \"2c7e4408-b25d-44fa-9587-3a1288a8fc87\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb"
Oct 30 20:54:14.919866 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:14.919734    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fcm2\" (UniqueName: \"kubernetes.io/projected/2c7e4408-b25d-44fa-9587-3a1288a8fc87-kube-api-access-8fcm2\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb\" (UID: \"2c7e4408-b25d-44fa-9587-3a1288a8fc87\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb"
Oct 30 20:54:14.919904 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:14.919865    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c7e4408-b25d-44fa-9587-3a1288a8fc87-credential-socket\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb\" (UID: \"2c7e4408-b25d-44fa-9587-3a1288a8fc87\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb"
Oct 30 20:54:15.021334 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:15.021302    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c7e4408-b25d-44fa-9587-3a1288a8fc87-istio-data\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb\" (UID: \"2c7e4408-b25d-44fa-9587-3a1288a8fc87\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb"
Oct 30 20:54:15.021334 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:15.021334    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c7e4408-b25d-44fa-9587-3a1288a8fc87-istio-podinfo\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb\" (UID: \"2c7e4408-b25d-44fa-9587-3a1288a8fc87\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb"
Oct 30 20:54:15.021571 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:15.021358    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c7e4408-b25d-44fa-9587-3a1288a8fc87-workload-certs\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb\" (UID: \"2c7e4408-b25d-44fa-9587-3a1288a8fc87\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb"
Oct 30 20:54:15.021571 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:15.021433    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c7e4408-b25d-44fa-9587-3a1288a8fc87-istio-envoy\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb\" (UID: \"2c7e4408-b25d-44fa-9587-3a1288a8fc87\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb"
Oct 30 20:54:15.021571 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:15.021477    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c7e4408-b25d-44fa-9587-3a1288a8fc87-istio-token\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb\" (UID: \"2c7e4408-b25d-44fa-9587-3a1288a8fc87\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb"
Oct 30 20:54:15.021571 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:15.021497    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fcm2\" (UniqueName: \"kubernetes.io/projected/2c7e4408-b25d-44fa-9587-3a1288a8fc87-kube-api-access-8fcm2\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb\" (UID: \"2c7e4408-b25d-44fa-9587-3a1288a8fc87\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb"
Oct 30 20:54:15.021766 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:15.021692    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c7e4408-b25d-44fa-9587-3a1288a8fc87-credential-socket\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb\" (UID: \"2c7e4408-b25d-44fa-9587-3a1288a8fc87\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb"
Oct 30 20:54:15.021859 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:15.021816    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c7e4408-b25d-44fa-9587-3a1288a8fc87-workload-certs\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb\" (UID: \"2c7e4408-b25d-44fa-9587-3a1288a8fc87\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb"
Oct 30 20:54:15.021859 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:15.021842    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c7e4408-b25d-44fa-9587-3a1288a8fc87-istio-data\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb\" (UID: \"2c7e4408-b25d-44fa-9587-3a1288a8fc87\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb"
Oct 30 20:54:15.021959 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:15.021883    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c7e4408-b25d-44fa-9587-3a1288a8fc87-istiod-ca-cert\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb\" (UID: \"2c7e4408-b25d-44fa-9587-3a1288a8fc87\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb"
Oct 30 20:54:15.021959 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:15.021922    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c7e4408-b25d-44fa-9587-3a1288a8fc87-workload-socket\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb\" (UID: \"2c7e4408-b25d-44fa-9587-3a1288a8fc87\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb"
Oct 30 20:54:15.022039 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:15.022020    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c7e4408-b25d-44fa-9587-3a1288a8fc87-credential-socket\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb\" (UID: \"2c7e4408-b25d-44fa-9587-3a1288a8fc87\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb"
Oct 30 20:54:15.022255 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:15.022234    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c7e4408-b25d-44fa-9587-3a1288a8fc87-workload-socket\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb\" (UID: \"2c7e4408-b25d-44fa-9587-3a1288a8fc87\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb"
Oct 30 20:54:15.022498 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:15.022477    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c7e4408-b25d-44fa-9587-3a1288a8fc87-istiod-ca-cert\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb\" (UID: \"2c7e4408-b25d-44fa-9587-3a1288a8fc87\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb"
Oct 30 20:54:15.024007 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:15.023984    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c7e4408-b25d-44fa-9587-3a1288a8fc87-istio-envoy\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb\" (UID: \"2c7e4408-b25d-44fa-9587-3a1288a8fc87\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb"
Oct 30 20:54:15.024334 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:15.024316    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c7e4408-b25d-44fa-9587-3a1288a8fc87-istio-podinfo\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb\" (UID: \"2c7e4408-b25d-44fa-9587-3a1288a8fc87\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb"
Oct 30 20:54:15.078511 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:15.078435    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fcm2\" (UniqueName: \"kubernetes.io/projected/2c7e4408-b25d-44fa-9587-3a1288a8fc87-kube-api-access-8fcm2\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb\" (UID: \"2c7e4408-b25d-44fa-9587-3a1288a8fc87\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb"
Oct 30 20:54:15.078511 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:15.078491    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c7e4408-b25d-44fa-9587-3a1288a8fc87-istio-token\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb\" (UID: \"2c7e4408-b25d-44fa-9587-3a1288a8fc87\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb"
Oct 30 20:54:15.110537 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:15.110511    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb"
Oct 30 20:54:15.540544 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:15.540521    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb"]
Oct 30 20:54:15.541725 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:54:15.541701    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c7e4408_b25d_44fa_9587_3a1288a8fc87.slice/crio-0ebcf54c326c29900aa6d6e383bb983bd4021417be3127a93ac4378491463954 WatchSource:0}: Error finding container 0ebcf54c326c29900aa6d6e383bb983bd4021417be3127a93ac4378491463954: Status 404 returned error can't find the container with id 0ebcf54c326c29900aa6d6e383bb983bd4021417be3127a93ac4378491463954
Oct 30 20:54:15.544188 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:15.544160    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 20:54:15.544251 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:15.544218    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 20:54:15.544287 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:15.544247    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 20:54:16.178530 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:16.178488    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb" event={"ID":"2c7e4408-b25d-44fa-9587-3a1288a8fc87","Type":"ContainerStarted","Data":"148708a70cf2d226013537be1c071f074cc1d2ac70302684097f722729bb6293"}
Oct 30 20:54:16.178942 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:16.178589    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb" event={"ID":"2c7e4408-b25d-44fa-9587-3a1288a8fc87","Type":"ContainerStarted","Data":"0ebcf54c326c29900aa6d6e383bb983bd4021417be3127a93ac4378491463954"}
Oct 30 20:54:16.361544 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:16.361476    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb" podStartSLOduration=2.361454829 podStartE2EDuration="2.361454829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:54:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:54:16.357337136 +0000 UTC m=+939.422113536" watchObservedRunningTime="2026-10-30 20:54:16.361454829 +0000 UTC m=+939.426231231"
Oct 30 20:54:17.110713 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:17.110680    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb"
Oct 30 20:54:17.112262 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:17.112241    5713 patch_prober.go:29] interesting pod/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Oct 30 20:54:17.112380 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:17.112288    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb" podUID="2c7e4408-b25d-44fa-9587-3a1288a8fc87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Oct 30 20:54:18.110841 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:18.110778    5713 patch_prober.go:29] interesting pod/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Oct 30 20:54:18.111204 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:18.110858    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb" podUID="2c7e4408-b25d-44fa-9587-3a1288a8fc87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Oct 30 20:54:19.111704 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:19.111677    5713 patch_prober.go:29] interesting pod/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Oct 30 20:54:19.112103 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:19.111731    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb" podUID="2c7e4408-b25d-44fa-9587-3a1288a8fc87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Oct 30 20:54:20.111360 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:20.111324    5713 patch_prober.go:29] interesting pod/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Oct 30 20:54:20.111534 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:20.111369    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb" podUID="2c7e4408-b25d-44fa-9587-3a1288a8fc87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Oct 30 20:54:21.111948 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:21.111908    5713 patch_prober.go:29] interesting pod/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Oct 30 20:54:21.112344 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:21.111981    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb" podUID="2c7e4408-b25d-44fa-9587-3a1288a8fc87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Oct 30 20:54:22.269240 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:22.269205    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb"
Oct 30 20:54:22.269724 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:22.269560    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb"
Oct 30 20:54:22.271503 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:22.271481    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb"
Oct 30 20:54:24.477517 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:54:24.477472    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 20:54:27.538959 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:27.538929    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8cb8d7c-784pk"]
Oct 30 20:54:27.542739 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:27.542722    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8cb8d7c-784pk"
Oct 30 20:54:27.602954 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:27.602912    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a300bba-c70c-4ff1-97d6-7517fa970477-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8cb8d7c-784pk\" (UID: \"1a300bba-c70c-4ff1-97d6-7517fa970477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8cb8d7c-784pk"
Oct 30 20:54:27.603125 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:27.602967    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a300bba-c70c-4ff1-97d6-7517fa970477-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8cb8d7c-784pk\" (UID: \"1a300bba-c70c-4ff1-97d6-7517fa970477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8cb8d7c-784pk"
Oct 30 20:54:27.603125 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:27.603106    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmtmd\" (UniqueName: \"kubernetes.io/projected/1a300bba-c70c-4ff1-97d6-7517fa970477-kube-api-access-kmtmd\") pod \"maas-api-revoke-sub-f8cb8d7c-784pk\" (UID: \"1a300bba-c70c-4ff1-97d6-7517fa970477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8cb8d7c-784pk"
Oct 30 20:54:27.645983 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:27.645951    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:54:27.646137 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:27.645974    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:54:27.646137 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:27.645980    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:54:27.646137 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:27.645993    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 20:54:27.648831 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:27.648812    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8cb8d7c-784pk"]
Oct 30 20:54:27.704352 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:27.704317    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a300bba-c70c-4ff1-97d6-7517fa970477-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8cb8d7c-784pk\" (UID: \"1a300bba-c70c-4ff1-97d6-7517fa970477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8cb8d7c-784pk"
Oct 30 20:54:27.704531 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:27.704432    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmtmd\" (UniqueName: \"kubernetes.io/projected/1a300bba-c70c-4ff1-97d6-7517fa970477-kube-api-access-kmtmd\") pod \"maas-api-revoke-sub-f8cb8d7c-784pk\" (UID: \"1a300bba-c70c-4ff1-97d6-7517fa970477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8cb8d7c-784pk"
Oct 30 20:54:27.704579 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:27.704555    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a300bba-c70c-4ff1-97d6-7517fa970477-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8cb8d7c-784pk\" (UID: \"1a300bba-c70c-4ff1-97d6-7517fa970477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8cb8d7c-784pk"
Oct 30 20:54:27.705130 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:27.705109    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a300bba-c70c-4ff1-97d6-7517fa970477-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8cb8d7c-784pk\" (UID: \"1a300bba-c70c-4ff1-97d6-7517fa970477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8cb8d7c-784pk"
Oct 30 20:54:27.707141 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:27.707118    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a300bba-c70c-4ff1-97d6-7517fa970477-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8cb8d7c-784pk\" (UID: \"1a300bba-c70c-4ff1-97d6-7517fa970477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8cb8d7c-784pk"
Oct 30 20:54:27.761487 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:27.761464    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmtmd\" (UniqueName: \"kubernetes.io/projected/1a300bba-c70c-4ff1-97d6-7517fa970477-kube-api-access-kmtmd\") pod \"maas-api-revoke-sub-f8cb8d7c-784pk\" (UID: \"1a300bba-c70c-4ff1-97d6-7517fa970477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8cb8d7c-784pk"
Oct 30 20:54:27.856183 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:27.856092    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8cb8d7c-784pk"
Oct 30 20:54:28.232025 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:28.232001    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8cb8d7c-784pk"]
Oct 30 20:54:28.234048 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:54:28.234022    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a300bba_c70c_4ff1_97d6_7517fa970477.slice/crio-963bb2c1cd0c1c4c95482e5531c1006ec36745e4dda5c9e9137b0ebc09f57a3c WatchSource:0}: Error finding container 963bb2c1cd0c1c4c95482e5531c1006ec36745e4dda5c9e9137b0ebc09f57a3c: Status 404 returned error can't find the container with id 963bb2c1cd0c1c4c95482e5531c1006ec36745e4dda5c9e9137b0ebc09f57a3c
Oct 30 20:54:28.263572 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:28.263546    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8cb8d7c-784pk" event={"ID":"1a300bba-c70c-4ff1-97d6-7517fa970477","Type":"ContainerStarted","Data":"963bb2c1cd0c1c4c95482e5531c1006ec36745e4dda5c9e9137b0ebc09f57a3c"}
Oct 30 20:54:29.271204 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:29.271162    5713 generic.go:356] "Generic (PLEG): container finished" podID="1a300bba-c70c-4ff1-97d6-7517fa970477" containerID="ba0a8578e50de52468a77ebfe40f9596feda3ada68001bdd9f377bc98411336e" exitCode=0
Oct 30 20:54:29.271611 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:29.271248    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8cb8d7c-784pk" event={"ID":"1a300bba-c70c-4ff1-97d6-7517fa970477","Type":"ContainerDied","Data":"ba0a8578e50de52468a77ebfe40f9596feda3ada68001bdd9f377bc98411336e"}
Oct 30 20:54:30.450895 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:30.450872    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8cb8d7c-784pk"
Oct 30 20:54:30.538373 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:30.538302    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a300bba-c70c-4ff1-97d6-7517fa970477-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a300bba-c70c-4ff1-97d6-7517fa970477-maas-api-xks-ca\") pod \"1a300bba-c70c-4ff1-97d6-7517fa970477\" (UID: \"1a300bba-c70c-4ff1-97d6-7517fa970477\") "
Oct 30 20:54:30.538373 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:30.538350    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a300bba-c70c-4ff1-97d6-7517fa970477-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a300bba-c70c-4ff1-97d6-7517fa970477-maas-api-openshift-ca\") pod \"1a300bba-c70c-4ff1-97d6-7517fa970477\" (UID: \"1a300bba-c70c-4ff1-97d6-7517fa970477\") "
Oct 30 20:54:30.538594 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:30.538576    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a300bba-c70c-4ff1-97d6-7517fa970477-kube-api-access-kmtmd\" (UniqueName: \"kubernetes.io/projected/1a300bba-c70c-4ff1-97d6-7517fa970477-kube-api-access-kmtmd\") pod \"1a300bba-c70c-4ff1-97d6-7517fa970477\" (UID: \"1a300bba-c70c-4ff1-97d6-7517fa970477\") "
Oct 30 20:54:30.538762 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:30.538732    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a300bba-c70c-4ff1-97d6-7517fa970477-maas-api-openshift-ca" pod "1a300bba-c70c-4ff1-97d6-7517fa970477" (UID: "1a300bba-c70c-4ff1-97d6-7517fa970477"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:54:30.539173 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:30.539076    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a300bba-c70c-4ff1-97d6-7517fa970477-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:30.540604 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:30.540577    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a300bba-c70c-4ff1-97d6-7517fa970477-maas-api-xks-ca" pod "1a300bba-c70c-4ff1-97d6-7517fa970477" (UID: "1a300bba-c70c-4ff1-97d6-7517fa970477"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:54:30.540803 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:30.540763    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a300bba-c70c-4ff1-97d6-7517fa970477-kube-api-access-kmtmd" pod "1a300bba-c70c-4ff1-97d6-7517fa970477" (UID: "1a300bba-c70c-4ff1-97d6-7517fa970477"). InnerVolumeSpecName "kube-api-access-kmtmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:30.640298 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:30.640264    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmtmd\" (UniqueName: \"kubernetes.io/projected/1a300bba-c70c-4ff1-97d6-7517fa970477-kube-api-access-kmtmd\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:30.640298 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:30.640295    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a300bba-c70c-4ff1-97d6-7517fa970477-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:31.304812 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:31.285103    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8cb8d7c-784pk"
Oct 30 20:54:31.304812 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:31.285108    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8cb8d7c-784pk" event={"ID":"1a300bba-c70c-4ff1-97d6-7517fa970477","Type":"ContainerDied","Data":"963bb2c1cd0c1c4c95482e5531c1006ec36745e4dda5c9e9137b0ebc09f57a3c"}
Oct 30 20:54:31.304812 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:31.285229    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="963bb2c1cd0c1c4c95482e5531c1006ec36745e4dda5c9e9137b0ebc09f57a3c"
Oct 30 20:54:34.846454 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:34.846422    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67ce5c56-b5hz9"]
Oct 30 20:54:34.846973 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:34.846958    5713 cpu_manager.go:414] "containerMap: removing container" podUID="1a300bba-c70c-4ff1-97d6-7517fa970477" containerName="revoke-keys"
Oct 30 20:54:34.847017 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:34.846975    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a300bba-c70c-4ff1-97d6-7517fa970477" containerName="revoke-keys"
Oct 30 20:54:34.847106 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:34.847075    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a300bba-c70c-4ff1-97d6-7517fa970477" containerName="revoke-keys"
Oct 30 20:54:34.859492 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:34.859468    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ce5c56-b5hz9"
Oct 30 20:54:34.906319 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:34.906282    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67ce5c56-b5hz9"]
Oct 30 20:54:34.952687 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:34.952654    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:54:34.952687 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:34.952672    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:54:34.952919 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:34.952724    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:54:34.952919 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:34.952777    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 20:54:34.993684 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:34.993656    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b1e7680-458d-4ad9-a82a-af10b825e27f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67ce5c56-b5hz9\" (UID: \"3b1e7680-458d-4ad9-a82a-af10b825e27f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ce5c56-b5hz9"
Oct 30 20:54:34.993684 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:34.993699    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b1e7680-458d-4ad9-a82a-af10b825e27f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67ce5c56-b5hz9\" (UID: \"3b1e7680-458d-4ad9-a82a-af10b825e27f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ce5c56-b5hz9"
Oct 30 20:54:34.993914 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:34.993796    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slf2r\" (UniqueName: \"kubernetes.io/projected/3b1e7680-458d-4ad9-a82a-af10b825e27f-kube-api-access-slf2r\") pod \"maas-api-revoke-sub-67ce5c56-b5hz9\" (UID: \"3b1e7680-458d-4ad9-a82a-af10b825e27f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ce5c56-b5hz9"
Oct 30 20:54:35.095313 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:35.095283    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b1e7680-458d-4ad9-a82a-af10b825e27f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67ce5c56-b5hz9\" (UID: \"3b1e7680-458d-4ad9-a82a-af10b825e27f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ce5c56-b5hz9"
Oct 30 20:54:35.095482 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:35.095328    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b1e7680-458d-4ad9-a82a-af10b825e27f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67ce5c56-b5hz9\" (UID: \"3b1e7680-458d-4ad9-a82a-af10b825e27f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ce5c56-b5hz9"
Oct 30 20:54:35.095482 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:35.095354    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slf2r\" (UniqueName: \"kubernetes.io/projected/3b1e7680-458d-4ad9-a82a-af10b825e27f-kube-api-access-slf2r\") pod \"maas-api-revoke-sub-67ce5c56-b5hz9\" (UID: \"3b1e7680-458d-4ad9-a82a-af10b825e27f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ce5c56-b5hz9"
Oct 30 20:54:35.096057 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:35.096039    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b1e7680-458d-4ad9-a82a-af10b825e27f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67ce5c56-b5hz9\" (UID: \"3b1e7680-458d-4ad9-a82a-af10b825e27f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ce5c56-b5hz9"
Oct 30 20:54:35.097796 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:35.097748    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b1e7680-458d-4ad9-a82a-af10b825e27f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67ce5c56-b5hz9\" (UID: \"3b1e7680-458d-4ad9-a82a-af10b825e27f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ce5c56-b5hz9"
Oct 30 20:54:35.152467 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:35.152429    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slf2r\" (UniqueName: \"kubernetes.io/projected/3b1e7680-458d-4ad9-a82a-af10b825e27f-kube-api-access-slf2r\") pod \"maas-api-revoke-sub-67ce5c56-b5hz9\" (UID: \"3b1e7680-458d-4ad9-a82a-af10b825e27f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ce5c56-b5hz9"
Oct 30 20:54:35.172272 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:35.172245    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ce5c56-b5hz9"
Oct 30 20:54:35.551739 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:35.551703    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67ce5c56-b5hz9"]
Oct 30 20:54:35.553295 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:54:35.553274    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b1e7680_458d_4ad9_a82a_af10b825e27f.slice/crio-944068b6ae0c4a8a0eba1419efc15bd9209a8f7e6676449c89c7f38b5a7dde31 WatchSource:0}: Error finding container 944068b6ae0c4a8a0eba1419efc15bd9209a8f7e6676449c89c7f38b5a7dde31: Status 404 returned error can't find the container with id 944068b6ae0c4a8a0eba1419efc15bd9209a8f7e6676449c89c7f38b5a7dde31
Oct 30 20:54:36.319661 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:36.319625    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ce5c56-b5hz9" event={"ID":"3b1e7680-458d-4ad9-a82a-af10b825e27f","Type":"ContainerStarted","Data":"10c0d056d691a6ba1653a30b9eadd54b64e72d77b050b7ff1e4d8751b285225a"}
Oct 30 20:54:36.320070 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:36.319679    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ce5c56-b5hz9" event={"ID":"3b1e7680-458d-4ad9-a82a-af10b825e27f","Type":"ContainerStarted","Data":"944068b6ae0c4a8a0eba1419efc15bd9209a8f7e6676449c89c7f38b5a7dde31"}
Oct 30 20:54:36.435929 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:36.435875    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ce5c56-b5hz9" podStartSLOduration=2.435861125 podStartE2EDuration="2.435861125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:54:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:54:36.432361441 +0000 UTC m=+959.497137840" watchObservedRunningTime="2026-10-30 20:54:36.435861125 +0000 UTC m=+959.500637522"
Oct 30 20:54:37.328751 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:37.328712    5713 generic.go:356] "Generic (PLEG): container finished" podID="3b1e7680-458d-4ad9-a82a-af10b825e27f" containerID="10c0d056d691a6ba1653a30b9eadd54b64e72d77b050b7ff1e4d8751b285225a" exitCode=0
Oct 30 20:54:37.329192 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:37.328823    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ce5c56-b5hz9" event={"ID":"3b1e7680-458d-4ad9-a82a-af10b825e27f","Type":"ContainerDied","Data":"10c0d056d691a6ba1653a30b9eadd54b64e72d77b050b7ff1e4d8751b285225a"}
Oct 30 20:54:38.472802 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:38.472758    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ce5c56-b5hz9"
Oct 30 20:54:38.637102 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:38.637013    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b1e7680-458d-4ad9-a82a-af10b825e27f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b1e7680-458d-4ad9-a82a-af10b825e27f-maas-api-openshift-ca\") pod \"3b1e7680-458d-4ad9-a82a-af10b825e27f\" (UID: \"3b1e7680-458d-4ad9-a82a-af10b825e27f\") "
Oct 30 20:54:38.637102 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:38.637080    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b1e7680-458d-4ad9-a82a-af10b825e27f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b1e7680-458d-4ad9-a82a-af10b825e27f-maas-api-xks-ca\") pod \"3b1e7680-458d-4ad9-a82a-af10b825e27f\" (UID: \"3b1e7680-458d-4ad9-a82a-af10b825e27f\") "
Oct 30 20:54:38.637339 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:38.637206    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b1e7680-458d-4ad9-a82a-af10b825e27f-kube-api-access-slf2r\" (UniqueName: \"kubernetes.io/projected/3b1e7680-458d-4ad9-a82a-af10b825e27f-kube-api-access-slf2r\") pod \"3b1e7680-458d-4ad9-a82a-af10b825e27f\" (UID: \"3b1e7680-458d-4ad9-a82a-af10b825e27f\") "
Oct 30 20:54:38.637448 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:38.637423    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b1e7680-458d-4ad9-a82a-af10b825e27f-maas-api-openshift-ca" pod "3b1e7680-458d-4ad9-a82a-af10b825e27f" (UID: "3b1e7680-458d-4ad9-a82a-af10b825e27f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:54:38.637638 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:38.637620    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b1e7680-458d-4ad9-a82a-af10b825e27f-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:38.639366 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:38.639343    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b1e7680-458d-4ad9-a82a-af10b825e27f-maas-api-xks-ca" pod "3b1e7680-458d-4ad9-a82a-af10b825e27f" (UID: "3b1e7680-458d-4ad9-a82a-af10b825e27f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:54:38.639460 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:38.639399    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b1e7680-458d-4ad9-a82a-af10b825e27f-kube-api-access-slf2r" pod "3b1e7680-458d-4ad9-a82a-af10b825e27f" (UID: "3b1e7680-458d-4ad9-a82a-af10b825e27f"). InnerVolumeSpecName "kube-api-access-slf2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:38.738956 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:38.738927    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b1e7680-458d-4ad9-a82a-af10b825e27f-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:38.738956 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:38.738951    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slf2r\" (UniqueName: \"kubernetes.io/projected/3b1e7680-458d-4ad9-a82a-af10b825e27f-kube-api-access-slf2r\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:39.341995 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:39.341907    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ce5c56-b5hz9"
Oct 30 20:54:39.341995 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:39.341934    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67ce5c56-b5hz9" event={"ID":"3b1e7680-458d-4ad9-a82a-af10b825e27f","Type":"ContainerDied","Data":"944068b6ae0c4a8a0eba1419efc15bd9209a8f7e6676449c89c7f38b5a7dde31"}
Oct 30 20:54:39.341995 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:39.341985    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="944068b6ae0c4a8a0eba1419efc15bd9209a8f7e6676449c89c7f38b5a7dde31"
Oct 30 20:54:42.884474 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:42.884435    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61484a2b-4qggj"]
Oct 30 20:54:42.885058 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:42.885041    5713 cpu_manager.go:414] "containerMap: removing container" podUID="3b1e7680-458d-4ad9-a82a-af10b825e27f" containerName="revoke-keys"
Oct 30 20:54:42.885119 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:42.885060    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b1e7680-458d-4ad9-a82a-af10b825e27f" containerName="revoke-keys"
Oct 30 20:54:42.885180 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:42.885170    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b1e7680-458d-4ad9-a82a-af10b825e27f" containerName="revoke-keys"
Oct 30 20:54:42.888797 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:42.888770    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61484a2b-4qggj"
Oct 30 20:54:42.945242 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:42.945213    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61484a2b-4qggj"]
Oct 30 20:54:42.988477 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:42.988449    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/922e26e1-9fc8-4e31-9897-1c80dcfe2cb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61484a2b-4qggj\" (UID: \"922e26e1-9fc8-4e31-9897-1c80dcfe2cb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61484a2b-4qggj"
Oct 30 20:54:42.988477 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:42.988480    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2zrz\" (UniqueName: \"kubernetes.io/projected/922e26e1-9fc8-4e31-9897-1c80dcfe2cb9-kube-api-access-h2zrz\") pod \"maas-api-revoke-sub-61484a2b-4qggj\" (UID: \"922e26e1-9fc8-4e31-9897-1c80dcfe2cb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61484a2b-4qggj"
Oct 30 20:54:42.988692 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:42.988500    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/922e26e1-9fc8-4e31-9897-1c80dcfe2cb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61484a2b-4qggj\" (UID: \"922e26e1-9fc8-4e31-9897-1c80dcfe2cb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61484a2b-4qggj"
Oct 30 20:54:42.991196 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:42.991178    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:54:42.991257 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:42.991183    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:54:42.991257 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:42.991226    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:54:42.991745 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:42.991728    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 20:54:43.089898 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:43.089857    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/922e26e1-9fc8-4e31-9897-1c80dcfe2cb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61484a2b-4qggj\" (UID: \"922e26e1-9fc8-4e31-9897-1c80dcfe2cb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61484a2b-4qggj"
Oct 30 20:54:43.089898 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:43.089895    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2zrz\" (UniqueName: \"kubernetes.io/projected/922e26e1-9fc8-4e31-9897-1c80dcfe2cb9-kube-api-access-h2zrz\") pod \"maas-api-revoke-sub-61484a2b-4qggj\" (UID: \"922e26e1-9fc8-4e31-9897-1c80dcfe2cb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61484a2b-4qggj"
Oct 30 20:54:43.090121 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:43.089915    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/922e26e1-9fc8-4e31-9897-1c80dcfe2cb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61484a2b-4qggj\" (UID: \"922e26e1-9fc8-4e31-9897-1c80dcfe2cb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61484a2b-4qggj"
Oct 30 20:54:43.090560 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:43.090535    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/922e26e1-9fc8-4e31-9897-1c80dcfe2cb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61484a2b-4qggj\" (UID: \"922e26e1-9fc8-4e31-9897-1c80dcfe2cb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61484a2b-4qggj"
Oct 30 20:54:43.092830 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:43.092810    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/922e26e1-9fc8-4e31-9897-1c80dcfe2cb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61484a2b-4qggj\" (UID: \"922e26e1-9fc8-4e31-9897-1c80dcfe2cb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61484a2b-4qggj"
Oct 30 20:54:43.147302 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:43.147231    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2zrz\" (UniqueName: \"kubernetes.io/projected/922e26e1-9fc8-4e31-9897-1c80dcfe2cb9-kube-api-access-h2zrz\") pod \"maas-api-revoke-sub-61484a2b-4qggj\" (UID: \"922e26e1-9fc8-4e31-9897-1c80dcfe2cb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61484a2b-4qggj"
Oct 30 20:54:43.203470 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:43.203432    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61484a2b-4qggj"
Oct 30 20:54:43.594492 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:43.594464    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61484a2b-4qggj"]
Oct 30 20:54:43.596316 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:54:43.596289    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod922e26e1_9fc8_4e31_9897_1c80dcfe2cb9.slice/crio-9bd2bb46d6a8b9289896404e889e0156506165c4777ddcd34d366bbdf0280fba WatchSource:0}: Error finding container 9bd2bb46d6a8b9289896404e889e0156506165c4777ddcd34d366bbdf0280fba: Status 404 returned error can't find the container with id 9bd2bb46d6a8b9289896404e889e0156506165c4777ddcd34d366bbdf0280fba
Oct 30 20:54:44.374704 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:44.374672    5713 generic.go:356] "Generic (PLEG): container finished" podID="922e26e1-9fc8-4e31-9897-1c80dcfe2cb9" containerID="747d94b4c6d169972205644b026e09606889201ec06c84c719d2165959dce273" exitCode=0
Oct 30 20:54:44.375082 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:44.374760    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61484a2b-4qggj" event={"ID":"922e26e1-9fc8-4e31-9897-1c80dcfe2cb9","Type":"ContainerDied","Data":"747d94b4c6d169972205644b026e09606889201ec06c84c719d2165959dce273"}
Oct 30 20:54:44.375082 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:44.374843    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61484a2b-4qggj" event={"ID":"922e26e1-9fc8-4e31-9897-1c80dcfe2cb9","Type":"ContainerStarted","Data":"9bd2bb46d6a8b9289896404e889e0156506165c4777ddcd34d366bbdf0280fba"}
Oct 30 20:54:45.536563 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:45.536537    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61484a2b-4qggj"
Oct 30 20:54:45.622432 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:45.622387    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/922e26e1-9fc8-4e31-9897-1c80dcfe2cb9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/922e26e1-9fc8-4e31-9897-1c80dcfe2cb9-maas-api-xks-ca\") pod \"922e26e1-9fc8-4e31-9897-1c80dcfe2cb9\" (UID: \"922e26e1-9fc8-4e31-9897-1c80dcfe2cb9\") "
Oct 30 20:54:45.622641 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:45.622545    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/922e26e1-9fc8-4e31-9897-1c80dcfe2cb9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/922e26e1-9fc8-4e31-9897-1c80dcfe2cb9-maas-api-openshift-ca\") pod \"922e26e1-9fc8-4e31-9897-1c80dcfe2cb9\" (UID: \"922e26e1-9fc8-4e31-9897-1c80dcfe2cb9\") "
Oct 30 20:54:45.622712 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:45.622698    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/922e26e1-9fc8-4e31-9897-1c80dcfe2cb9-kube-api-access-h2zrz\" (UniqueName: \"kubernetes.io/projected/922e26e1-9fc8-4e31-9897-1c80dcfe2cb9-kube-api-access-h2zrz\") pod \"922e26e1-9fc8-4e31-9897-1c80dcfe2cb9\" (UID: \"922e26e1-9fc8-4e31-9897-1c80dcfe2cb9\") "
Oct 30 20:54:45.625458 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:45.625429    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/922e26e1-9fc8-4e31-9897-1c80dcfe2cb9-kube-api-access-h2zrz" pod "922e26e1-9fc8-4e31-9897-1c80dcfe2cb9" (UID: "922e26e1-9fc8-4e31-9897-1c80dcfe2cb9"). InnerVolumeSpecName "kube-api-access-h2zrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:45.627759 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:45.627730    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/922e26e1-9fc8-4e31-9897-1c80dcfe2cb9-maas-api-openshift-ca" pod "922e26e1-9fc8-4e31-9897-1c80dcfe2cb9" (UID: "922e26e1-9fc8-4e31-9897-1c80dcfe2cb9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:54:45.628022 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:45.627988    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/922e26e1-9fc8-4e31-9897-1c80dcfe2cb9-maas-api-xks-ca" pod "922e26e1-9fc8-4e31-9897-1c80dcfe2cb9" (UID: "922e26e1-9fc8-4e31-9897-1c80dcfe2cb9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:54:45.724141 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:45.724102    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2zrz\" (UniqueName: \"kubernetes.io/projected/922e26e1-9fc8-4e31-9897-1c80dcfe2cb9-kube-api-access-h2zrz\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:45.724141 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:45.724141    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/922e26e1-9fc8-4e31-9897-1c80dcfe2cb9-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:45.724339 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:45.724152    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/922e26e1-9fc8-4e31-9897-1c80dcfe2cb9-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:46.391081 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:46.391053    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61484a2b-4qggj"
Oct 30 20:54:46.391249 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:46.391052    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61484a2b-4qggj" event={"ID":"922e26e1-9fc8-4e31-9897-1c80dcfe2cb9","Type":"ContainerDied","Data":"9bd2bb46d6a8b9289896404e889e0156506165c4777ddcd34d366bbdf0280fba"}
Oct 30 20:54:46.391249 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:46.391176    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9bd2bb46d6a8b9289896404e889e0156506165c4777ddcd34d366bbdf0280fba"
Oct 30 20:54:54.459083 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:54:54.459046    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 20:54:54.880124 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:54.880023    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69129130-79t4h"]
Oct 30 20:54:54.880551 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:54.880538    5713 cpu_manager.go:414] "containerMap: removing container" podUID="922e26e1-9fc8-4e31-9897-1c80dcfe2cb9" containerName="revoke-keys"
Oct 30 20:54:54.880592 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:54.880553    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="922e26e1-9fc8-4e31-9897-1c80dcfe2cb9" containerName="revoke-keys"
Oct 30 20:54:54.880662 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:54.880652    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="922e26e1-9fc8-4e31-9897-1c80dcfe2cb9" containerName="revoke-keys"
Oct 30 20:54:54.884059 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:54.884044    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69129130-79t4h"
Oct 30 20:54:54.944636 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:54.944596    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30683061-8eeb-4ade-bb56-23c3aac52094-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69129130-79t4h\" (UID: \"30683061-8eeb-4ade-bb56-23c3aac52094\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69129130-79t4h"
Oct 30 20:54:54.944862 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:54.944746    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cmh4\" (UniqueName: \"kubernetes.io/projected/30683061-8eeb-4ade-bb56-23c3aac52094-kube-api-access-4cmh4\") pod \"maas-api-revoke-sub-69129130-79t4h\" (UID: \"30683061-8eeb-4ade-bb56-23c3aac52094\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69129130-79t4h"
Oct 30 20:54:54.944978 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:54.944953    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30683061-8eeb-4ade-bb56-23c3aac52094-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69129130-79t4h\" (UID: \"30683061-8eeb-4ade-bb56-23c3aac52094\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69129130-79t4h"
Oct 30 20:54:54.945895 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:54.945872    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69129130-79t4h"]
Oct 30 20:54:54.991448 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:54.991415    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:54:54.991448 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:54.991431    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:54:54.991448 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:54.991422    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 20:54:54.991448 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:54.991451    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:54:55.045711 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:55.045681    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30683061-8eeb-4ade-bb56-23c3aac52094-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69129130-79t4h\" (UID: \"30683061-8eeb-4ade-bb56-23c3aac52094\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69129130-79t4h"
Oct 30 20:54:55.045919 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:55.045730    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cmh4\" (UniqueName: \"kubernetes.io/projected/30683061-8eeb-4ade-bb56-23c3aac52094-kube-api-access-4cmh4\") pod \"maas-api-revoke-sub-69129130-79t4h\" (UID: \"30683061-8eeb-4ade-bb56-23c3aac52094\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69129130-79t4h"
Oct 30 20:54:55.045919 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:55.045843    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30683061-8eeb-4ade-bb56-23c3aac52094-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69129130-79t4h\" (UID: \"30683061-8eeb-4ade-bb56-23c3aac52094\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69129130-79t4h"
Oct 30 20:54:55.046380 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:55.046356    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30683061-8eeb-4ade-bb56-23c3aac52094-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69129130-79t4h\" (UID: \"30683061-8eeb-4ade-bb56-23c3aac52094\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69129130-79t4h"
Oct 30 20:54:55.048291 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:55.048266    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30683061-8eeb-4ade-bb56-23c3aac52094-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69129130-79t4h\" (UID: \"30683061-8eeb-4ade-bb56-23c3aac52094\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69129130-79t4h"
Oct 30 20:54:55.103038 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:55.103010    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cmh4\" (UniqueName: \"kubernetes.io/projected/30683061-8eeb-4ade-bb56-23c3aac52094-kube-api-access-4cmh4\") pod \"maas-api-revoke-sub-69129130-79t4h\" (UID: \"30683061-8eeb-4ade-bb56-23c3aac52094\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69129130-79t4h"
Oct 30 20:54:55.197914 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:55.197889    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69129130-79t4h"
Oct 30 20:54:55.577129 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:55.577098    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69129130-79t4h"]
Oct 30 20:54:55.578477 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:54:55.578450    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30683061_8eeb_4ade_bb56_23c3aac52094.slice/crio-dbd96b056409d6932abf8d4aeae21ab11dd3cebd4b2859815ab4ade711fee4e0 WatchSource:0}: Error finding container dbd96b056409d6932abf8d4aeae21ab11dd3cebd4b2859815ab4ade711fee4e0: Status 404 returned error can't find the container with id dbd96b056409d6932abf8d4aeae21ab11dd3cebd4b2859815ab4ade711fee4e0
Oct 30 20:54:56.459595 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:56.459566    5713 generic.go:356] "Generic (PLEG): container finished" podID="30683061-8eeb-4ade-bb56-23c3aac52094" containerID="8fa00e8c613e38bc5d1770513f6157d96fd4f9021d1fae57a7249a67307190cc" exitCode=0
Oct 30 20:54:56.459726 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:56.459603    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69129130-79t4h" event={"ID":"30683061-8eeb-4ade-bb56-23c3aac52094","Type":"ContainerDied","Data":"8fa00e8c613e38bc5d1770513f6157d96fd4f9021d1fae57a7249a67307190cc"}
Oct 30 20:54:56.459726 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:56.459639    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69129130-79t4h" event={"ID":"30683061-8eeb-4ade-bb56-23c3aac52094","Type":"ContainerStarted","Data":"dbd96b056409d6932abf8d4aeae21ab11dd3cebd4b2859815ab4ade711fee4e0"}
Oct 30 20:54:57.619576 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:57.619553    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69129130-79t4h"
Oct 30 20:54:57.776400 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:57.776315    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30683061-8eeb-4ade-bb56-23c3aac52094-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30683061-8eeb-4ade-bb56-23c3aac52094-maas-api-openshift-ca\") pod \"30683061-8eeb-4ade-bb56-23c3aac52094\" (UID: \"30683061-8eeb-4ade-bb56-23c3aac52094\") "
Oct 30 20:54:57.776400 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:57.776360    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30683061-8eeb-4ade-bb56-23c3aac52094-kube-api-access-4cmh4\" (UniqueName: \"kubernetes.io/projected/30683061-8eeb-4ade-bb56-23c3aac52094-kube-api-access-4cmh4\") pod \"30683061-8eeb-4ade-bb56-23c3aac52094\" (UID: \"30683061-8eeb-4ade-bb56-23c3aac52094\") "
Oct 30 20:54:57.776589 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:57.776543    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30683061-8eeb-4ade-bb56-23c3aac52094-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30683061-8eeb-4ade-bb56-23c3aac52094-maas-api-xks-ca\") pod \"30683061-8eeb-4ade-bb56-23c3aac52094\" (UID: \"30683061-8eeb-4ade-bb56-23c3aac52094\") "
Oct 30 20:54:57.776782 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:57.776752    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30683061-8eeb-4ade-bb56-23c3aac52094-maas-api-openshift-ca" pod "30683061-8eeb-4ade-bb56-23c3aac52094" (UID: "30683061-8eeb-4ade-bb56-23c3aac52094"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:54:57.776944 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:57.776927    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30683061-8eeb-4ade-bb56-23c3aac52094-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:57.778647 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:57.778619    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30683061-8eeb-4ade-bb56-23c3aac52094-kube-api-access-4cmh4" pod "30683061-8eeb-4ade-bb56-23c3aac52094" (UID: "30683061-8eeb-4ade-bb56-23c3aac52094"). InnerVolumeSpecName "kube-api-access-4cmh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:54:57.778769 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:57.778736    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30683061-8eeb-4ade-bb56-23c3aac52094-maas-api-xks-ca" pod "30683061-8eeb-4ade-bb56-23c3aac52094" (UID: "30683061-8eeb-4ade-bb56-23c3aac52094"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:54:57.877970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:57.877936    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30683061-8eeb-4ade-bb56-23c3aac52094-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:57.877970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:57.877970    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cmh4\" (UniqueName: \"kubernetes.io/projected/30683061-8eeb-4ade-bb56-23c3aac52094-kube-api-access-4cmh4\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:54:58.474440 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:58.474403    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69129130-79t4h"
Oct 30 20:54:58.474631 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:58.474444    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69129130-79t4h" event={"ID":"30683061-8eeb-4ade-bb56-23c3aac52094","Type":"ContainerDied","Data":"dbd96b056409d6932abf8d4aeae21ab11dd3cebd4b2859815ab4ade711fee4e0"}
Oct 30 20:54:58.474631 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:54:58.474491    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dbd96b056409d6932abf8d4aeae21ab11dd3cebd4b2859815ab4ade711fee4e0"
Oct 30 20:55:00.660100 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:00.660067    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62ce42bb-vw2k7"]
Oct 30 20:55:00.660666 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:00.660647    5713 cpu_manager.go:414] "containerMap: removing container" podUID="30683061-8eeb-4ade-bb56-23c3aac52094" containerName="revoke-keys"
Oct 30 20:55:00.660720 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:00.660671    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30683061-8eeb-4ade-bb56-23c3aac52094" containerName="revoke-keys"
Oct 30 20:55:00.660894 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:00.660880    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="30683061-8eeb-4ade-bb56-23c3aac52094" containerName="revoke-keys"
Oct 30 20:55:00.664438 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:00.664421    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ce42bb-vw2k7"
Oct 30 20:55:00.766925 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:00.766889    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:55:00.766925 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:00.766907    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 20:55:00.767184 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:00.766907    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:55:00.767184 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:00.766955    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:55:00.770450 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:00.770429    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62ce42bb-vw2k7"]
Oct 30 20:55:00.813238 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:00.813199    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q99dk\" (UniqueName: \"kubernetes.io/projected/9b97e94c-3c5d-4733-bde5-9edb818d2a6b-kube-api-access-q99dk\") pod \"maas-api-revoke-sub-62ce42bb-vw2k7\" (UID: \"9b97e94c-3c5d-4733-bde5-9edb818d2a6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ce42bb-vw2k7"
Oct 30 20:55:00.813414 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:00.813393    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b97e94c-3c5d-4733-bde5-9edb818d2a6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62ce42bb-vw2k7\" (UID: \"9b97e94c-3c5d-4733-bde5-9edb818d2a6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ce42bb-vw2k7"
Oct 30 20:55:00.813534 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:00.813512    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b97e94c-3c5d-4733-bde5-9edb818d2a6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62ce42bb-vw2k7\" (UID: \"9b97e94c-3c5d-4733-bde5-9edb818d2a6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ce42bb-vw2k7"
Oct 30 20:55:00.914933 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:00.914840    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b97e94c-3c5d-4733-bde5-9edb818d2a6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62ce42bb-vw2k7\" (UID: \"9b97e94c-3c5d-4733-bde5-9edb818d2a6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ce42bb-vw2k7"
Oct 30 20:55:00.915103 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:00.914968    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b97e94c-3c5d-4733-bde5-9edb818d2a6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62ce42bb-vw2k7\" (UID: \"9b97e94c-3c5d-4733-bde5-9edb818d2a6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ce42bb-vw2k7"
Oct 30 20:55:00.915103 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:00.915032    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q99dk\" (UniqueName: \"kubernetes.io/projected/9b97e94c-3c5d-4733-bde5-9edb818d2a6b-kube-api-access-q99dk\") pod \"maas-api-revoke-sub-62ce42bb-vw2k7\" (UID: \"9b97e94c-3c5d-4733-bde5-9edb818d2a6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ce42bb-vw2k7"
Oct 30 20:55:00.915683 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:00.915658    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b97e94c-3c5d-4733-bde5-9edb818d2a6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62ce42bb-vw2k7\" (UID: \"9b97e94c-3c5d-4733-bde5-9edb818d2a6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ce42bb-vw2k7"
Oct 30 20:55:00.917572 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:00.917551    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b97e94c-3c5d-4733-bde5-9edb818d2a6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62ce42bb-vw2k7\" (UID: \"9b97e94c-3c5d-4733-bde5-9edb818d2a6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ce42bb-vw2k7"
Oct 30 20:55:00.973931 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:00.973892    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q99dk\" (UniqueName: \"kubernetes.io/projected/9b97e94c-3c5d-4733-bde5-9edb818d2a6b-kube-api-access-q99dk\") pod \"maas-api-revoke-sub-62ce42bb-vw2k7\" (UID: \"9b97e94c-3c5d-4733-bde5-9edb818d2a6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ce42bb-vw2k7"
Oct 30 20:55:00.978194 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:00.978173    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ce42bb-vw2k7"
Oct 30 20:55:01.357045 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:01.357014    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62ce42bb-vw2k7"]
Oct 30 20:55:01.358495 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:55:01.358470    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b97e94c_3c5d_4733_bde5_9edb818d2a6b.slice/crio-a9cbfcfad71b558c57c79d6e37cb6ab2ff2995ef151ccdddf54e351f1c790ec5 WatchSource:0}: Error finding container a9cbfcfad71b558c57c79d6e37cb6ab2ff2995ef151ccdddf54e351f1c790ec5: Status 404 returned error can't find the container with id a9cbfcfad71b558c57c79d6e37cb6ab2ff2995ef151ccdddf54e351f1c790ec5
Oct 30 20:55:01.495056 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:01.495024    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ce42bb-vw2k7" event={"ID":"9b97e94c-3c5d-4733-bde5-9edb818d2a6b","Type":"ContainerStarted","Data":"a9cbfcfad71b558c57c79d6e37cb6ab2ff2995ef151ccdddf54e351f1c790ec5"}
Oct 30 20:55:02.503256 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:02.503167    5713 generic.go:356] "Generic (PLEG): container finished" podID="9b97e94c-3c5d-4733-bde5-9edb818d2a6b" containerID="c24b023506187f7d2b23eedd5545c0e60acf4b338ef952f2a198f86fdcf5bff8" exitCode=0
Oct 30 20:55:02.503636 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:02.503246    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ce42bb-vw2k7" event={"ID":"9b97e94c-3c5d-4733-bde5-9edb818d2a6b","Type":"ContainerDied","Data":"c24b023506187f7d2b23eedd5545c0e60acf4b338ef952f2a198f86fdcf5bff8"}
Oct 30 20:55:03.661695 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:03.661670    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ce42bb-vw2k7"
Oct 30 20:55:03.850598 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:03.850515    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b97e94c-3c5d-4733-bde5-9edb818d2a6b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b97e94c-3c5d-4733-bde5-9edb818d2a6b-maas-api-xks-ca\") pod \"9b97e94c-3c5d-4733-bde5-9edb818d2a6b\" (UID: \"9b97e94c-3c5d-4733-bde5-9edb818d2a6b\") "
Oct 30 20:55:03.850598 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:03.850586    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b97e94c-3c5d-4733-bde5-9edb818d2a6b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b97e94c-3c5d-4733-bde5-9edb818d2a6b-maas-api-openshift-ca\") pod \"9b97e94c-3c5d-4733-bde5-9edb818d2a6b\" (UID: \"9b97e94c-3c5d-4733-bde5-9edb818d2a6b\") "
Oct 30 20:55:03.850842 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:03.850737    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b97e94c-3c5d-4733-bde5-9edb818d2a6b-kube-api-access-q99dk\" (UniqueName: \"kubernetes.io/projected/9b97e94c-3c5d-4733-bde5-9edb818d2a6b-kube-api-access-q99dk\") pod \"9b97e94c-3c5d-4733-bde5-9edb818d2a6b\" (UID: \"9b97e94c-3c5d-4733-bde5-9edb818d2a6b\") "
Oct 30 20:55:03.850940 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:03.850904    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b97e94c-3c5d-4733-bde5-9edb818d2a6b-maas-api-openshift-ca" pod "9b97e94c-3c5d-4733-bde5-9edb818d2a6b" (UID: "9b97e94c-3c5d-4733-bde5-9edb818d2a6b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:55:03.851182 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:03.851164    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b97e94c-3c5d-4733-bde5-9edb818d2a6b-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:03.852866 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:03.852839    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b97e94c-3c5d-4733-bde5-9edb818d2a6b-maas-api-xks-ca" pod "9b97e94c-3c5d-4733-bde5-9edb818d2a6b" (UID: "9b97e94c-3c5d-4733-bde5-9edb818d2a6b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:55:03.852967 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:03.852935    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b97e94c-3c5d-4733-bde5-9edb818d2a6b-kube-api-access-q99dk" pod "9b97e94c-3c5d-4733-bde5-9edb818d2a6b" (UID: "9b97e94c-3c5d-4733-bde5-9edb818d2a6b"). InnerVolumeSpecName "kube-api-access-q99dk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:55:03.952540 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:03.952502    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q99dk\" (UniqueName: \"kubernetes.io/projected/9b97e94c-3c5d-4733-bde5-9edb818d2a6b-kube-api-access-q99dk\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:03.952540 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:03.952534    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b97e94c-3c5d-4733-bde5-9edb818d2a6b-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:04.519155 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:04.519128    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ce42bb-vw2k7"
Oct 30 20:55:04.519311 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:04.519166    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62ce42bb-vw2k7" event={"ID":"9b97e94c-3c5d-4733-bde5-9edb818d2a6b","Type":"ContainerDied","Data":"a9cbfcfad71b558c57c79d6e37cb6ab2ff2995ef151ccdddf54e351f1c790ec5"}
Oct 30 20:55:04.519311 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:04.519214    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9cbfcfad71b558c57c79d6e37cb6ab2ff2995ef151ccdddf54e351f1c790ec5"
Oct 30 20:55:10.799803 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:10.799756    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-590deeb6-s2789"]
Oct 30 20:55:10.800331 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:10.800316    5713 cpu_manager.go:414] "containerMap: removing container" podUID="9b97e94c-3c5d-4733-bde5-9edb818d2a6b" containerName="revoke-keys"
Oct 30 20:55:10.800376 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:10.800333    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b97e94c-3c5d-4733-bde5-9edb818d2a6b" containerName="revoke-keys"
Oct 30 20:55:10.800426 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:10.800417    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b97e94c-3c5d-4733-bde5-9edb818d2a6b" containerName="revoke-keys"
Oct 30 20:55:10.805423 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:10.805371    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-590deeb6-s2789"
Oct 30 20:55:10.833131 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:10.833101    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f51aea6-3cbd-4c94-b1d6-589a3ca05cc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-590deeb6-s2789\" (UID: \"3f51aea6-3cbd-4c94-b1d6-589a3ca05cc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-590deeb6-s2789"
Oct 30 20:55:10.833291 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:10.833242    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmdpn\" (UniqueName: \"kubernetes.io/projected/3f51aea6-3cbd-4c94-b1d6-589a3ca05cc5-kube-api-access-jmdpn\") pod \"maas-api-revoke-sub-590deeb6-s2789\" (UID: \"3f51aea6-3cbd-4c94-b1d6-589a3ca05cc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-590deeb6-s2789"
Oct 30 20:55:10.833359 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:10.833328    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f51aea6-3cbd-4c94-b1d6-589a3ca05cc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-590deeb6-s2789\" (UID: \"3f51aea6-3cbd-4c94-b1d6-589a3ca05cc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-590deeb6-s2789"
Oct 30 20:55:10.866864 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:10.866829    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-590deeb6-s2789"]
Oct 30 20:55:10.909287 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:10.909256    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:55:10.909287 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:10.909256    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:55:10.912041 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:10.912021    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:55:10.912138 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:10.912103    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 20:55:10.934339 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:10.934294    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f51aea6-3cbd-4c94-b1d6-589a3ca05cc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-590deeb6-s2789\" (UID: \"3f51aea6-3cbd-4c94-b1d6-589a3ca05cc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-590deeb6-s2789"
Oct 30 20:55:10.934556 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:10.934409    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmdpn\" (UniqueName: \"kubernetes.io/projected/3f51aea6-3cbd-4c94-b1d6-589a3ca05cc5-kube-api-access-jmdpn\") pod \"maas-api-revoke-sub-590deeb6-s2789\" (UID: \"3f51aea6-3cbd-4c94-b1d6-589a3ca05cc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-590deeb6-s2789"
Oct 30 20:55:10.934556 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:10.934467    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f51aea6-3cbd-4c94-b1d6-589a3ca05cc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-590deeb6-s2789\" (UID: \"3f51aea6-3cbd-4c94-b1d6-589a3ca05cc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-590deeb6-s2789"
Oct 30 20:55:10.935071 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:10.935051    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f51aea6-3cbd-4c94-b1d6-589a3ca05cc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-590deeb6-s2789\" (UID: \"3f51aea6-3cbd-4c94-b1d6-589a3ca05cc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-590deeb6-s2789"
Oct 30 20:55:10.936919 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:10.936900    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f51aea6-3cbd-4c94-b1d6-589a3ca05cc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-590deeb6-s2789\" (UID: \"3f51aea6-3cbd-4c94-b1d6-589a3ca05cc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-590deeb6-s2789"
Oct 30 20:55:10.990840 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:10.990811    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmdpn\" (UniqueName: \"kubernetes.io/projected/3f51aea6-3cbd-4c94-b1d6-589a3ca05cc5-kube-api-access-jmdpn\") pod \"maas-api-revoke-sub-590deeb6-s2789\" (UID: \"3f51aea6-3cbd-4c94-b1d6-589a3ca05cc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-590deeb6-s2789"
Oct 30 20:55:11.124010 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:11.123934    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-590deeb6-s2789"
Oct 30 20:55:11.506973 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:11.506940    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-590deeb6-s2789"]
Oct 30 20:55:11.508356 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:55:11.508324    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f51aea6_3cbd_4c94_b1d6_589a3ca05cc5.slice/crio-72f83705b0f8f05c7c9617ab0e1daa247bb1f25027ecd6df47244d983d068f91 WatchSource:0}: Error finding container 72f83705b0f8f05c7c9617ab0e1daa247bb1f25027ecd6df47244d983d068f91: Status 404 returned error can't find the container with id 72f83705b0f8f05c7c9617ab0e1daa247bb1f25027ecd6df47244d983d068f91
Oct 30 20:55:11.564822 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:11.564777    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-590deeb6-s2789" event={"ID":"3f51aea6-3cbd-4c94-b1d6-589a3ca05cc5","Type":"ContainerStarted","Data":"72f83705b0f8f05c7c9617ab0e1daa247bb1f25027ecd6df47244d983d068f91"}
Oct 30 20:55:11.628342 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:11.628309    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bad2ec09-tjpll"]
Oct 30 20:55:11.633770 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:11.633752    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad2ec09-tjpll"
Oct 30 20:55:11.690707 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:11.690674    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bad2ec09-tjpll"]
Oct 30 20:55:11.745754 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:11.745719    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd57ccd2-d0d7-491a-8545-12c942aea379-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bad2ec09-tjpll\" (UID: \"dd57ccd2-d0d7-491a-8545-12c942aea379\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad2ec09-tjpll"
Oct 30 20:55:11.745985 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:11.745954    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd57ccd2-d0d7-491a-8545-12c942aea379-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bad2ec09-tjpll\" (UID: \"dd57ccd2-d0d7-491a-8545-12c942aea379\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad2ec09-tjpll"
Oct 30 20:55:11.746967 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:11.746941    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm4x2\" (UniqueName: \"kubernetes.io/projected/dd57ccd2-d0d7-491a-8545-12c942aea379-kube-api-access-nm4x2\") pod \"maas-api-revoke-sub-bad2ec09-tjpll\" (UID: \"dd57ccd2-d0d7-491a-8545-12c942aea379\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad2ec09-tjpll"
Oct 30 20:55:11.848579 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:11.848548    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd57ccd2-d0d7-491a-8545-12c942aea379-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bad2ec09-tjpll\" (UID: \"dd57ccd2-d0d7-491a-8545-12c942aea379\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad2ec09-tjpll"
Oct 30 20:55:11.848931 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:11.848662    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm4x2\" (UniqueName: \"kubernetes.io/projected/dd57ccd2-d0d7-491a-8545-12c942aea379-kube-api-access-nm4x2\") pod \"maas-api-revoke-sub-bad2ec09-tjpll\" (UID: \"dd57ccd2-d0d7-491a-8545-12c942aea379\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad2ec09-tjpll"
Oct 30 20:55:11.848931 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:11.848730    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd57ccd2-d0d7-491a-8545-12c942aea379-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bad2ec09-tjpll\" (UID: \"dd57ccd2-d0d7-491a-8545-12c942aea379\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad2ec09-tjpll"
Oct 30 20:55:11.849486 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:11.849461    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd57ccd2-d0d7-491a-8545-12c942aea379-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bad2ec09-tjpll\" (UID: \"dd57ccd2-d0d7-491a-8545-12c942aea379\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad2ec09-tjpll"
Oct 30 20:55:11.851081 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:11.851060    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd57ccd2-d0d7-491a-8545-12c942aea379-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bad2ec09-tjpll\" (UID: \"dd57ccd2-d0d7-491a-8545-12c942aea379\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad2ec09-tjpll"
Oct 30 20:55:11.905698 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:11.905668    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm4x2\" (UniqueName: \"kubernetes.io/projected/dd57ccd2-d0d7-491a-8545-12c942aea379-kube-api-access-nm4x2\") pod \"maas-api-revoke-sub-bad2ec09-tjpll\" (UID: \"dd57ccd2-d0d7-491a-8545-12c942aea379\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad2ec09-tjpll"
Oct 30 20:55:11.947006 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:11.946973    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad2ec09-tjpll"
Oct 30 20:55:12.325653 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:12.325622    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bad2ec09-tjpll"]
Oct 30 20:55:12.326867 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:55:12.326836    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd57ccd2_d0d7_491a_8545_12c942aea379.slice/crio-cce588107d75556439cac821689f001cebd68c9438b4a431847c1445ca0b80de WatchSource:0}: Error finding container cce588107d75556439cac821689f001cebd68c9438b4a431847c1445ca0b80de: Status 404 returned error can't find the container with id cce588107d75556439cac821689f001cebd68c9438b4a431847c1445ca0b80de
Oct 30 20:55:12.572966 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:12.572930    5713 generic.go:356] "Generic (PLEG): container finished" podID="3f51aea6-3cbd-4c94-b1d6-589a3ca05cc5" containerID="e8ca99fafd4d8b072fde632768f0eee6c06dfaf96e7070ec5b1993531976c777" exitCode=0
Oct 30 20:55:12.573156 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:12.573020    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-590deeb6-s2789" event={"ID":"3f51aea6-3cbd-4c94-b1d6-589a3ca05cc5","Type":"ContainerDied","Data":"e8ca99fafd4d8b072fde632768f0eee6c06dfaf96e7070ec5b1993531976c777"}
Oct 30 20:55:12.574466 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:12.574412    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad2ec09-tjpll" event={"ID":"dd57ccd2-d0d7-491a-8545-12c942aea379","Type":"ContainerStarted","Data":"cce588107d75556439cac821689f001cebd68c9438b4a431847c1445ca0b80de"}
Oct 30 20:55:13.583128 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:13.583037    5713 generic.go:356] "Generic (PLEG): container finished" podID="dd57ccd2-d0d7-491a-8545-12c942aea379" containerID="e220454cb4a55cfb39f5123ba87c80aba4f903ca022aac0c5c79257f4ff30fdc" exitCode=0
Oct 30 20:55:13.583586 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:13.583119    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad2ec09-tjpll" event={"ID":"dd57ccd2-d0d7-491a-8545-12c942aea379","Type":"ContainerDied","Data":"e220454cb4a55cfb39f5123ba87c80aba4f903ca022aac0c5c79257f4ff30fdc"}
Oct 30 20:55:13.728817 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:13.728777    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-590deeb6-s2789"
Oct 30 20:55:13.770898 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:13.770863    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f51aea6-3cbd-4c94-b1d6-589a3ca05cc5-kube-api-access-jmdpn\" (UniqueName: \"kubernetes.io/projected/3f51aea6-3cbd-4c94-b1d6-589a3ca05cc5-kube-api-access-jmdpn\") pod \"3f51aea6-3cbd-4c94-b1d6-589a3ca05cc5\" (UID: \"3f51aea6-3cbd-4c94-b1d6-589a3ca05cc5\") "
Oct 30 20:55:13.771044 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:13.771025    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f51aea6-3cbd-4c94-b1d6-589a3ca05cc5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f51aea6-3cbd-4c94-b1d6-589a3ca05cc5-maas-api-openshift-ca\") pod \"3f51aea6-3cbd-4c94-b1d6-589a3ca05cc5\" (UID: \"3f51aea6-3cbd-4c94-b1d6-589a3ca05cc5\") "
Oct 30 20:55:13.771108 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:13.771094    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f51aea6-3cbd-4c94-b1d6-589a3ca05cc5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f51aea6-3cbd-4c94-b1d6-589a3ca05cc5-maas-api-xks-ca\") pod \"3f51aea6-3cbd-4c94-b1d6-589a3ca05cc5\" (UID: \"3f51aea6-3cbd-4c94-b1d6-589a3ca05cc5\") "
Oct 30 20:55:13.771430 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:13.771407    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f51aea6-3cbd-4c94-b1d6-589a3ca05cc5-maas-api-openshift-ca" pod "3f51aea6-3cbd-4c94-b1d6-589a3ca05cc5" (UID: "3f51aea6-3cbd-4c94-b1d6-589a3ca05cc5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:55:13.771529 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:13.771514    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f51aea6-3cbd-4c94-b1d6-589a3ca05cc5-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:13.773173 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:13.773150    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f51aea6-3cbd-4c94-b1d6-589a3ca05cc5-kube-api-access-jmdpn" pod "3f51aea6-3cbd-4c94-b1d6-589a3ca05cc5" (UID: "3f51aea6-3cbd-4c94-b1d6-589a3ca05cc5"). InnerVolumeSpecName "kube-api-access-jmdpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:55:13.773277 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:13.773254    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f51aea6-3cbd-4c94-b1d6-589a3ca05cc5-maas-api-xks-ca" pod "3f51aea6-3cbd-4c94-b1d6-589a3ca05cc5" (UID: "3f51aea6-3cbd-4c94-b1d6-589a3ca05cc5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:55:13.872483 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:13.872407    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f51aea6-3cbd-4c94-b1d6-589a3ca05cc5-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:13.872483 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:13.872438    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmdpn\" (UniqueName: \"kubernetes.io/projected/3f51aea6-3cbd-4c94-b1d6-589a3ca05cc5-kube-api-access-jmdpn\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:14.595026 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:14.594992    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-590deeb6-s2789"
Oct 30 20:55:14.595492 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:14.595018    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-590deeb6-s2789" event={"ID":"3f51aea6-3cbd-4c94-b1d6-589a3ca05cc5","Type":"ContainerDied","Data":"72f83705b0f8f05c7c9617ab0e1daa247bb1f25027ecd6df47244d983d068f91"}
Oct 30 20:55:14.595492 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:14.595106    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72f83705b0f8f05c7c9617ab0e1daa247bb1f25027ecd6df47244d983d068f91"
Oct 30 20:55:14.864003 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:14.863926    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad2ec09-tjpll"
Oct 30 20:55:14.987813 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:14.986931    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd57ccd2-d0d7-491a-8545-12c942aea379-kube-api-access-nm4x2\" (UniqueName: \"kubernetes.io/projected/dd57ccd2-d0d7-491a-8545-12c942aea379-kube-api-access-nm4x2\") pod \"dd57ccd2-d0d7-491a-8545-12c942aea379\" (UID: \"dd57ccd2-d0d7-491a-8545-12c942aea379\") "
Oct 30 20:55:14.987813 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:14.986992    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd57ccd2-d0d7-491a-8545-12c942aea379-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd57ccd2-d0d7-491a-8545-12c942aea379-maas-api-xks-ca\") pod \"dd57ccd2-d0d7-491a-8545-12c942aea379\" (UID: \"dd57ccd2-d0d7-491a-8545-12c942aea379\") "
Oct 30 20:55:14.987813 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:14.987158    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd57ccd2-d0d7-491a-8545-12c942aea379-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd57ccd2-d0d7-491a-8545-12c942aea379-maas-api-openshift-ca\") pod \"dd57ccd2-d0d7-491a-8545-12c942aea379\" (UID: \"dd57ccd2-d0d7-491a-8545-12c942aea379\") "
Oct 30 20:55:14.988142 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:14.988089    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd57ccd2-d0d7-491a-8545-12c942aea379-maas-api-openshift-ca" pod "dd57ccd2-d0d7-491a-8545-12c942aea379" (UID: "dd57ccd2-d0d7-491a-8545-12c942aea379"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:55:15.004736 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:15.004701    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd57ccd2-d0d7-491a-8545-12c942aea379-maas-api-xks-ca" pod "dd57ccd2-d0d7-491a-8545-12c942aea379" (UID: "dd57ccd2-d0d7-491a-8545-12c942aea379"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:55:15.009138 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:15.008299    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd57ccd2-d0d7-491a-8545-12c942aea379-kube-api-access-nm4x2" pod "dd57ccd2-d0d7-491a-8545-12c942aea379" (UID: "dd57ccd2-d0d7-491a-8545-12c942aea379"). InnerVolumeSpecName "kube-api-access-nm4x2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:55:15.097546 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:15.097511    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd57ccd2-d0d7-491a-8545-12c942aea379-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:15.098304 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:15.098281    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm4x2\" (UniqueName: \"kubernetes.io/projected/dd57ccd2-d0d7-491a-8545-12c942aea379-kube-api-access-nm4x2\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:15.098473 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:15.098455    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd57ccd2-d0d7-491a-8545-12c942aea379-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:15.610314 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:15.610289    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad2ec09-tjpll"
Oct 30 20:55:15.610676 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:15.610289    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad2ec09-tjpll" event={"ID":"dd57ccd2-d0d7-491a-8545-12c942aea379","Type":"ContainerDied","Data":"cce588107d75556439cac821689f001cebd68c9438b4a431847c1445ca0b80de"}
Oct 30 20:55:15.610676 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:15.610409    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cce588107d75556439cac821689f001cebd68c9438b4a431847c1445ca0b80de"
Oct 30 20:55:18.247996 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:18.247913    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03415aa4-pckkr"]
Oct 30 20:55:18.248445 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:18.248431    5713 cpu_manager.go:414] "containerMap: removing container" podUID="3f51aea6-3cbd-4c94-b1d6-589a3ca05cc5" containerName="revoke-keys"
Oct 30 20:55:18.248485 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:18.248447    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f51aea6-3cbd-4c94-b1d6-589a3ca05cc5" containerName="revoke-keys"
Oct 30 20:55:18.248485 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:18.248457    5713 cpu_manager.go:414] "containerMap: removing container" podUID="dd57ccd2-d0d7-491a-8545-12c942aea379" containerName="revoke-keys"
Oct 30 20:55:18.248485 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:18.248462    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd57ccd2-d0d7-491a-8545-12c942aea379" containerName="revoke-keys"
Oct 30 20:55:18.248580 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:18.248576    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f51aea6-3cbd-4c94-b1d6-589a3ca05cc5" containerName="revoke-keys"
Oct 30 20:55:18.248611 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:18.248584    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd57ccd2-d0d7-491a-8545-12c942aea379" containerName="revoke-keys"
Oct 30 20:55:18.253830 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:18.253002    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03415aa4-pckkr"
Oct 30 20:55:18.356016 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:18.355982    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:55:18.356016 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:18.356012    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:55:18.356264 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:18.355981    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:55:18.356264 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:18.356065    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 20:55:18.356547 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:18.356524    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/960607a7-c808-45b5-aa2f-00d454c2dac9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03415aa4-pckkr\" (UID: \"960607a7-c808-45b5-aa2f-00d454c2dac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03415aa4-pckkr"
Oct 30 20:55:18.356614 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:18.356601    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/960607a7-c808-45b5-aa2f-00d454c2dac9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03415aa4-pckkr\" (UID: \"960607a7-c808-45b5-aa2f-00d454c2dac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03415aa4-pckkr"
Oct 30 20:55:18.356677 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:18.356660    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqb6l\" (UniqueName: \"kubernetes.io/projected/960607a7-c808-45b5-aa2f-00d454c2dac9-kube-api-access-sqb6l\") pod \"maas-api-revoke-sub-03415aa4-pckkr\" (UID: \"960607a7-c808-45b5-aa2f-00d454c2dac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03415aa4-pckkr"
Oct 30 20:55:18.359502 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:18.359473    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03415aa4-pckkr"]
Oct 30 20:55:18.457457 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:18.457417    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/960607a7-c808-45b5-aa2f-00d454c2dac9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03415aa4-pckkr\" (UID: \"960607a7-c808-45b5-aa2f-00d454c2dac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03415aa4-pckkr"
Oct 30 20:55:18.457635 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:18.457467    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqb6l\" (UniqueName: \"kubernetes.io/projected/960607a7-c808-45b5-aa2f-00d454c2dac9-kube-api-access-sqb6l\") pod \"maas-api-revoke-sub-03415aa4-pckkr\" (UID: \"960607a7-c808-45b5-aa2f-00d454c2dac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03415aa4-pckkr"
Oct 30 20:55:18.457635 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:18.457610    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/960607a7-c808-45b5-aa2f-00d454c2dac9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03415aa4-pckkr\" (UID: \"960607a7-c808-45b5-aa2f-00d454c2dac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03415aa4-pckkr"
Oct 30 20:55:18.458235 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:18.458208    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/960607a7-c808-45b5-aa2f-00d454c2dac9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03415aa4-pckkr\" (UID: \"960607a7-c808-45b5-aa2f-00d454c2dac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03415aa4-pckkr"
Oct 30 20:55:18.460009 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:18.459981    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/960607a7-c808-45b5-aa2f-00d454c2dac9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03415aa4-pckkr\" (UID: \"960607a7-c808-45b5-aa2f-00d454c2dac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03415aa4-pckkr"
Oct 30 20:55:18.515620 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:18.515546    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqb6l\" (UniqueName: \"kubernetes.io/projected/960607a7-c808-45b5-aa2f-00d454c2dac9-kube-api-access-sqb6l\") pod \"maas-api-revoke-sub-03415aa4-pckkr\" (UID: \"960607a7-c808-45b5-aa2f-00d454c2dac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03415aa4-pckkr"
Oct 30 20:55:18.569079 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:18.569039    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03415aa4-pckkr"
Oct 30 20:55:18.952192 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:18.952162    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03415aa4-pckkr"]
Oct 30 20:55:18.953205 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:55:18.953180    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod960607a7_c808_45b5_aa2f_00d454c2dac9.slice/crio-e5ab130ed012a014b9f816dddc8b1fe10ac8f2de9bf2c816a6a979b4c7bfe5f4 WatchSource:0}: Error finding container e5ab130ed012a014b9f816dddc8b1fe10ac8f2de9bf2c816a6a979b4c7bfe5f4: Status 404 returned error can't find the container with id e5ab130ed012a014b9f816dddc8b1fe10ac8f2de9bf2c816a6a979b4c7bfe5f4
Oct 30 20:55:19.653303 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:19.653217    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03415aa4-pckkr" event={"ID":"960607a7-c808-45b5-aa2f-00d454c2dac9","Type":"ContainerStarted","Data":"f8344bb7dc6199a55a12b9e777476a9bbc8b93dc5cf98e356493dfb0a73ec515"}
Oct 30 20:55:19.653303 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:19.653267    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03415aa4-pckkr" event={"ID":"960607a7-c808-45b5-aa2f-00d454c2dac9","Type":"ContainerStarted","Data":"e5ab130ed012a014b9f816dddc8b1fe10ac8f2de9bf2c816a6a979b4c7bfe5f4"}
Oct 30 20:55:19.768441 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:19.768387    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03415aa4-pckkr" podStartSLOduration=1.768369734 podStartE2EDuration="1.768369734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:55:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:55:19.765664572 +0000 UTC m=+1002.830440972" watchObservedRunningTime="2026-10-30 20:55:19.768369734 +0000 UTC m=+1002.833146134"
Oct 30 20:55:20.661109 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:20.661072    5713 generic.go:356] "Generic (PLEG): container finished" podID="960607a7-c808-45b5-aa2f-00d454c2dac9" containerID="f8344bb7dc6199a55a12b9e777476a9bbc8b93dc5cf98e356493dfb0a73ec515" exitCode=0
Oct 30 20:55:20.661532 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:20.661149    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03415aa4-pckkr" event={"ID":"960607a7-c808-45b5-aa2f-00d454c2dac9","Type":"ContainerDied","Data":"f8344bb7dc6199a55a12b9e777476a9bbc8b93dc5cf98e356493dfb0a73ec515"}
Oct 30 20:55:21.838317 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:21.838285    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03415aa4-pckkr"
Oct 30 20:55:22.000294 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:22.000263    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/960607a7-c808-45b5-aa2f-00d454c2dac9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/960607a7-c808-45b5-aa2f-00d454c2dac9-maas-api-xks-ca\") pod \"960607a7-c808-45b5-aa2f-00d454c2dac9\" (UID: \"960607a7-c808-45b5-aa2f-00d454c2dac9\") "
Oct 30 20:55:22.000472 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:22.000429    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/960607a7-c808-45b5-aa2f-00d454c2dac9-kube-api-access-sqb6l\" (UniqueName: \"kubernetes.io/projected/960607a7-c808-45b5-aa2f-00d454c2dac9-kube-api-access-sqb6l\") pod \"960607a7-c808-45b5-aa2f-00d454c2dac9\" (UID: \"960607a7-c808-45b5-aa2f-00d454c2dac9\") "
Oct 30 20:55:22.000472 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:22.000467    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/960607a7-c808-45b5-aa2f-00d454c2dac9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/960607a7-c808-45b5-aa2f-00d454c2dac9-maas-api-openshift-ca\") pod \"960607a7-c808-45b5-aa2f-00d454c2dac9\" (UID: \"960607a7-c808-45b5-aa2f-00d454c2dac9\") "
Oct 30 20:55:22.000973 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:22.000888    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/960607a7-c808-45b5-aa2f-00d454c2dac9-maas-api-openshift-ca" pod "960607a7-c808-45b5-aa2f-00d454c2dac9" (UID: "960607a7-c808-45b5-aa2f-00d454c2dac9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:55:22.002513 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:22.002487    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/960607a7-c808-45b5-aa2f-00d454c2dac9-maas-api-xks-ca" pod "960607a7-c808-45b5-aa2f-00d454c2dac9" (UID: "960607a7-c808-45b5-aa2f-00d454c2dac9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:55:22.002705 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:22.002686    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/960607a7-c808-45b5-aa2f-00d454c2dac9-kube-api-access-sqb6l" pod "960607a7-c808-45b5-aa2f-00d454c2dac9" (UID: "960607a7-c808-45b5-aa2f-00d454c2dac9"). InnerVolumeSpecName "kube-api-access-sqb6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:55:22.102622 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:22.102583    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqb6l\" (UniqueName: \"kubernetes.io/projected/960607a7-c808-45b5-aa2f-00d454c2dac9-kube-api-access-sqb6l\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:22.102622 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:22.102614    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/960607a7-c808-45b5-aa2f-00d454c2dac9-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:22.102622 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:22.102629    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/960607a7-c808-45b5-aa2f-00d454c2dac9-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:22.676285 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:22.676254    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03415aa4-pckkr"
Oct 30 20:55:22.676440 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:22.676254    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03415aa4-pckkr" event={"ID":"960607a7-c808-45b5-aa2f-00d454c2dac9","Type":"ContainerDied","Data":"e5ab130ed012a014b9f816dddc8b1fe10ac8f2de9bf2c816a6a979b4c7bfe5f4"}
Oct 30 20:55:22.676440 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:22.676377    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5ab130ed012a014b9f816dddc8b1fe10ac8f2de9bf2c816a6a979b4c7bfe5f4"
Oct 30 20:55:24.458827 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:55:24.458771    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 20:55:27.924269 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:27.924232    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49ff7895-7z6ww"]
Oct 30 20:55:27.924847 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:27.924831    5713 cpu_manager.go:414] "containerMap: removing container" podUID="960607a7-c808-45b5-aa2f-00d454c2dac9" containerName="revoke-keys"
Oct 30 20:55:27.924920 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:27.924848    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="960607a7-c808-45b5-aa2f-00d454c2dac9" containerName="revoke-keys"
Oct 30 20:55:27.924976 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:27.924952    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="960607a7-c808-45b5-aa2f-00d454c2dac9" containerName="revoke-keys"
Oct 30 20:55:27.928498 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:27.928479    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49ff7895-7z6ww"
Oct 30 20:55:27.975149 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:27.975106    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl822\" (UniqueName: \"kubernetes.io/projected/c81ff9be-12b4-41f3-8ea2-f86a043f5e2a-kube-api-access-xl822\") pod \"maas-api-revoke-sub-49ff7895-7z6ww\" (UID: \"c81ff9be-12b4-41f3-8ea2-f86a043f5e2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49ff7895-7z6ww"
Oct 30 20:55:27.975320 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:27.975287    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c81ff9be-12b4-41f3-8ea2-f86a043f5e2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49ff7895-7z6ww\" (UID: \"c81ff9be-12b4-41f3-8ea2-f86a043f5e2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49ff7895-7z6ww"
Oct 30 20:55:27.975395 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:27.975378    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c81ff9be-12b4-41f3-8ea2-f86a043f5e2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49ff7895-7z6ww\" (UID: \"c81ff9be-12b4-41f3-8ea2-f86a043f5e2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49ff7895-7z6ww"
Oct 30 20:55:28.030967 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:28.030932    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:55:28.031139 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:28.030937    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:55:28.031139 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:28.031042    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:55:28.031139 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:28.031042    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 20:55:28.034334 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:28.034315    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49ff7895-7z6ww"]
Oct 30 20:55:28.076920 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:28.076890    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c81ff9be-12b4-41f3-8ea2-f86a043f5e2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49ff7895-7z6ww\" (UID: \"c81ff9be-12b4-41f3-8ea2-f86a043f5e2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49ff7895-7z6ww"
Oct 30 20:55:28.077087 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:28.076991    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl822\" (UniqueName: \"kubernetes.io/projected/c81ff9be-12b4-41f3-8ea2-f86a043f5e2a-kube-api-access-xl822\") pod \"maas-api-revoke-sub-49ff7895-7z6ww\" (UID: \"c81ff9be-12b4-41f3-8ea2-f86a043f5e2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49ff7895-7z6ww"
Oct 30 20:55:28.077131 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:28.077119    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c81ff9be-12b4-41f3-8ea2-f86a043f5e2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49ff7895-7z6ww\" (UID: \"c81ff9be-12b4-41f3-8ea2-f86a043f5e2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49ff7895-7z6ww"
Oct 30 20:55:28.077720 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:28.077693    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c81ff9be-12b4-41f3-8ea2-f86a043f5e2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49ff7895-7z6ww\" (UID: \"c81ff9be-12b4-41f3-8ea2-f86a043f5e2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49ff7895-7z6ww"
Oct 30 20:55:28.079385 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:28.079361    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c81ff9be-12b4-41f3-8ea2-f86a043f5e2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49ff7895-7z6ww\" (UID: \"c81ff9be-12b4-41f3-8ea2-f86a043f5e2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49ff7895-7z6ww"
Oct 30 20:55:28.134354 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:28.134325    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl822\" (UniqueName: \"kubernetes.io/projected/c81ff9be-12b4-41f3-8ea2-f86a043f5e2a-kube-api-access-xl822\") pod \"maas-api-revoke-sub-49ff7895-7z6ww\" (UID: \"c81ff9be-12b4-41f3-8ea2-f86a043f5e2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49ff7895-7z6ww"
Oct 30 20:55:28.242559 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:28.242521    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49ff7895-7z6ww"
Oct 30 20:55:28.620153 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:28.620119    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49ff7895-7z6ww"]
Oct 30 20:55:28.620959 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:55:28.620931    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc81ff9be_12b4_41f3_8ea2_f86a043f5e2a.slice/crio-007f465b69165fbf59881b6dc59266045d46d368482ca9cf73f32ab68ea424c9 WatchSource:0}: Error finding container 007f465b69165fbf59881b6dc59266045d46d368482ca9cf73f32ab68ea424c9: Status 404 returned error can't find the container with id 007f465b69165fbf59881b6dc59266045d46d368482ca9cf73f32ab68ea424c9
Oct 30 20:55:28.715680 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:28.715648    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49ff7895-7z6ww" event={"ID":"c81ff9be-12b4-41f3-8ea2-f86a043f5e2a","Type":"ContainerStarted","Data":"007f465b69165fbf59881b6dc59266045d46d368482ca9cf73f32ab68ea424c9"}
Oct 30 20:55:28.964575 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:28.964544    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h"]
Oct 30 20:55:28.968390 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:28.968371    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h"
Oct 30 20:55:28.987731 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:28.987704    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66df6c04-8987-4f07-9fab-4e80cf71f6a2-workload-socket\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h\" (UID: \"66df6c04-8987-4f07-9fab-4e80cf71f6a2\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h"
Oct 30 20:55:28.987907 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:28.987771    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66df6c04-8987-4f07-9fab-4e80cf71f6a2-istiod-ca-cert\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h\" (UID: \"66df6c04-8987-4f07-9fab-4e80cf71f6a2\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h"
Oct 30 20:55:28.987982 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:28.987958    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66df6c04-8987-4f07-9fab-4e80cf71f6a2-istio-envoy\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h\" (UID: \"66df6c04-8987-4f07-9fab-4e80cf71f6a2\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h"
Oct 30 20:55:28.988032 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:28.987995    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66df6c04-8987-4f07-9fab-4e80cf71f6a2-credential-socket\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h\" (UID: \"66df6c04-8987-4f07-9fab-4e80cf71f6a2\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h"
Oct 30 20:55:28.988194 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:28.988119    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66df6c04-8987-4f07-9fab-4e80cf71f6a2-workload-certs\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h\" (UID: \"66df6c04-8987-4f07-9fab-4e80cf71f6a2\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h"
Oct 30 20:55:28.988194 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:28.988159    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66df6c04-8987-4f07-9fab-4e80cf71f6a2-istio-podinfo\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h\" (UID: \"66df6c04-8987-4f07-9fab-4e80cf71f6a2\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h"
Oct 30 20:55:28.988194 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:28.988180    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw2k5\" (UniqueName: \"kubernetes.io/projected/66df6c04-8987-4f07-9fab-4e80cf71f6a2-kube-api-access-vw2k5\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h\" (UID: \"66df6c04-8987-4f07-9fab-4e80cf71f6a2\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h"
Oct 30 20:55:28.988291 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:28.988245    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66df6c04-8987-4f07-9fab-4e80cf71f6a2-istio-token\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h\" (UID: \"66df6c04-8987-4f07-9fab-4e80cf71f6a2\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h"
Oct 30 20:55:28.988291 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:28.988267    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66df6c04-8987-4f07-9fab-4e80cf71f6a2-istio-data\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h\" (UID: \"66df6c04-8987-4f07-9fab-4e80cf71f6a2\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h"
Oct 30 20:55:29.021336 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:29.021308    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-f9f277-openshift-default-dockercfg-5xmn8\""
Oct 30 20:55:29.075184 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:29.075151    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h"]
Oct 30 20:55:29.090802 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:29.090749    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66df6c04-8987-4f07-9fab-4e80cf71f6a2-istio-podinfo\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h\" (UID: \"66df6c04-8987-4f07-9fab-4e80cf71f6a2\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h"
Oct 30 20:55:29.090983 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:29.090812    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw2k5\" (UniqueName: \"kubernetes.io/projected/66df6c04-8987-4f07-9fab-4e80cf71f6a2-kube-api-access-vw2k5\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h\" (UID: \"66df6c04-8987-4f07-9fab-4e80cf71f6a2\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h"
Oct 30 20:55:29.090983 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:29.090850    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66df6c04-8987-4f07-9fab-4e80cf71f6a2-istio-token\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h\" (UID: \"66df6c04-8987-4f07-9fab-4e80cf71f6a2\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h"
Oct 30 20:55:29.090983 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:29.090877    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66df6c04-8987-4f07-9fab-4e80cf71f6a2-istio-data\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h\" (UID: \"66df6c04-8987-4f07-9fab-4e80cf71f6a2\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h"
Oct 30 20:55:29.090983 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:29.090904    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66df6c04-8987-4f07-9fab-4e80cf71f6a2-workload-socket\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h\" (UID: \"66df6c04-8987-4f07-9fab-4e80cf71f6a2\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h"
Oct 30 20:55:29.091195 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:29.090993    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66df6c04-8987-4f07-9fab-4e80cf71f6a2-istiod-ca-cert\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h\" (UID: \"66df6c04-8987-4f07-9fab-4e80cf71f6a2\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h"
Oct 30 20:55:29.091195 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:29.091084    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66df6c04-8987-4f07-9fab-4e80cf71f6a2-istio-envoy\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h\" (UID: \"66df6c04-8987-4f07-9fab-4e80cf71f6a2\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h"
Oct 30 20:55:29.091195 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:29.091119    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66df6c04-8987-4f07-9fab-4e80cf71f6a2-credential-socket\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h\" (UID: \"66df6c04-8987-4f07-9fab-4e80cf71f6a2\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h"
Oct 30 20:55:29.091827 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:29.091302    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66df6c04-8987-4f07-9fab-4e80cf71f6a2-workload-certs\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h\" (UID: \"66df6c04-8987-4f07-9fab-4e80cf71f6a2\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h"
Oct 30 20:55:29.091827 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:29.091731    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66df6c04-8987-4f07-9fab-4e80cf71f6a2-istio-data\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h\" (UID: \"66df6c04-8987-4f07-9fab-4e80cf71f6a2\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h"
Oct 30 20:55:29.091827 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:29.091734    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66df6c04-8987-4f07-9fab-4e80cf71f6a2-workload-certs\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h\" (UID: \"66df6c04-8987-4f07-9fab-4e80cf71f6a2\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h"
Oct 30 20:55:29.092191 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:29.092165    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66df6c04-8987-4f07-9fab-4e80cf71f6a2-workload-socket\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h\" (UID: \"66df6c04-8987-4f07-9fab-4e80cf71f6a2\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h"
Oct 30 20:55:29.092344 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:29.092321    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66df6c04-8987-4f07-9fab-4e80cf71f6a2-istiod-ca-cert\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h\" (UID: \"66df6c04-8987-4f07-9fab-4e80cf71f6a2\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h"
Oct 30 20:55:29.092480 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:29.092457    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66df6c04-8987-4f07-9fab-4e80cf71f6a2-credential-socket\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h\" (UID: \"66df6c04-8987-4f07-9fab-4e80cf71f6a2\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h"
Oct 30 20:55:29.094217 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:29.094188    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66df6c04-8987-4f07-9fab-4e80cf71f6a2-istio-podinfo\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h\" (UID: \"66df6c04-8987-4f07-9fab-4e80cf71f6a2\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h"
Oct 30 20:55:29.095064 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:29.095042    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66df6c04-8987-4f07-9fab-4e80cf71f6a2-istio-envoy\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h\" (UID: \"66df6c04-8987-4f07-9fab-4e80cf71f6a2\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h"
Oct 30 20:55:29.148715 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:29.148678    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw2k5\" (UniqueName: \"kubernetes.io/projected/66df6c04-8987-4f07-9fab-4e80cf71f6a2-kube-api-access-vw2k5\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h\" (UID: \"66df6c04-8987-4f07-9fab-4e80cf71f6a2\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h"
Oct 30 20:55:29.149372 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:29.149352    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66df6c04-8987-4f07-9fab-4e80cf71f6a2-istio-token\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h\" (UID: \"66df6c04-8987-4f07-9fab-4e80cf71f6a2\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h"
Oct 30 20:55:29.282909 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:29.282822    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h"
Oct 30 20:55:29.714157 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:29.714131    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h"]
Oct 30 20:55:29.714948 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:55:29.714926    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66df6c04_8987_4f07_9fab_4e80cf71f6a2.slice/crio-cc0a69a853eaf8088d66fe777ab0b8771686120181ed0176992f26a9dfd18faf WatchSource:0}: Error finding container cc0a69a853eaf8088d66fe777ab0b8771686120181ed0176992f26a9dfd18faf: Status 404 returned error can't find the container with id cc0a69a853eaf8088d66fe777ab0b8771686120181ed0176992f26a9dfd18faf
Oct 30 20:55:29.717611 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:29.717577    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 20:55:29.717692 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:29.717646    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 20:55:29.717692 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:29.717681    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 20:55:29.729123 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:29.729097    5713 generic.go:356] "Generic (PLEG): container finished" podID="c81ff9be-12b4-41f3-8ea2-f86a043f5e2a" containerID="b59fa4862beb1d44536ef29572ec10aeaaa7c5a99a711fd4d3cc5df27f5d60a0" exitCode=0
Oct 30 20:55:29.729264 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:29.729188    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49ff7895-7z6ww" event={"ID":"c81ff9be-12b4-41f3-8ea2-f86a043f5e2a","Type":"ContainerDied","Data":"b59fa4862beb1d44536ef29572ec10aeaaa7c5a99a711fd4d3cc5df27f5d60a0"}
Oct 30 20:55:29.730644 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:29.730621    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h" event={"ID":"66df6c04-8987-4f07-9fab-4e80cf71f6a2","Type":"ContainerStarted","Data":"cc0a69a853eaf8088d66fe777ab0b8771686120181ed0176992f26a9dfd18faf"}
Oct 30 20:55:30.762363 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:30.762323    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h" event={"ID":"66df6c04-8987-4f07-9fab-4e80cf71f6a2","Type":"ContainerStarted","Data":"74a6e9e8f8edbf94c47086069fa1c7a56bd97f78cdff54c04ac77192491e2e6c"}
Oct 30 20:55:30.966110 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:30.966077    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49ff7895-7z6ww"
Oct 30 20:55:31.023711 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:31.023623    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c81ff9be-12b4-41f3-8ea2-f86a043f5e2a-kube-api-access-xl822\" (UniqueName: \"kubernetes.io/projected/c81ff9be-12b4-41f3-8ea2-f86a043f5e2a-kube-api-access-xl822\") pod \"c81ff9be-12b4-41f3-8ea2-f86a043f5e2a\" (UID: \"c81ff9be-12b4-41f3-8ea2-f86a043f5e2a\") "
Oct 30 20:55:31.023996 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:31.023757    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c81ff9be-12b4-41f3-8ea2-f86a043f5e2a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c81ff9be-12b4-41f3-8ea2-f86a043f5e2a-maas-api-openshift-ca\") pod \"c81ff9be-12b4-41f3-8ea2-f86a043f5e2a\" (UID: \"c81ff9be-12b4-41f3-8ea2-f86a043f5e2a\") "
Oct 30 20:55:31.023996 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:31.023832    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c81ff9be-12b4-41f3-8ea2-f86a043f5e2a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c81ff9be-12b4-41f3-8ea2-f86a043f5e2a-maas-api-xks-ca\") pod \"c81ff9be-12b4-41f3-8ea2-f86a043f5e2a\" (UID: \"c81ff9be-12b4-41f3-8ea2-f86a043f5e2a\") "
Oct 30 20:55:31.024208 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:31.024182    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c81ff9be-12b4-41f3-8ea2-f86a043f5e2a-maas-api-openshift-ca" pod "c81ff9be-12b4-41f3-8ea2-f86a043f5e2a" (UID: "c81ff9be-12b4-41f3-8ea2-f86a043f5e2a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:55:31.026017 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:31.025994    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c81ff9be-12b4-41f3-8ea2-f86a043f5e2a-kube-api-access-xl822" pod "c81ff9be-12b4-41f3-8ea2-f86a043f5e2a" (UID: "c81ff9be-12b4-41f3-8ea2-f86a043f5e2a"). InnerVolumeSpecName "kube-api-access-xl822". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:55:31.026118 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:31.026041    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c81ff9be-12b4-41f3-8ea2-f86a043f5e2a-maas-api-xks-ca" pod "c81ff9be-12b4-41f3-8ea2-f86a043f5e2a" (UID: "c81ff9be-12b4-41f3-8ea2-f86a043f5e2a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:55:31.081928 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:31.081869    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h" podStartSLOduration=3.081852063 podStartE2EDuration="3.081852063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:55:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:55:30.960684703 +0000 UTC m=+1014.025461106" watchObservedRunningTime="2026-10-30 20:55:31.081852063 +0000 UTC m=+1014.146628461"
Oct 30 20:55:31.124906 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:31.124869    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl822\" (UniqueName: \"kubernetes.io/projected/c81ff9be-12b4-41f3-8ea2-f86a043f5e2a-kube-api-access-xl822\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:31.124906 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:31.124904    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c81ff9be-12b4-41f3-8ea2-f86a043f5e2a-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:31.125108 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:31.124919    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c81ff9be-12b4-41f3-8ea2-f86a043f5e2a-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:31.283467 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:31.283381    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h"
Oct 30 20:55:31.285109 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:31.285081    5713 patch_prober.go:29] interesting pod/e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 20:55:31.285211 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:31.285145    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h" podUID="66df6c04-8987-4f07-9fab-4e80cf71f6a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 20:55:31.770857 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:31.770824    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49ff7895-7z6ww"
Oct 30 20:55:31.771280 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:31.770822    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49ff7895-7z6ww" event={"ID":"c81ff9be-12b4-41f3-8ea2-f86a043f5e2a","Type":"ContainerDied","Data":"007f465b69165fbf59881b6dc59266045d46d368482ca9cf73f32ab68ea424c9"}
Oct 30 20:55:31.771280 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:31.770945    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="007f465b69165fbf59881b6dc59266045d46d368482ca9cf73f32ab68ea424c9"
Oct 30 20:55:32.283465 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:32.283432    5713 patch_prober.go:29] interesting pod/e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 20:55:32.283657 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:32.283484    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h" podUID="66df6c04-8987-4f07-9fab-4e80cf71f6a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 20:55:33.283700 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:33.283666    5713 patch_prober.go:29] interesting pod/e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 20:55:33.284225 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:33.283726    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h" podUID="66df6c04-8987-4f07-9fab-4e80cf71f6a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 20:55:34.283660 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:34.283620    5713 patch_prober.go:29] interesting pod/e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 20:55:34.283870 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:34.283681    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h" podUID="66df6c04-8987-4f07-9fab-4e80cf71f6a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 20:55:35.283940 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:35.283891    5713 patch_prober.go:29] interesting pod/e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 20:55:35.284380 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:35.283975    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h" podUID="66df6c04-8987-4f07-9fab-4e80cf71f6a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 20:55:36.283695 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:36.283659    5713 patch_prober.go:29] interesting pod/e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 20:55:36.283954 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:36.283730    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h" podUID="66df6c04-8987-4f07-9fab-4e80cf71f6a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 20:55:37.299266 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:37.299234    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h"
Oct 30 20:55:37.299818 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:37.299762    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h"
Oct 30 20:55:37.302311 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:37.302287    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h"
Oct 30 20:55:40.270249 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:40.270204    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7jfsh"]
Oct 30 20:55:40.270685 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:40.270413    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7jfsh" podUID="2f321ee6-f63d-43b5-a1e9-983985a789ff" containerName="istio-proxy" containerID="cri-o://2ee967897bdc9567d9005aa1b3818cc5ea9b5d72a1dd38998a2675edb05dbf77" gracePeriod=30
Oct 30 20:55:45.546760 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:45.546689    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7jfsh"
Oct 30 20:55:45.624130 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:45.624091    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f321ee6-f63d-43b5-a1e9-983985a789ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f321ee6-f63d-43b5-a1e9-983985a789ff-workload-certs\") pod \"2f321ee6-f63d-43b5-a1e9-983985a789ff\" (UID: \"2f321ee6-f63d-43b5-a1e9-983985a789ff\") "
Oct 30 20:55:45.624325 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:45.624186    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2f321ee6-f63d-43b5-a1e9-983985a789ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f321ee6-f63d-43b5-a1e9-983985a789ff-istio-podinfo\") pod \"2f321ee6-f63d-43b5-a1e9-983985a789ff\" (UID: \"2f321ee6-f63d-43b5-a1e9-983985a789ff\") "
Oct 30 20:55:45.624325 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:45.624213    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f321ee6-f63d-43b5-a1e9-983985a789ff-istio-token\" (UniqueName: \"kubernetes.io/projected/2f321ee6-f63d-43b5-a1e9-983985a789ff-istio-token\") pod \"2f321ee6-f63d-43b5-a1e9-983985a789ff\" (UID: \"2f321ee6-f63d-43b5-a1e9-983985a789ff\") "
Oct 30 20:55:45.624325 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:45.624254    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f321ee6-f63d-43b5-a1e9-983985a789ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f321ee6-f63d-43b5-a1e9-983985a789ff-credential-socket\") pod \"2f321ee6-f63d-43b5-a1e9-983985a789ff\" (UID: \"2f321ee6-f63d-43b5-a1e9-983985a789ff\") "
Oct 30 20:55:45.624325 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:45.624306    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f321ee6-f63d-43b5-a1e9-983985a789ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f321ee6-f63d-43b5-a1e9-983985a789ff-istio-envoy\") pod \"2f321ee6-f63d-43b5-a1e9-983985a789ff\" (UID: \"2f321ee6-f63d-43b5-a1e9-983985a789ff\") "
Oct 30 20:55:45.624549 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:45.624421    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f321ee6-f63d-43b5-a1e9-983985a789ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f321ee6-f63d-43b5-a1e9-983985a789ff-istiod-ca-cert\") pod \"2f321ee6-f63d-43b5-a1e9-983985a789ff\" (UID: \"2f321ee6-f63d-43b5-a1e9-983985a789ff\") "
Oct 30 20:55:45.624549 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:45.624477    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f321ee6-f63d-43b5-a1e9-983985a789ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f321ee6-f63d-43b5-a1e9-983985a789ff-istio-data\") pod \"2f321ee6-f63d-43b5-a1e9-983985a789ff\" (UID: \"2f321ee6-f63d-43b5-a1e9-983985a789ff\") "
Oct 30 20:55:45.624549 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:45.624531    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f321ee6-f63d-43b5-a1e9-983985a789ff-kube-api-access-p88qr\" (UniqueName: \"kubernetes.io/projected/2f321ee6-f63d-43b5-a1e9-983985a789ff-kube-api-access-p88qr\") pod \"2f321ee6-f63d-43b5-a1e9-983985a789ff\" (UID: \"2f321ee6-f63d-43b5-a1e9-983985a789ff\") "
Oct 30 20:55:45.624710 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:45.624568    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f321ee6-f63d-43b5-a1e9-983985a789ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f321ee6-f63d-43b5-a1e9-983985a789ff-workload-socket\") pod \"2f321ee6-f63d-43b5-a1e9-983985a789ff\" (UID: \"2f321ee6-f63d-43b5-a1e9-983985a789ff\") "
Oct 30 20:55:45.624710 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:45.624620    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f321ee6-f63d-43b5-a1e9-983985a789ff-credential-socket" pod "2f321ee6-f63d-43b5-a1e9-983985a789ff" (UID: "2f321ee6-f63d-43b5-a1e9-983985a789ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:55:45.624992 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:45.624964    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f321ee6-f63d-43b5-a1e9-983985a789ff-istio-data" pod "2f321ee6-f63d-43b5-a1e9-983985a789ff" (UID: "2f321ee6-f63d-43b5-a1e9-983985a789ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:55:45.625187 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:45.625166    5713 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f321ee6-f63d-43b5-a1e9-983985a789ff-credential-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:45.625352 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:45.625335    5713 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f321ee6-f63d-43b5-a1e9-983985a789ff-istio-data\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:45.625465 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:45.625188    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f321ee6-f63d-43b5-a1e9-983985a789ff-workload-socket" pod "2f321ee6-f63d-43b5-a1e9-983985a789ff" (UID: "2f321ee6-f63d-43b5-a1e9-983985a789ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:55:45.625465 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:45.624971    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f321ee6-f63d-43b5-a1e9-983985a789ff-istiod-ca-cert" pod "2f321ee6-f63d-43b5-a1e9-983985a789ff" (UID: "2f321ee6-f63d-43b5-a1e9-983985a789ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:55:45.625632 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:45.625365    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f321ee6-f63d-43b5-a1e9-983985a789ff-workload-certs" pod "2f321ee6-f63d-43b5-a1e9-983985a789ff" (UID: "2f321ee6-f63d-43b5-a1e9-983985a789ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:55:45.627539 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:45.627513    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f321ee6-f63d-43b5-a1e9-983985a789ff-istio-envoy" pod "2f321ee6-f63d-43b5-a1e9-983985a789ff" (UID: "2f321ee6-f63d-43b5-a1e9-983985a789ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:55:45.628152 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:45.628127    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f321ee6-f63d-43b5-a1e9-983985a789ff-istio-podinfo" pod "2f321ee6-f63d-43b5-a1e9-983985a789ff" (UID: "2f321ee6-f63d-43b5-a1e9-983985a789ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:55:45.630027 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:45.629989    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f321ee6-f63d-43b5-a1e9-983985a789ff-kube-api-access-p88qr" pod "2f321ee6-f63d-43b5-a1e9-983985a789ff" (UID: "2f321ee6-f63d-43b5-a1e9-983985a789ff"). InnerVolumeSpecName "kube-api-access-p88qr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:55:45.630027 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:45.630015    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f321ee6-f63d-43b5-a1e9-983985a789ff-istio-token" pod "2f321ee6-f63d-43b5-a1e9-983985a789ff" (UID: "2f321ee6-f63d-43b5-a1e9-983985a789ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:55:45.727151 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:45.727115    5713 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f321ee6-f63d-43b5-a1e9-983985a789ff-istio-envoy\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:45.727151 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:45.727152    5713 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f321ee6-f63d-43b5-a1e9-983985a789ff-istiod-ca-cert\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:45.727347 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:45.727168    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p88qr\" (UniqueName: \"kubernetes.io/projected/2f321ee6-f63d-43b5-a1e9-983985a789ff-kube-api-access-p88qr\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:45.727347 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:45.727183    5713 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f321ee6-f63d-43b5-a1e9-983985a789ff-workload-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:45.727347 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:45.727195    5713 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f321ee6-f63d-43b5-a1e9-983985a789ff-workload-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:45.727347 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:45.727208    5713 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f321ee6-f63d-43b5-a1e9-983985a789ff-istio-podinfo\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:45.727347 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:45.727222    5713 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f321ee6-f63d-43b5-a1e9-983985a789ff-istio-token\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:55:45.869613 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:45.869532    5713 generic.go:356] "Generic (PLEG): container finished" podID="2f321ee6-f63d-43b5-a1e9-983985a789ff" containerID="2ee967897bdc9567d9005aa1b3818cc5ea9b5d72a1dd38998a2675edb05dbf77" exitCode=0
Oct 30 20:55:45.869613 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:45.869591    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7jfsh"
Oct 30 20:55:45.869881 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:45.869616    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7jfsh" event={"ID":"2f321ee6-f63d-43b5-a1e9-983985a789ff","Type":"ContainerDied","Data":"2ee967897bdc9567d9005aa1b3818cc5ea9b5d72a1dd38998a2675edb05dbf77"}
Oct 30 20:55:45.869881 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:45.869705    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7jfsh" event={"ID":"2f321ee6-f63d-43b5-a1e9-983985a789ff","Type":"ContainerDied","Data":"58729d3db1ce9773115663c2b59f96491cf842c96081813c83aa6789184400fd"}
Oct 30 20:55:45.869881 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:45.869736    5713 scope.go:122] "RemoveContainer" containerID="2ee967897bdc9567d9005aa1b3818cc5ea9b5d72a1dd38998a2675edb05dbf77"
Oct 30 20:55:45.880877 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:45.880861    5713 scope.go:122] "RemoveContainer" containerID="2ee967897bdc9567d9005aa1b3818cc5ea9b5d72a1dd38998a2675edb05dbf77"
Oct 30 20:55:45.881180 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:55:45.881161    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ee967897bdc9567d9005aa1b3818cc5ea9b5d72a1dd38998a2675edb05dbf77\": container with ID starting with 2ee967897bdc9567d9005aa1b3818cc5ea9b5d72a1dd38998a2675edb05dbf77 not found: ID does not exist" containerID="2ee967897bdc9567d9005aa1b3818cc5ea9b5d72a1dd38998a2675edb05dbf77"
Oct 30 20:55:45.881255 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:45.881189    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ee967897bdc9567d9005aa1b3818cc5ea9b5d72a1dd38998a2675edb05dbf77"} err="failed to get container status \"2ee967897bdc9567d9005aa1b3818cc5ea9b5d72a1dd38998a2675edb05dbf77\": rpc error: code = NotFound desc = could not find container \"2ee967897bdc9567d9005aa1b3818cc5ea9b5d72a1dd38998a2675edb05dbf77\": container with ID starting with 2ee967897bdc9567d9005aa1b3818cc5ea9b5d72a1dd38998a2675edb05dbf77 not found: ID does not exist"
Oct 30 20:55:46.038486 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:46.038455    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7jfsh"]
Oct 30 20:55:46.041956 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:46.041932    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7jfsh"]
Oct 30 20:55:47.063408 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:47.063374    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f321ee6-f63d-43b5-a1e9-983985a789ff" path="/var/lib/kubelet/pods/2f321ee6-f63d-43b5-a1e9-983985a789ff/volumes"
Oct 30 20:55:54.460316 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:55:54.460276    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 20:55:59.907930 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:59.907888    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9"]
Oct 30 20:55:59.908516 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:59.908498    5713 cpu_manager.go:414] "containerMap: removing container" podUID="2f321ee6-f63d-43b5-a1e9-983985a789ff" containerName="istio-proxy"
Oct 30 20:55:59.908562 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:59.908518    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f321ee6-f63d-43b5-a1e9-983985a789ff" containerName="istio-proxy"
Oct 30 20:55:59.908562 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:59.908553    5713 cpu_manager.go:414] "containerMap: removing container" podUID="c81ff9be-12b4-41f3-8ea2-f86a043f5e2a" containerName="revoke-keys"
Oct 30 20:55:59.908562 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:59.908559    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c81ff9be-12b4-41f3-8ea2-f86a043f5e2a" containerName="revoke-keys"
Oct 30 20:55:59.908658 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:59.908641    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="c81ff9be-12b4-41f3-8ea2-f86a043f5e2a" containerName="revoke-keys"
Oct 30 20:55:59.908658 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:59.908653    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f321ee6-f63d-43b5-a1e9-983985a789ff" containerName="istio-proxy"
Oct 30 20:55:59.912305 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:59.912287    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9"
Oct 30 20:55:59.991287 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:59.991253    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a88fd37-ae88-4075-8203-024e856be6ee-istio-envoy\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9\" (UID: \"0a88fd37-ae88-4075-8203-024e856be6ee\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9"
Oct 30 20:55:59.991478 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:59.991304    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a88fd37-ae88-4075-8203-024e856be6ee-istio-data\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9\" (UID: \"0a88fd37-ae88-4075-8203-024e856be6ee\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9"
Oct 30 20:55:59.991478 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:59.991443    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a88fd37-ae88-4075-8203-024e856be6ee-istio-token\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9\" (UID: \"0a88fd37-ae88-4075-8203-024e856be6ee\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9"
Oct 30 20:55:59.991586 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:59.991478    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hzrh\" (UniqueName: \"kubernetes.io/projected/0a88fd37-ae88-4075-8203-024e856be6ee-kube-api-access-6hzrh\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9\" (UID: \"0a88fd37-ae88-4075-8203-024e856be6ee\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9"
Oct 30 20:55:59.991643 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:59.991603    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a88fd37-ae88-4075-8203-024e856be6ee-istiod-ca-cert\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9\" (UID: \"0a88fd37-ae88-4075-8203-024e856be6ee\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9"
Oct 30 20:55:59.991701 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:59.991650    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a88fd37-ae88-4075-8203-024e856be6ee-workload-certs\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9\" (UID: \"0a88fd37-ae88-4075-8203-024e856be6ee\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9"
Oct 30 20:55:59.991756 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:59.991734    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a88fd37-ae88-4075-8203-024e856be6ee-credential-socket\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9\" (UID: \"0a88fd37-ae88-4075-8203-024e856be6ee\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9"
Oct 30 20:55:59.991828 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:59.991801    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a88fd37-ae88-4075-8203-024e856be6ee-workload-socket\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9\" (UID: \"0a88fd37-ae88-4075-8203-024e856be6ee\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9"
Oct 30 20:55:59.991828 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:55:59.991822    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a88fd37-ae88-4075-8203-024e856be6ee-istio-podinfo\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9\" (UID: \"0a88fd37-ae88-4075-8203-024e856be6ee\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9"
Oct 30 20:56:00.019209 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:00.019175    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9"]
Oct 30 20:56:00.093159 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:00.093122    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a88fd37-ae88-4075-8203-024e856be6ee-istiod-ca-cert\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9\" (UID: \"0a88fd37-ae88-4075-8203-024e856be6ee\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9"
Oct 30 20:56:00.093367 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:00.093178    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a88fd37-ae88-4075-8203-024e856be6ee-workload-certs\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9\" (UID: \"0a88fd37-ae88-4075-8203-024e856be6ee\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9"
Oct 30 20:56:00.093367 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:00.093231    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a88fd37-ae88-4075-8203-024e856be6ee-credential-socket\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9\" (UID: \"0a88fd37-ae88-4075-8203-024e856be6ee\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9"
Oct 30 20:56:00.093367 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:00.093257    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a88fd37-ae88-4075-8203-024e856be6ee-workload-socket\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9\" (UID: \"0a88fd37-ae88-4075-8203-024e856be6ee\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9"
Oct 30 20:56:00.093367 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:00.093287    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a88fd37-ae88-4075-8203-024e856be6ee-istio-podinfo\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9\" (UID: \"0a88fd37-ae88-4075-8203-024e856be6ee\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9"
Oct 30 20:56:00.093367 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:00.093337    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a88fd37-ae88-4075-8203-024e856be6ee-istio-envoy\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9\" (UID: \"0a88fd37-ae88-4075-8203-024e856be6ee\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9"
Oct 30 20:56:00.093640 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:00.093373    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a88fd37-ae88-4075-8203-024e856be6ee-istio-data\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9\" (UID: \"0a88fd37-ae88-4075-8203-024e856be6ee\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9"
Oct 30 20:56:00.093640 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:00.093585    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a88fd37-ae88-4075-8203-024e856be6ee-istio-token\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9\" (UID: \"0a88fd37-ae88-4075-8203-024e856be6ee\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9"
Oct 30 20:56:00.093744 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:00.093645    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hzrh\" (UniqueName: \"kubernetes.io/projected/0a88fd37-ae88-4075-8203-024e856be6ee-kube-api-access-6hzrh\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9\" (UID: \"0a88fd37-ae88-4075-8203-024e856be6ee\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9"
Oct 30 20:56:00.093744 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:00.093706    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a88fd37-ae88-4075-8203-024e856be6ee-credential-socket\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9\" (UID: \"0a88fd37-ae88-4075-8203-024e856be6ee\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9"
Oct 30 20:56:00.093890 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:00.093737    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a88fd37-ae88-4075-8203-024e856be6ee-workload-socket\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9\" (UID: \"0a88fd37-ae88-4075-8203-024e856be6ee\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9"
Oct 30 20:56:00.093890 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:00.093856    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a88fd37-ae88-4075-8203-024e856be6ee-workload-certs\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9\" (UID: \"0a88fd37-ae88-4075-8203-024e856be6ee\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9"
Oct 30 20:56:00.093997 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:00.093931    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a88fd37-ae88-4075-8203-024e856be6ee-istio-data\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9\" (UID: \"0a88fd37-ae88-4075-8203-024e856be6ee\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9"
Oct 30 20:56:00.093997 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:00.093975    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a88fd37-ae88-4075-8203-024e856be6ee-istiod-ca-cert\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9\" (UID: \"0a88fd37-ae88-4075-8203-024e856be6ee\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9"
Oct 30 20:56:00.095867 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:00.095843    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a88fd37-ae88-4075-8203-024e856be6ee-istio-envoy\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9\" (UID: \"0a88fd37-ae88-4075-8203-024e856be6ee\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9"
Oct 30 20:56:00.095966 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:00.095949    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a88fd37-ae88-4075-8203-024e856be6ee-istio-podinfo\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9\" (UID: \"0a88fd37-ae88-4075-8203-024e856be6ee\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9"
Oct 30 20:56:00.153087 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:00.153059    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hzrh\" (UniqueName: \"kubernetes.io/projected/0a88fd37-ae88-4075-8203-024e856be6ee-kube-api-access-6hzrh\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9\" (UID: \"0a88fd37-ae88-4075-8203-024e856be6ee\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9"
Oct 30 20:56:00.153236 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:00.153060    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a88fd37-ae88-4075-8203-024e856be6ee-istio-token\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9\" (UID: \"0a88fd37-ae88-4075-8203-024e856be6ee\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9"
Oct 30 20:56:00.225813 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:00.225754    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9"
Oct 30 20:56:00.657140 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:00.657110    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9"]
Oct 30 20:56:00.658153 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:56:00.658123    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a88fd37_ae88_4075_8203_024e856be6ee.slice/crio-bd0b37e3c654461d3f8ca4a1005729b134ddad3803064ade331c1a6dbc7f8a4f WatchSource:0}: Error finding container bd0b37e3c654461d3f8ca4a1005729b134ddad3803064ade331c1a6dbc7f8a4f: Status 404 returned error can't find the container with id bd0b37e3c654461d3f8ca4a1005729b134ddad3803064ade331c1a6dbc7f8a4f
Oct 30 20:56:00.660497 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:00.660460    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 20:56:00.660560 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:00.660539    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 20:56:00.660644 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:00.660591    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 20:56:00.980368 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:00.980329    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9" event={"ID":"0a88fd37-ae88-4075-8203-024e856be6ee","Type":"ContainerStarted","Data":"4bfc0ffcf59e5706d3ec5d7854427e5e6b5ea158791048ed7fb2487dea0f7656"}
Oct 30 20:56:00.980819 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:00.980397    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9" event={"ID":"0a88fd37-ae88-4075-8203-024e856be6ee","Type":"ContainerStarted","Data":"bd0b37e3c654461d3f8ca4a1005729b134ddad3803064ade331c1a6dbc7f8a4f"}
Oct 30 20:56:01.100998 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:01.100936    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9" podStartSLOduration=2.100919725 podStartE2EDuration="2.100919725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:55:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:56:01.097706031 +0000 UTC m=+1044.162482453" watchObservedRunningTime="2026-10-30 20:56:01.100919725 +0000 UTC m=+1044.165696123"
Oct 30 20:56:01.226976 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:01.226943    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9"
Oct 30 20:56:01.228498 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:01.228474    5713 patch_prober.go:29] interesting pod/e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 30 20:56:01.228603 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:01.228523    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9" podUID="0a88fd37-ae88-4075-8203-024e856be6ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 30 20:56:02.227078 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:02.227050    5713 patch_prober.go:29] interesting pod/e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 30 20:56:02.227481 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:02.227111    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9" podUID="0a88fd37-ae88-4075-8203-024e856be6ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 30 20:56:03.226866 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:03.226826    5713 patch_prober.go:29] interesting pod/e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 30 20:56:03.227046 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:03.226883    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9" podUID="0a88fd37-ae88-4075-8203-024e856be6ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 30 20:56:04.227042 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:04.227010    5713 patch_prober.go:29] interesting pod/e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 30 20:56:04.227441 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:04.227066    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9" podUID="0a88fd37-ae88-4075-8203-024e856be6ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 30 20:56:05.226699 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:05.226665    5713 patch_prober.go:29] interesting pod/e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 30 20:56:05.226926 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:05.226726    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9" podUID="0a88fd37-ae88-4075-8203-024e856be6ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 30 20:56:06.226668 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:06.226630    5713 patch_prober.go:29] interesting pod/e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 30 20:56:06.227078 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:06.226686    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9" podUID="0a88fd37-ae88-4075-8203-024e856be6ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 30 20:56:07.462296 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:07.462261    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9"
Oct 30 20:56:07.462851 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:07.462775    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9"
Oct 30 20:56:07.463869 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:07.463847    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9"
Oct 30 20:56:13.551453 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:13.551417    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-8fa2a7/distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7"]
Oct 30 20:56:13.555287 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:13.555271    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7"
Oct 30 20:56:13.658191 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:13.658156    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-8fa2a7\"/\"openshift-service-ca.crt\""
Oct 30 20:56:13.658191 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:13.658168    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-8fa2a7\"/\"kube-root-ca.crt\""
Oct 30 20:56:13.658455 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:13.658283    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-8fa2a7\"/\"distinct-w6-8fa2a7-kserve-self-signed-certs\""
Oct 30 20:56:13.658455 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:13.658311    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-8fa2a7\"/\"default-dockercfg-q9gbm\""
Oct 30 20:56:13.661769 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:13.661736    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-8fa2a7/distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7"]
Oct 30 20:56:13.700744 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:13.700216    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2193213-ebae-498a-911a-6056a8e94312-tmp-dir\") pod \"distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7\" (UID: \"b2193213-ebae-498a-911a-6056a8e94312\") " pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7"
Oct 30 20:56:13.700744 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:13.700312    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2193213-ebae-498a-911a-6056a8e94312-model-cache\") pod \"distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7\" (UID: \"b2193213-ebae-498a-911a-6056a8e94312\") " pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7"
Oct 30 20:56:13.700744 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:13.700353    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2193213-ebae-498a-911a-6056a8e94312-tls-certs\") pod \"distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7\" (UID: \"b2193213-ebae-498a-911a-6056a8e94312\") " pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7"
Oct 30 20:56:13.700744 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:13.700399    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlncq\" (UniqueName: \"kubernetes.io/projected/b2193213-ebae-498a-911a-6056a8e94312-kube-api-access-qlncq\") pod \"distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7\" (UID: \"b2193213-ebae-498a-911a-6056a8e94312\") " pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7"
Oct 30 20:56:13.700744 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:13.700448    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2193213-ebae-498a-911a-6056a8e94312-dshm\") pod \"distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7\" (UID: \"b2193213-ebae-498a-911a-6056a8e94312\") " pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7"
Oct 30 20:56:13.700744 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:13.700541    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2193213-ebae-498a-911a-6056a8e94312-home\") pod \"distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7\" (UID: \"b2193213-ebae-498a-911a-6056a8e94312\") " pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7"
Oct 30 20:56:13.801910 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:13.801819    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2193213-ebae-498a-911a-6056a8e94312-tmp-dir\") pod \"distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7\" (UID: \"b2193213-ebae-498a-911a-6056a8e94312\") " pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7"
Oct 30 20:56:13.801910 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:13.801880    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2193213-ebae-498a-911a-6056a8e94312-model-cache\") pod \"distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7\" (UID: \"b2193213-ebae-498a-911a-6056a8e94312\") " pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7"
Oct 30 20:56:13.801910 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:13.801904    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2193213-ebae-498a-911a-6056a8e94312-tls-certs\") pod \"distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7\" (UID: \"b2193213-ebae-498a-911a-6056a8e94312\") " pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7"
Oct 30 20:56:13.802192 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:13.801931    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlncq\" (UniqueName: \"kubernetes.io/projected/b2193213-ebae-498a-911a-6056a8e94312-kube-api-access-qlncq\") pod \"distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7\" (UID: \"b2193213-ebae-498a-911a-6056a8e94312\") " pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7"
Oct 30 20:56:13.802192 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:13.801962    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2193213-ebae-498a-911a-6056a8e94312-dshm\") pod \"distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7\" (UID: \"b2193213-ebae-498a-911a-6056a8e94312\") " pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7"
Oct 30 20:56:13.802192 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:13.802043    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2193213-ebae-498a-911a-6056a8e94312-home\") pod \"distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7\" (UID: \"b2193213-ebae-498a-911a-6056a8e94312\") " pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7"
Oct 30 20:56:13.802357 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:13.802301    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2193213-ebae-498a-911a-6056a8e94312-tmp-dir\") pod \"distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7\" (UID: \"b2193213-ebae-498a-911a-6056a8e94312\") " pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7"
Oct 30 20:56:13.802357 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:13.802344    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2193213-ebae-498a-911a-6056a8e94312-model-cache\") pod \"distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7\" (UID: \"b2193213-ebae-498a-911a-6056a8e94312\") " pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7"
Oct 30 20:56:13.802445 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:13.802391    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2193213-ebae-498a-911a-6056a8e94312-home\") pod \"distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7\" (UID: \"b2193213-ebae-498a-911a-6056a8e94312\") " pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7"
Oct 30 20:56:13.804387 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:13.804364    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2193213-ebae-498a-911a-6056a8e94312-dshm\") pod \"distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7\" (UID: \"b2193213-ebae-498a-911a-6056a8e94312\") " pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7"
Oct 30 20:56:13.804542 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:13.804522    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2193213-ebae-498a-911a-6056a8e94312-tls-certs\") pod \"distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7\" (UID: \"b2193213-ebae-498a-911a-6056a8e94312\") " pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7"
Oct 30 20:56:13.858715 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:13.858686    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlncq\" (UniqueName: \"kubernetes.io/projected/b2193213-ebae-498a-911a-6056a8e94312-kube-api-access-qlncq\") pod \"distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7\" (UID: \"b2193213-ebae-498a-911a-6056a8e94312\") " pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7"
Oct 30 20:56:13.868612 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:13.868588    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7"
Oct 30 20:56:14.272874 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:14.272847    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-8fa2a7/distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7"]
Oct 30 20:56:14.274219 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:56:14.274188    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb2193213_ebae_498a_911a_6056a8e94312.slice/crio-5ebde4ea2bb0553729b57d8e72f5fcc2bdb6243621d3523a48b108d76e542dba WatchSource:0}: Error finding container 5ebde4ea2bb0553729b57d8e72f5fcc2bdb6243621d3523a48b108d76e542dba: Status 404 returned error can't find the container with id 5ebde4ea2bb0553729b57d8e72f5fcc2bdb6243621d3523a48b108d76e542dba
Oct 30 20:56:15.078732 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:15.078692    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7" event={"ID":"b2193213-ebae-498a-911a-6056a8e94312","Type":"ContainerStarted","Data":"248aa9b531f637cfc83c8a4e0738ad785d6d0518394e7efbe5223b0eee488774"}
Oct 30 20:56:15.079161 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:15.078745    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7" event={"ID":"b2193213-ebae-498a-911a-6056a8e94312","Type":"ContainerStarted","Data":"5ebde4ea2bb0553729b57d8e72f5fcc2bdb6243621d3523a48b108d76e542dba"}
Oct 30 20:56:15.202303 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:15.202248    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7" podStartSLOduration=2.202229095 podStartE2EDuration="2.202229095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:56:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:56:15.200709297 +0000 UTC m=+1058.265485695" watchObservedRunningTime="2026-10-30 20:56:15.202229095 +0000 UTC m=+1058.267005493"
Oct 30 20:56:15.869668 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:15.869627    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7"
Oct 30 20:56:23.868909 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:23.868873    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7"
Oct 30 20:56:23.881864 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:23.881831    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7"
Oct 30 20:56:24.151006 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:24.150913    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7"
Oct 30 20:56:24.459919 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:56:24.459882    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 20:56:49.268223 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:49.268144    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-682999/distinct-w3-682999-kserve-74d49d4955-nndlc"]
Oct 30 20:56:49.273521 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:49.273502    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-682999/distinct-w3-682999-kserve-74d49d4955-nndlc"
Oct 30 20:56:49.345821 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:49.345744    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a35314b-31ea-42e6-b49a-15ab1305b338-home\") pod \"distinct-w3-682999-kserve-74d49d4955-nndlc\" (UID: \"2a35314b-31ea-42e6-b49a-15ab1305b338\") " pod="e2e-models-e2e-worker-w3-682999/distinct-w3-682999-kserve-74d49d4955-nndlc"
Oct 30 20:56:49.345821 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:49.345781    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a35314b-31ea-42e6-b49a-15ab1305b338-dshm\") pod \"distinct-w3-682999-kserve-74d49d4955-nndlc\" (UID: \"2a35314b-31ea-42e6-b49a-15ab1305b338\") " pod="e2e-models-e2e-worker-w3-682999/distinct-w3-682999-kserve-74d49d4955-nndlc"
Oct 30 20:56:49.346144 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:49.345983    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a35314b-31ea-42e6-b49a-15ab1305b338-tmp-dir\") pod \"distinct-w3-682999-kserve-74d49d4955-nndlc\" (UID: \"2a35314b-31ea-42e6-b49a-15ab1305b338\") " pod="e2e-models-e2e-worker-w3-682999/distinct-w3-682999-kserve-74d49d4955-nndlc"
Oct 30 20:56:49.346144 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:49.346019    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qb6b\" (UniqueName: \"kubernetes.io/projected/2a35314b-31ea-42e6-b49a-15ab1305b338-kube-api-access-5qb6b\") pod \"distinct-w3-682999-kserve-74d49d4955-nndlc\" (UID: \"2a35314b-31ea-42e6-b49a-15ab1305b338\") " pod="e2e-models-e2e-worker-w3-682999/distinct-w3-682999-kserve-74d49d4955-nndlc"
Oct 30 20:56:49.346260 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:49.346148    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a35314b-31ea-42e6-b49a-15ab1305b338-tls-certs\") pod \"distinct-w3-682999-kserve-74d49d4955-nndlc\" (UID: \"2a35314b-31ea-42e6-b49a-15ab1305b338\") " pod="e2e-models-e2e-worker-w3-682999/distinct-w3-682999-kserve-74d49d4955-nndlc"
Oct 30 20:56:49.346260 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:49.346213    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a35314b-31ea-42e6-b49a-15ab1305b338-model-cache\") pod \"distinct-w3-682999-kserve-74d49d4955-nndlc\" (UID: \"2a35314b-31ea-42e6-b49a-15ab1305b338\") " pod="e2e-models-e2e-worker-w3-682999/distinct-w3-682999-kserve-74d49d4955-nndlc"
Oct 30 20:56:49.348319 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:49.348293    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-682999/distinct-w3-682999-kserve-74d49d4955-nndlc"]
Oct 30 20:56:49.375581 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:49.375543    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-682999\"/\"distinct-w3-682999-kserve-self-signed-certs\""
Oct 30 20:56:49.446913 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:49.446881    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a35314b-31ea-42e6-b49a-15ab1305b338-home\") pod \"distinct-w3-682999-kserve-74d49d4955-nndlc\" (UID: \"2a35314b-31ea-42e6-b49a-15ab1305b338\") " pod="e2e-models-e2e-worker-w3-682999/distinct-w3-682999-kserve-74d49d4955-nndlc"
Oct 30 20:56:49.446913 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:49.446915    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a35314b-31ea-42e6-b49a-15ab1305b338-dshm\") pod \"distinct-w3-682999-kserve-74d49d4955-nndlc\" (UID: \"2a35314b-31ea-42e6-b49a-15ab1305b338\") " pod="e2e-models-e2e-worker-w3-682999/distinct-w3-682999-kserve-74d49d4955-nndlc"
Oct 30 20:56:49.447157 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:49.446972    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a35314b-31ea-42e6-b49a-15ab1305b338-tmp-dir\") pod \"distinct-w3-682999-kserve-74d49d4955-nndlc\" (UID: \"2a35314b-31ea-42e6-b49a-15ab1305b338\") " pod="e2e-models-e2e-worker-w3-682999/distinct-w3-682999-kserve-74d49d4955-nndlc"
Oct 30 20:56:49.447157 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:49.446990    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qb6b\" (UniqueName: \"kubernetes.io/projected/2a35314b-31ea-42e6-b49a-15ab1305b338-kube-api-access-5qb6b\") pod \"distinct-w3-682999-kserve-74d49d4955-nndlc\" (UID: \"2a35314b-31ea-42e6-b49a-15ab1305b338\") " pod="e2e-models-e2e-worker-w3-682999/distinct-w3-682999-kserve-74d49d4955-nndlc"
Oct 30 20:56:49.447157 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:49.447038    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a35314b-31ea-42e6-b49a-15ab1305b338-tls-certs\") pod \"distinct-w3-682999-kserve-74d49d4955-nndlc\" (UID: \"2a35314b-31ea-42e6-b49a-15ab1305b338\") " pod="e2e-models-e2e-worker-w3-682999/distinct-w3-682999-kserve-74d49d4955-nndlc"
Oct 30 20:56:49.447157 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:49.447070    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a35314b-31ea-42e6-b49a-15ab1305b338-model-cache\") pod \"distinct-w3-682999-kserve-74d49d4955-nndlc\" (UID: \"2a35314b-31ea-42e6-b49a-15ab1305b338\") " pod="e2e-models-e2e-worker-w3-682999/distinct-w3-682999-kserve-74d49d4955-nndlc"
Oct 30 20:56:49.447376 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:49.447344    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a35314b-31ea-42e6-b49a-15ab1305b338-home\") pod \"distinct-w3-682999-kserve-74d49d4955-nndlc\" (UID: \"2a35314b-31ea-42e6-b49a-15ab1305b338\") " pod="e2e-models-e2e-worker-w3-682999/distinct-w3-682999-kserve-74d49d4955-nndlc"
Oct 30 20:56:49.447434 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:49.447419    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a35314b-31ea-42e6-b49a-15ab1305b338-tmp-dir\") pod \"distinct-w3-682999-kserve-74d49d4955-nndlc\" (UID: \"2a35314b-31ea-42e6-b49a-15ab1305b338\") " pod="e2e-models-e2e-worker-w3-682999/distinct-w3-682999-kserve-74d49d4955-nndlc"
Oct 30 20:56:49.447518 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:49.447501    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a35314b-31ea-42e6-b49a-15ab1305b338-model-cache\") pod \"distinct-w3-682999-kserve-74d49d4955-nndlc\" (UID: \"2a35314b-31ea-42e6-b49a-15ab1305b338\") " pod="e2e-models-e2e-worker-w3-682999/distinct-w3-682999-kserve-74d49d4955-nndlc"
Oct 30 20:56:49.449395 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:49.449373    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a35314b-31ea-42e6-b49a-15ab1305b338-dshm\") pod \"distinct-w3-682999-kserve-74d49d4955-nndlc\" (UID: \"2a35314b-31ea-42e6-b49a-15ab1305b338\") " pod="e2e-models-e2e-worker-w3-682999/distinct-w3-682999-kserve-74d49d4955-nndlc"
Oct 30 20:56:49.449818 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:49.449781    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a35314b-31ea-42e6-b49a-15ab1305b338-tls-certs\") pod \"distinct-w3-682999-kserve-74d49d4955-nndlc\" (UID: \"2a35314b-31ea-42e6-b49a-15ab1305b338\") " pod="e2e-models-e2e-worker-w3-682999/distinct-w3-682999-kserve-74d49d4955-nndlc"
Oct 30 20:56:49.504850 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:49.504817    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qb6b\" (UniqueName: \"kubernetes.io/projected/2a35314b-31ea-42e6-b49a-15ab1305b338-kube-api-access-5qb6b\") pod \"distinct-w3-682999-kserve-74d49d4955-nndlc\" (UID: \"2a35314b-31ea-42e6-b49a-15ab1305b338\") " pod="e2e-models-e2e-worker-w3-682999/distinct-w3-682999-kserve-74d49d4955-nndlc"
Oct 30 20:56:49.585520 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:49.585437    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-682999/distinct-w3-682999-kserve-74d49d4955-nndlc"
Oct 30 20:56:49.985047 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:49.985018    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-682999/distinct-w3-682999-kserve-74d49d4955-nndlc"]
Oct 30 20:56:49.985363 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:56:49.985340    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a35314b_31ea_42e6_b49a_15ab1305b338.slice/crio-63c003533c2abf821ef17b3a61bd8ce7e0536031572ec2888255a5828da9378f WatchSource:0}: Error finding container 63c003533c2abf821ef17b3a61bd8ce7e0536031572ec2888255a5828da9378f: Status 404 returned error can't find the container with id 63c003533c2abf821ef17b3a61bd8ce7e0536031572ec2888255a5828da9378f
Oct 30 20:56:50.333956 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:50.333862    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-682999/distinct-w3-682999-kserve-74d49d4955-nndlc" event={"ID":"2a35314b-31ea-42e6-b49a-15ab1305b338","Type":"ContainerStarted","Data":"e187cec6c8d8aa1be06bc77fccee4db794c953591f172f7af75e0d8bb68cbbdf"}
Oct 30 20:56:50.333956 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:50.333909    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-682999/distinct-w3-682999-kserve-74d49d4955-nndlc" event={"ID":"2a35314b-31ea-42e6-b49a-15ab1305b338","Type":"ContainerStarted","Data":"63c003533c2abf821ef17b3a61bd8ce7e0536031572ec2888255a5828da9378f"}
Oct 30 20:56:50.450858 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:50.450798    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-682999/distinct-w3-682999-kserve-74d49d4955-nndlc" podStartSLOduration=1.45077007 podStartE2EDuration="1.45077007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:56:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:56:50.449037729 +0000 UTC m=+1093.513814127" watchObservedRunningTime="2026-10-30 20:56:50.45077007 +0000 UTC m=+1093.515546468"
Oct 30 20:56:50.585606 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:50.585517    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-682999/distinct-w3-682999-kserve-74d49d4955-nndlc"
Oct 30 20:56:54.460027 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:56:54.459991    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 20:56:58.337525 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:58.337493    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-781058/distinct-2-w0-781058-kserve-7f6c654757-ppmhm"]
Oct 30 20:56:58.341648 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:58.341629    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-781058/distinct-2-w0-781058-kserve-7f6c654757-ppmhm"
Oct 30 20:56:58.415117 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:58.415076    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-781058/distinct-2-w0-781058-kserve-7f6c654757-ppmhm"]
Oct 30 20:56:58.443969 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:58.443930    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-781058\"/\"distinct-2-w0-781058-kserve-self-signed-certs\""
Oct 30 20:56:58.450710 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:58.450683    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e818303-0e9f-4ba0-a635-b80b92c9e36d-home\") pod \"distinct-2-w0-781058-kserve-7f6c654757-ppmhm\" (UID: \"5e818303-0e9f-4ba0-a635-b80b92c9e36d\") " pod="e2e-models-e2e-worker-w0-781058/distinct-2-w0-781058-kserve-7f6c654757-ppmhm"
Oct 30 20:56:58.450837 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:58.450714    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e818303-0e9f-4ba0-a635-b80b92c9e36d-model-cache\") pod \"distinct-2-w0-781058-kserve-7f6c654757-ppmhm\" (UID: \"5e818303-0e9f-4ba0-a635-b80b92c9e36d\") " pod="e2e-models-e2e-worker-w0-781058/distinct-2-w0-781058-kserve-7f6c654757-ppmhm"
Oct 30 20:56:58.450837 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:58.450743    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e818303-0e9f-4ba0-a635-b80b92c9e36d-tmp-dir\") pod \"distinct-2-w0-781058-kserve-7f6c654757-ppmhm\" (UID: \"5e818303-0e9f-4ba0-a635-b80b92c9e36d\") " pod="e2e-models-e2e-worker-w0-781058/distinct-2-w0-781058-kserve-7f6c654757-ppmhm"
Oct 30 20:56:58.450935 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:58.450921    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krnzf\" (UniqueName: \"kubernetes.io/projected/5e818303-0e9f-4ba0-a635-b80b92c9e36d-kube-api-access-krnzf\") pod \"distinct-2-w0-781058-kserve-7f6c654757-ppmhm\" (UID: \"5e818303-0e9f-4ba0-a635-b80b92c9e36d\") " pod="e2e-models-e2e-worker-w0-781058/distinct-2-w0-781058-kserve-7f6c654757-ppmhm"
Oct 30 20:56:58.450987 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:58.450973    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e818303-0e9f-4ba0-a635-b80b92c9e36d-dshm\") pod \"distinct-2-w0-781058-kserve-7f6c654757-ppmhm\" (UID: \"5e818303-0e9f-4ba0-a635-b80b92c9e36d\") " pod="e2e-models-e2e-worker-w0-781058/distinct-2-w0-781058-kserve-7f6c654757-ppmhm"
Oct 30 20:56:58.451082 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:58.451067    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e818303-0e9f-4ba0-a635-b80b92c9e36d-tls-certs\") pod \"distinct-2-w0-781058-kserve-7f6c654757-ppmhm\" (UID: \"5e818303-0e9f-4ba0-a635-b80b92c9e36d\") " pod="e2e-models-e2e-worker-w0-781058/distinct-2-w0-781058-kserve-7f6c654757-ppmhm"
Oct 30 20:56:58.551931 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:58.551895    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krnzf\" (UniqueName: \"kubernetes.io/projected/5e818303-0e9f-4ba0-a635-b80b92c9e36d-kube-api-access-krnzf\") pod \"distinct-2-w0-781058-kserve-7f6c654757-ppmhm\" (UID: \"5e818303-0e9f-4ba0-a635-b80b92c9e36d\") " pod="e2e-models-e2e-worker-w0-781058/distinct-2-w0-781058-kserve-7f6c654757-ppmhm"
Oct 30 20:56:58.552148 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:58.551968    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e818303-0e9f-4ba0-a635-b80b92c9e36d-dshm\") pod \"distinct-2-w0-781058-kserve-7f6c654757-ppmhm\" (UID: \"5e818303-0e9f-4ba0-a635-b80b92c9e36d\") " pod="e2e-models-e2e-worker-w0-781058/distinct-2-w0-781058-kserve-7f6c654757-ppmhm"
Oct 30 20:56:58.552148 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:58.552020    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e818303-0e9f-4ba0-a635-b80b92c9e36d-tls-certs\") pod \"distinct-2-w0-781058-kserve-7f6c654757-ppmhm\" (UID: \"5e818303-0e9f-4ba0-a635-b80b92c9e36d\") " pod="e2e-models-e2e-worker-w0-781058/distinct-2-w0-781058-kserve-7f6c654757-ppmhm"
Oct 30 20:56:58.552148 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:58.552095    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e818303-0e9f-4ba0-a635-b80b92c9e36d-home\") pod \"distinct-2-w0-781058-kserve-7f6c654757-ppmhm\" (UID: \"5e818303-0e9f-4ba0-a635-b80b92c9e36d\") " pod="e2e-models-e2e-worker-w0-781058/distinct-2-w0-781058-kserve-7f6c654757-ppmhm"
Oct 30 20:56:58.552148 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:58.552126    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e818303-0e9f-4ba0-a635-b80b92c9e36d-model-cache\") pod \"distinct-2-w0-781058-kserve-7f6c654757-ppmhm\" (UID: \"5e818303-0e9f-4ba0-a635-b80b92c9e36d\") " pod="e2e-models-e2e-worker-w0-781058/distinct-2-w0-781058-kserve-7f6c654757-ppmhm"
Oct 30 20:56:58.552365 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:58.552172    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e818303-0e9f-4ba0-a635-b80b92c9e36d-tmp-dir\") pod \"distinct-2-w0-781058-kserve-7f6c654757-ppmhm\" (UID: \"5e818303-0e9f-4ba0-a635-b80b92c9e36d\") " pod="e2e-models-e2e-worker-w0-781058/distinct-2-w0-781058-kserve-7f6c654757-ppmhm"
Oct 30 20:56:58.552484 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:58.552462    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e818303-0e9f-4ba0-a635-b80b92c9e36d-home\") pod \"distinct-2-w0-781058-kserve-7f6c654757-ppmhm\" (UID: \"5e818303-0e9f-4ba0-a635-b80b92c9e36d\") " pod="e2e-models-e2e-worker-w0-781058/distinct-2-w0-781058-kserve-7f6c654757-ppmhm"
Oct 30 20:56:58.552538 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:58.552487    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e818303-0e9f-4ba0-a635-b80b92c9e36d-model-cache\") pod \"distinct-2-w0-781058-kserve-7f6c654757-ppmhm\" (UID: \"5e818303-0e9f-4ba0-a635-b80b92c9e36d\") " pod="e2e-models-e2e-worker-w0-781058/distinct-2-w0-781058-kserve-7f6c654757-ppmhm"
Oct 30 20:56:58.552589 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:58.552537    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e818303-0e9f-4ba0-a635-b80b92c9e36d-tmp-dir\") pod \"distinct-2-w0-781058-kserve-7f6c654757-ppmhm\" (UID: \"5e818303-0e9f-4ba0-a635-b80b92c9e36d\") " pod="e2e-models-e2e-worker-w0-781058/distinct-2-w0-781058-kserve-7f6c654757-ppmhm"
Oct 30 20:56:58.554466 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:58.554447    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e818303-0e9f-4ba0-a635-b80b92c9e36d-dshm\") pod \"distinct-2-w0-781058-kserve-7f6c654757-ppmhm\" (UID: \"5e818303-0e9f-4ba0-a635-b80b92c9e36d\") " pod="e2e-models-e2e-worker-w0-781058/distinct-2-w0-781058-kserve-7f6c654757-ppmhm"
Oct 30 20:56:58.554707 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:58.554690    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e818303-0e9f-4ba0-a635-b80b92c9e36d-tls-certs\") pod \"distinct-2-w0-781058-kserve-7f6c654757-ppmhm\" (UID: \"5e818303-0e9f-4ba0-a635-b80b92c9e36d\") " pod="e2e-models-e2e-worker-w0-781058/distinct-2-w0-781058-kserve-7f6c654757-ppmhm"
Oct 30 20:56:58.610275 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:58.610193    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krnzf\" (UniqueName: \"kubernetes.io/projected/5e818303-0e9f-4ba0-a635-b80b92c9e36d-kube-api-access-krnzf\") pod \"distinct-2-w0-781058-kserve-7f6c654757-ppmhm\" (UID: \"5e818303-0e9f-4ba0-a635-b80b92c9e36d\") " pod="e2e-models-e2e-worker-w0-781058/distinct-2-w0-781058-kserve-7f6c654757-ppmhm"
Oct 30 20:56:58.654884 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:58.654834    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-781058/distinct-2-w0-781058-kserve-7f6c654757-ppmhm"
Oct 30 20:56:59.064598 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:56:59.064569    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5e818303_0e9f_4ba0_a635_b80b92c9e36d.slice/crio-9ae3ab343b880d6ca1d77d18eb1e1fc6f0c7995109fae2526ee365a334fc5d5a WatchSource:0}: Error finding container 9ae3ab343b880d6ca1d77d18eb1e1fc6f0c7995109fae2526ee365a334fc5d5a: Status 404 returned error can't find the container with id 9ae3ab343b880d6ca1d77d18eb1e1fc6f0c7995109fae2526ee365a334fc5d5a
Oct 30 20:56:59.065832 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:59.065807    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-781058/distinct-2-w0-781058-kserve-7f6c654757-ppmhm"]
Oct 30 20:56:59.399126 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:59.399034    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-781058/distinct-2-w0-781058-kserve-7f6c654757-ppmhm" event={"ID":"5e818303-0e9f-4ba0-a635-b80b92c9e36d","Type":"ContainerStarted","Data":"d274c2e68c23dcac393ac0a2d3e75375c212cda703367c93087c24fe7a7708b3"}
Oct 30 20:56:59.399126 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:59.399085    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-781058/distinct-2-w0-781058-kserve-7f6c654757-ppmhm" event={"ID":"5e818303-0e9f-4ba0-a635-b80b92c9e36d","Type":"ContainerStarted","Data":"9ae3ab343b880d6ca1d77d18eb1e1fc6f0c7995109fae2526ee365a334fc5d5a"}
Oct 30 20:56:59.515436 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:59.515381    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-781058/distinct-2-w0-781058-kserve-7f6c654757-ppmhm" podStartSLOduration=1.5153657040000001 podStartE2EDuration="1.515365704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:56:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:56:59.512233049 +0000 UTC m=+1102.577009447" watchObservedRunningTime="2026-10-30 20:56:59.515365704 +0000 UTC m=+1102.580142109"
Oct 30 20:56:59.586305 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:59.586269    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-682999/distinct-w3-682999-kserve-74d49d4955-nndlc"
Oct 30 20:56:59.599140 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:59.599111    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-682999/distinct-w3-682999-kserve-74d49d4955-nndlc"
Oct 30 20:56:59.655916 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:56:59.655841    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-781058/distinct-2-w0-781058-kserve-7f6c654757-ppmhm"
Oct 30 20:57:00.417891 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:00.417865    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-682999/distinct-w3-682999-kserve-74d49d4955-nndlc"
Oct 30 20:57:08.655535 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:08.655496    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-781058/distinct-2-w0-781058-kserve-7f6c654757-ppmhm"
Oct 30 20:57:08.668346 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:08.668317    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-781058/distinct-2-w0-781058-kserve-7f6c654757-ppmhm"
Oct 30 20:57:09.480916 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:09.480887    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-781058/distinct-2-w0-781058-kserve-7f6c654757-ppmhm"
Oct 30 20:57:24.462124 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:57:24.462082    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 20:57:46.996534 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:46.996500    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv"]
Oct 30 20:57:47.001028 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:47.001002    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv"
Oct 30 20:57:47.053610 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:47.053577    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-781058-openshift-default-dockercfg-mkvr5\""
Oct 30 20:57:47.107122 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:47.107091    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv"]
Oct 30 20:57:47.157461 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:47.157410    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f02b6741-5ee5-49d1-8d70-775cadd1b75d-istio-envoy\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv\" (UID: \"f02b6741-5ee5-49d1-8d70-775cadd1b75d\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv"
Oct 30 20:57:47.157660 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:47.157485    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f02b6741-5ee5-49d1-8d70-775cadd1b75d-istio-token\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv\" (UID: \"f02b6741-5ee5-49d1-8d70-775cadd1b75d\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv"
Oct 30 20:57:47.157660 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:47.157560    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f02b6741-5ee5-49d1-8d70-775cadd1b75d-istiod-ca-cert\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv\" (UID: \"f02b6741-5ee5-49d1-8d70-775cadd1b75d\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv"
Oct 30 20:57:47.157660 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:47.157625    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f02b6741-5ee5-49d1-8d70-775cadd1b75d-workload-socket\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv\" (UID: \"f02b6741-5ee5-49d1-8d70-775cadd1b75d\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv"
Oct 30 20:57:47.157800 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:47.157662    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f02b6741-5ee5-49d1-8d70-775cadd1b75d-workload-certs\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv\" (UID: \"f02b6741-5ee5-49d1-8d70-775cadd1b75d\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv"
Oct 30 20:57:47.157800 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:47.157735    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f02b6741-5ee5-49d1-8d70-775cadd1b75d-istio-podinfo\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv\" (UID: \"f02b6741-5ee5-49d1-8d70-775cadd1b75d\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv"
Oct 30 20:57:47.157902 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:47.157840    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f02b6741-5ee5-49d1-8d70-775cadd1b75d-credential-socket\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv\" (UID: \"f02b6741-5ee5-49d1-8d70-775cadd1b75d\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv"
Oct 30 20:57:47.157902 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:47.157862    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f02b6741-5ee5-49d1-8d70-775cadd1b75d-istio-data\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv\" (UID: \"f02b6741-5ee5-49d1-8d70-775cadd1b75d\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv"
Oct 30 20:57:47.157985 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:47.157898    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx6ld\" (UniqueName: \"kubernetes.io/projected/f02b6741-5ee5-49d1-8d70-775cadd1b75d-kube-api-access-sx6ld\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv\" (UID: \"f02b6741-5ee5-49d1-8d70-775cadd1b75d\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv"
Oct 30 20:57:47.259752 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:47.259662    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f02b6741-5ee5-49d1-8d70-775cadd1b75d-istio-envoy\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv\" (UID: \"f02b6741-5ee5-49d1-8d70-775cadd1b75d\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv"
Oct 30 20:57:47.259752 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:47.259723    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f02b6741-5ee5-49d1-8d70-775cadd1b75d-istio-token\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv\" (UID: \"f02b6741-5ee5-49d1-8d70-775cadd1b75d\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv"
Oct 30 20:57:47.260025 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:47.259758    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f02b6741-5ee5-49d1-8d70-775cadd1b75d-istiod-ca-cert\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv\" (UID: \"f02b6741-5ee5-49d1-8d70-775cadd1b75d\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv"
Oct 30 20:57:47.260025 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:47.259873    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f02b6741-5ee5-49d1-8d70-775cadd1b75d-workload-socket\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv\" (UID: \"f02b6741-5ee5-49d1-8d70-775cadd1b75d\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv"
Oct 30 20:57:47.260025 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:47.259911    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f02b6741-5ee5-49d1-8d70-775cadd1b75d-workload-certs\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv\" (UID: \"f02b6741-5ee5-49d1-8d70-775cadd1b75d\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv"
Oct 30 20:57:47.260025 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:47.259941    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f02b6741-5ee5-49d1-8d70-775cadd1b75d-istio-podinfo\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv\" (UID: \"f02b6741-5ee5-49d1-8d70-775cadd1b75d\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv"
Oct 30 20:57:47.260025 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:47.260009    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f02b6741-5ee5-49d1-8d70-775cadd1b75d-credential-socket\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv\" (UID: \"f02b6741-5ee5-49d1-8d70-775cadd1b75d\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv"
Oct 30 20:57:47.260283 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:47.260041    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f02b6741-5ee5-49d1-8d70-775cadd1b75d-istio-data\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv\" (UID: \"f02b6741-5ee5-49d1-8d70-775cadd1b75d\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv"
Oct 30 20:57:47.260283 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:47.260131    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx6ld\" (UniqueName: \"kubernetes.io/projected/f02b6741-5ee5-49d1-8d70-775cadd1b75d-kube-api-access-sx6ld\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv\" (UID: \"f02b6741-5ee5-49d1-8d70-775cadd1b75d\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv"
Oct 30 20:57:47.260394 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:47.260294    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f02b6741-5ee5-49d1-8d70-775cadd1b75d-workload-socket\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv\" (UID: \"f02b6741-5ee5-49d1-8d70-775cadd1b75d\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv"
Oct 30 20:57:47.260394 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:47.260366    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f02b6741-5ee5-49d1-8d70-775cadd1b75d-istio-data\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv\" (UID: \"f02b6741-5ee5-49d1-8d70-775cadd1b75d\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv"
Oct 30 20:57:47.260499 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:47.260401    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f02b6741-5ee5-49d1-8d70-775cadd1b75d-workload-certs\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv\" (UID: \"f02b6741-5ee5-49d1-8d70-775cadd1b75d\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv"
Oct 30 20:57:47.260499 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:47.260464    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f02b6741-5ee5-49d1-8d70-775cadd1b75d-credential-socket\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv\" (UID: \"f02b6741-5ee5-49d1-8d70-775cadd1b75d\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv"
Oct 30 20:57:47.260624 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:47.260604    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f02b6741-5ee5-49d1-8d70-775cadd1b75d-istiod-ca-cert\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv\" (UID: \"f02b6741-5ee5-49d1-8d70-775cadd1b75d\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv"
Oct 30 20:57:47.262299 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:47.262272    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f02b6741-5ee5-49d1-8d70-775cadd1b75d-istio-envoy\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv\" (UID: \"f02b6741-5ee5-49d1-8d70-775cadd1b75d\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv"
Oct 30 20:57:47.262930 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:47.262906    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f02b6741-5ee5-49d1-8d70-775cadd1b75d-istio-podinfo\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv\" (UID: \"f02b6741-5ee5-49d1-8d70-775cadd1b75d\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv"
Oct 30 20:57:47.317410 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:47.317381    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f02b6741-5ee5-49d1-8d70-775cadd1b75d-istio-token\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv\" (UID: \"f02b6741-5ee5-49d1-8d70-775cadd1b75d\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv"
Oct 30 20:57:47.317591 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:47.317488    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx6ld\" (UniqueName: \"kubernetes.io/projected/f02b6741-5ee5-49d1-8d70-775cadd1b75d-kube-api-access-sx6ld\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv\" (UID: \"f02b6741-5ee5-49d1-8d70-775cadd1b75d\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv"
Oct 30 20:57:47.617364 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:47.617277    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv"
Oct 30 20:57:48.048138 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:48.048107    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv"]
Oct 30 20:57:48.048943 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:57:48.048915    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf02b6741_5ee5_49d1_8d70_775cadd1b75d.slice/crio-91ce0b08b77bb7cad1b152c4b68870724f3ddf6f306fe83b2d678311d9e2e34b WatchSource:0}: Error finding container 91ce0b08b77bb7cad1b152c4b68870724f3ddf6f306fe83b2d678311d9e2e34b: Status 404 returned error can't find the container with id 91ce0b08b77bb7cad1b152c4b68870724f3ddf6f306fe83b2d678311d9e2e34b
Oct 30 20:57:48.051514 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:48.051483    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 20:57:48.051589 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:48.051558    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 20:57:48.051633 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:48.051587    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 20:57:48.749734 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:48.749686    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv" event={"ID":"f02b6741-5ee5-49d1-8d70-775cadd1b75d","Type":"ContainerStarted","Data":"3da89a0b2ec5b12d7de5cbb79bc162d51563dced32e513eaa8d36b659c65e6ca"}
Oct 30 20:57:48.750075 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:48.750052    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv" event={"ID":"f02b6741-5ee5-49d1-8d70-775cadd1b75d","Type":"ContainerStarted","Data":"91ce0b08b77bb7cad1b152c4b68870724f3ddf6f306fe83b2d678311d9e2e34b"}
Oct 30 20:57:48.927632 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:48.927570    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv" podStartSLOduration=2.927552087 podStartE2EDuration="2.927552087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:57:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:57:48.923727754 +0000 UTC m=+1151.988504153" watchObservedRunningTime="2026-10-30 20:57:48.927552087 +0000 UTC m=+1151.992328550"
Oct 30 20:57:49.618442 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:49.618409    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv"
Oct 30 20:57:49.619988 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:49.619963    5713 patch_prober.go:29] interesting pod/e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 30 20:57:49.620101 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:49.620014    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv" podUID="f02b6741-5ee5-49d1-8d70-775cadd1b75d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 30 20:57:50.617777 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:50.617731    5713 patch_prober.go:29] interesting pod/e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 30 20:57:50.617993 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:50.617815    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv" podUID="f02b6741-5ee5-49d1-8d70-775cadd1b75d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 30 20:57:51.617704 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:51.617668    5713 patch_prober.go:29] interesting pod/e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 30 20:57:51.618150 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:51.617728    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv" podUID="f02b6741-5ee5-49d1-8d70-775cadd1b75d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 30 20:57:52.618152 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:52.618120    5713 patch_prober.go:29] interesting pod/e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 30 20:57:52.618648 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:52.618172    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv" podUID="f02b6741-5ee5-49d1-8d70-775cadd1b75d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 30 20:57:53.618022 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:53.617983    5713 patch_prober.go:29] interesting pod/e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 30 20:57:53.618216 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:53.618062    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv" podUID="f02b6741-5ee5-49d1-8d70-775cadd1b75d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 30 20:57:54.459341 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:57:54.459302    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 20:57:54.624813 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:54.619215    5713 patch_prober.go:29] interesting pod/e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 30 20:57:54.624813 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:54.619296    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv" podUID="f02b6741-5ee5-49d1-8d70-775cadd1b75d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 30 20:57:55.065264 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:55.065229    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c72e675-npr68"]
Oct 30 20:57:55.069463 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:55.069440    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c72e675-npr68"
Oct 30 20:57:55.074689 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:55.074661    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bfeafa0-144d-4b07-acf4-800dbb9a8072-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c72e675-npr68\" (UID: \"6bfeafa0-144d-4b07-acf4-800dbb9a8072\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c72e675-npr68"
Oct 30 20:57:55.074850 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:55.074700    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sfp9\" (UniqueName: \"kubernetes.io/projected/6bfeafa0-144d-4b07-acf4-800dbb9a8072-kube-api-access-4sfp9\") pod \"maas-api-revoke-sub-8c72e675-npr68\" (UID: \"6bfeafa0-144d-4b07-acf4-800dbb9a8072\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c72e675-npr68"
Oct 30 20:57:55.074929 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:55.074868    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bfeafa0-144d-4b07-acf4-800dbb9a8072-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c72e675-npr68\" (UID: \"6bfeafa0-144d-4b07-acf4-800dbb9a8072\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c72e675-npr68"
Oct 30 20:57:55.118661 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:55.118627    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c72e675-npr68"]
Oct 30 20:57:55.163159 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:55.163123    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:57:55.163349 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:55.163198    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 20:57:55.163349 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:55.163276    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:57:55.171411 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:55.171393    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:57:55.176078 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:55.176056    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bfeafa0-144d-4b07-acf4-800dbb9a8072-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c72e675-npr68\" (UID: \"6bfeafa0-144d-4b07-acf4-800dbb9a8072\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c72e675-npr68"
Oct 30 20:57:55.176189 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:55.176090    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sfp9\" (UniqueName: \"kubernetes.io/projected/6bfeafa0-144d-4b07-acf4-800dbb9a8072-kube-api-access-4sfp9\") pod \"maas-api-revoke-sub-8c72e675-npr68\" (UID: \"6bfeafa0-144d-4b07-acf4-800dbb9a8072\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c72e675-npr68"
Oct 30 20:57:55.176189 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:55.176153    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bfeafa0-144d-4b07-acf4-800dbb9a8072-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c72e675-npr68\" (UID: \"6bfeafa0-144d-4b07-acf4-800dbb9a8072\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c72e675-npr68"
Oct 30 20:57:55.177063 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:55.177037    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bfeafa0-144d-4b07-acf4-800dbb9a8072-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c72e675-npr68\" (UID: \"6bfeafa0-144d-4b07-acf4-800dbb9a8072\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c72e675-npr68"
Oct 30 20:57:55.178563 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:55.178542    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bfeafa0-144d-4b07-acf4-800dbb9a8072-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c72e675-npr68\" (UID: \"6bfeafa0-144d-4b07-acf4-800dbb9a8072\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c72e675-npr68"
Oct 30 20:57:55.233488 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:55.233458    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sfp9\" (UniqueName: \"kubernetes.io/projected/6bfeafa0-144d-4b07-acf4-800dbb9a8072-kube-api-access-4sfp9\") pod \"maas-api-revoke-sub-8c72e675-npr68\" (UID: \"6bfeafa0-144d-4b07-acf4-800dbb9a8072\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c72e675-npr68"
Oct 30 20:57:55.386997 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:55.386902    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c72e675-npr68"
Oct 30 20:57:55.622736 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:55.622708    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv"
Oct 30 20:57:55.623217 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:55.623202    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv"
Oct 30 20:57:55.623933 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:55.623916    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv"
Oct 30 20:57:55.776979 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:55.776953    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c72e675-npr68"]
Oct 30 20:57:55.816532 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:57:55.816503    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bfeafa0_144d_4b07_acf4_800dbb9a8072.slice/crio-3e4a9f21c698379ad0b745a097474da30bcbb3f291426df70f3228578d5a8747 WatchSource:0}: Error finding container 3e4a9f21c698379ad0b745a097474da30bcbb3f291426df70f3228578d5a8747: Status 404 returned error can't find the container with id 3e4a9f21c698379ad0b745a097474da30bcbb3f291426df70f3228578d5a8747
Oct 30 20:57:56.813773 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:56.813737    5713 generic.go:356] "Generic (PLEG): container finished" podID="6bfeafa0-144d-4b07-acf4-800dbb9a8072" containerID="f422395d38ac91efcb119f8113c34046dfd647edce573fa71089afa63a298c63" exitCode=0
Oct 30 20:57:56.814202 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:56.813825    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c72e675-npr68" event={"ID":"6bfeafa0-144d-4b07-acf4-800dbb9a8072","Type":"ContainerDied","Data":"f422395d38ac91efcb119f8113c34046dfd647edce573fa71089afa63a298c63"}
Oct 30 20:57:56.814202 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:56.813891    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c72e675-npr68" event={"ID":"6bfeafa0-144d-4b07-acf4-800dbb9a8072","Type":"ContainerStarted","Data":"3e4a9f21c698379ad0b745a097474da30bcbb3f291426df70f3228578d5a8747"}
Oct 30 20:57:57.962779 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:57.962746    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c72e675-npr68"
Oct 30 20:57:58.006518 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:58.006488    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bfeafa0-144d-4b07-acf4-800dbb9a8072-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bfeafa0-144d-4b07-acf4-800dbb9a8072-maas-api-openshift-ca\") pod \"6bfeafa0-144d-4b07-acf4-800dbb9a8072\" (UID: \"6bfeafa0-144d-4b07-acf4-800dbb9a8072\") "
Oct 30 20:57:58.006723 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:58.006615    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bfeafa0-144d-4b07-acf4-800dbb9a8072-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bfeafa0-144d-4b07-acf4-800dbb9a8072-maas-api-xks-ca\") pod \"6bfeafa0-144d-4b07-acf4-800dbb9a8072\" (UID: \"6bfeafa0-144d-4b07-acf4-800dbb9a8072\") "
Oct 30 20:57:58.006723 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:58.006665    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bfeafa0-144d-4b07-acf4-800dbb9a8072-kube-api-access-4sfp9\" (UniqueName: \"kubernetes.io/projected/6bfeafa0-144d-4b07-acf4-800dbb9a8072-kube-api-access-4sfp9\") pod \"6bfeafa0-144d-4b07-acf4-800dbb9a8072\" (UID: \"6bfeafa0-144d-4b07-acf4-800dbb9a8072\") "
Oct 30 20:57:58.006944 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:58.006915    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bfeafa0-144d-4b07-acf4-800dbb9a8072-maas-api-openshift-ca" pod "6bfeafa0-144d-4b07-acf4-800dbb9a8072" (UID: "6bfeafa0-144d-4b07-acf4-800dbb9a8072"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:57:58.007201 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:58.007184    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bfeafa0-144d-4b07-acf4-800dbb9a8072-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:58.009472 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:58.009446    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bfeafa0-144d-4b07-acf4-800dbb9a8072-maas-api-xks-ca" pod "6bfeafa0-144d-4b07-acf4-800dbb9a8072" (UID: "6bfeafa0-144d-4b07-acf4-800dbb9a8072"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:57:58.009588 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:58.009457    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bfeafa0-144d-4b07-acf4-800dbb9a8072-kube-api-access-4sfp9" pod "6bfeafa0-144d-4b07-acf4-800dbb9a8072" (UID: "6bfeafa0-144d-4b07-acf4-800dbb9a8072"). InnerVolumeSpecName "kube-api-access-4sfp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:57:58.108420 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:58.108334    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bfeafa0-144d-4b07-acf4-800dbb9a8072-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:58.108420 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:58.108365    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sfp9\" (UniqueName: \"kubernetes.io/projected/6bfeafa0-144d-4b07-acf4-800dbb9a8072-kube-api-access-4sfp9\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:57:58.829569 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:58.829536    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c72e675-npr68" event={"ID":"6bfeafa0-144d-4b07-acf4-800dbb9a8072","Type":"ContainerDied","Data":"3e4a9f21c698379ad0b745a097474da30bcbb3f291426df70f3228578d5a8747"}
Oct 30 20:57:58.829569 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:58.829567    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c72e675-npr68"
Oct 30 20:57:58.829823 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:57:58.829583    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e4a9f21c698379ad0b745a097474da30bcbb3f291426df70f3228578d5a8747"
Oct 30 20:58:10.153967 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:10.153935    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4278716-rj7l7"]
Oct 30 20:58:10.155044 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:10.154778    5713 cpu_manager.go:414] "containerMap: removing container" podUID="6bfeafa0-144d-4b07-acf4-800dbb9a8072" containerName="revoke-keys"
Oct 30 20:58:10.155044 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:10.154820    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bfeafa0-144d-4b07-acf4-800dbb9a8072" containerName="revoke-keys"
Oct 30 20:58:10.155044 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:10.154955    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bfeafa0-144d-4b07-acf4-800dbb9a8072" containerName="revoke-keys"
Oct 30 20:58:10.158687 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:10.158668    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4278716-rj7l7"
Oct 30 20:58:10.214046 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:10.214006    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4278716-rj7l7"]
Oct 30 20:58:10.253574 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:10.253530    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n69dg\" (UniqueName: \"kubernetes.io/projected/8481d84b-2a48-4160-8f0e-b239d27d2d2b-kube-api-access-n69dg\") pod \"maas-api-revoke-sub-d4278716-rj7l7\" (UID: \"8481d84b-2a48-4160-8f0e-b239d27d2d2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4278716-rj7l7"
Oct 30 20:58:10.253754 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:10.253720    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8481d84b-2a48-4160-8f0e-b239d27d2d2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4278716-rj7l7\" (UID: \"8481d84b-2a48-4160-8f0e-b239d27d2d2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4278716-rj7l7"
Oct 30 20:58:10.253828 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:10.253757    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8481d84b-2a48-4160-8f0e-b239d27d2d2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4278716-rj7l7\" (UID: \"8481d84b-2a48-4160-8f0e-b239d27d2d2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4278716-rj7l7"
Oct 30 20:58:10.259275 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:10.259253    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:58:10.259275 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:10.259265    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:58:10.259421 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:10.259286    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:58:10.260317 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:10.260300    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 20:58:10.354983 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:10.354947    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8481d84b-2a48-4160-8f0e-b239d27d2d2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4278716-rj7l7\" (UID: \"8481d84b-2a48-4160-8f0e-b239d27d2d2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4278716-rj7l7"
Oct 30 20:58:10.354983 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:10.354988    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8481d84b-2a48-4160-8f0e-b239d27d2d2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4278716-rj7l7\" (UID: \"8481d84b-2a48-4160-8f0e-b239d27d2d2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4278716-rj7l7"
Oct 30 20:58:10.355205 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:10.355072    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n69dg\" (UniqueName: \"kubernetes.io/projected/8481d84b-2a48-4160-8f0e-b239d27d2d2b-kube-api-access-n69dg\") pod \"maas-api-revoke-sub-d4278716-rj7l7\" (UID: \"8481d84b-2a48-4160-8f0e-b239d27d2d2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4278716-rj7l7"
Oct 30 20:58:10.355673 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:10.355653    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8481d84b-2a48-4160-8f0e-b239d27d2d2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4278716-rj7l7\" (UID: \"8481d84b-2a48-4160-8f0e-b239d27d2d2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4278716-rj7l7"
Oct 30 20:58:10.357514 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:10.357495    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8481d84b-2a48-4160-8f0e-b239d27d2d2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4278716-rj7l7\" (UID: \"8481d84b-2a48-4160-8f0e-b239d27d2d2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4278716-rj7l7"
Oct 30 20:58:10.412050 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:10.411980    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n69dg\" (UniqueName: \"kubernetes.io/projected/8481d84b-2a48-4160-8f0e-b239d27d2d2b-kube-api-access-n69dg\") pod \"maas-api-revoke-sub-d4278716-rj7l7\" (UID: \"8481d84b-2a48-4160-8f0e-b239d27d2d2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4278716-rj7l7"
Oct 30 20:58:10.471842 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:10.471765    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4278716-rj7l7"
Oct 30 20:58:10.849496 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:58:10.849463    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8481d84b_2a48_4160_8f0e_b239d27d2d2b.slice/crio-9adef4143cc6ca1adbefa1f52556e63243a760cd9d09317af8250bfcc21c7e1f WatchSource:0}: Error finding container 9adef4143cc6ca1adbefa1f52556e63243a760cd9d09317af8250bfcc21c7e1f: Status 404 returned error can't find the container with id 9adef4143cc6ca1adbefa1f52556e63243a760cd9d09317af8250bfcc21c7e1f
Oct 30 20:58:10.849668 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:10.849517    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4278716-rj7l7"]
Oct 30 20:58:10.930856 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:10.930820    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4278716-rj7l7" event={"ID":"8481d84b-2a48-4160-8f0e-b239d27d2d2b","Type":"ContainerStarted","Data":"9adef4143cc6ca1adbefa1f52556e63243a760cd9d09317af8250bfcc21c7e1f"}
Oct 30 20:58:11.940398 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:11.940314    5713 generic.go:356] "Generic (PLEG): container finished" podID="8481d84b-2a48-4160-8f0e-b239d27d2d2b" containerID="12b13e7f0ad225d372ba1b7dd88d224c95a1382b608683a1615b7f7aaeb05d8e" exitCode=0
Oct 30 20:58:11.940814 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:11.940401    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4278716-rj7l7" event={"ID":"8481d84b-2a48-4160-8f0e-b239d27d2d2b","Type":"ContainerDied","Data":"12b13e7f0ad225d372ba1b7dd88d224c95a1382b608683a1615b7f7aaeb05d8e"}
Oct 30 20:58:13.107665 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:13.107638    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4278716-rj7l7"
Oct 30 20:58:13.196166 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:13.196129    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8481d84b-2a48-4160-8f0e-b239d27d2d2b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8481d84b-2a48-4160-8f0e-b239d27d2d2b-maas-api-openshift-ca\") pod \"8481d84b-2a48-4160-8f0e-b239d27d2d2b\" (UID: \"8481d84b-2a48-4160-8f0e-b239d27d2d2b\") "
Oct 30 20:58:13.196348 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:13.196286    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8481d84b-2a48-4160-8f0e-b239d27d2d2b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8481d84b-2a48-4160-8f0e-b239d27d2d2b-maas-api-xks-ca\") pod \"8481d84b-2a48-4160-8f0e-b239d27d2d2b\" (UID: \"8481d84b-2a48-4160-8f0e-b239d27d2d2b\") "
Oct 30 20:58:13.196461 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:13.196447    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8481d84b-2a48-4160-8f0e-b239d27d2d2b-kube-api-access-n69dg\" (UniqueName: \"kubernetes.io/projected/8481d84b-2a48-4160-8f0e-b239d27d2d2b-kube-api-access-n69dg\") pod \"8481d84b-2a48-4160-8f0e-b239d27d2d2b\" (UID: \"8481d84b-2a48-4160-8f0e-b239d27d2d2b\") "
Oct 30 20:58:13.196565 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:13.196539    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8481d84b-2a48-4160-8f0e-b239d27d2d2b-maas-api-openshift-ca" pod "8481d84b-2a48-4160-8f0e-b239d27d2d2b" (UID: "8481d84b-2a48-4160-8f0e-b239d27d2d2b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:58:13.196847 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:13.196825    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8481d84b-2a48-4160-8f0e-b239d27d2d2b-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:13.198633 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:13.198607    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8481d84b-2a48-4160-8f0e-b239d27d2d2b-kube-api-access-n69dg" pod "8481d84b-2a48-4160-8f0e-b239d27d2d2b" (UID: "8481d84b-2a48-4160-8f0e-b239d27d2d2b"). InnerVolumeSpecName "kube-api-access-n69dg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:58:13.198721 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:13.198636    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8481d84b-2a48-4160-8f0e-b239d27d2d2b-maas-api-xks-ca" pod "8481d84b-2a48-4160-8f0e-b239d27d2d2b" (UID: "8481d84b-2a48-4160-8f0e-b239d27d2d2b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:58:13.298180 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:13.298142    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n69dg\" (UniqueName: \"kubernetes.io/projected/8481d84b-2a48-4160-8f0e-b239d27d2d2b-kube-api-access-n69dg\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:13.298180 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:13.298174    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8481d84b-2a48-4160-8f0e-b239d27d2d2b-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:13.958633 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:13.958604    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4278716-rj7l7"
Oct 30 20:58:13.958838 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:13.958639    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4278716-rj7l7" event={"ID":"8481d84b-2a48-4160-8f0e-b239d27d2d2b","Type":"ContainerDied","Data":"9adef4143cc6ca1adbefa1f52556e63243a760cd9d09317af8250bfcc21c7e1f"}
Oct 30 20:58:13.958838 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:13.958687    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9adef4143cc6ca1adbefa1f52556e63243a760cd9d09317af8250bfcc21c7e1f"
Oct 30 20:58:19.155955 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:19.155922    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-8fa2a7/distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm"]
Oct 30 20:58:19.156490 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:19.156476    5713 cpu_manager.go:414] "containerMap: removing container" podUID="8481d84b-2a48-4160-8f0e-b239d27d2d2b" containerName="revoke-keys"
Oct 30 20:58:19.156539 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:19.156492    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8481d84b-2a48-4160-8f0e-b239d27d2d2b" containerName="revoke-keys"
Oct 30 20:58:19.156596 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:19.156587    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="8481d84b-2a48-4160-8f0e-b239d27d2d2b" containerName="revoke-keys"
Oct 30 20:58:19.160214 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:19.160197    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm"
Oct 30 20:58:19.218820 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:19.218757    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-8fa2a7/distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm"]
Oct 30 20:58:19.263733 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:19.263702    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-8fa2a7\"/\"distinct-2-w6-8fa2a7-kserve-self-signed-certs\""
Oct 30 20:58:19.280407 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:19.280372    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776-tls-certs\") pod \"distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm\" (UID: \"cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776\") " pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm"
Oct 30 20:58:19.280407 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:19.280409    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776-model-cache\") pod \"distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm\" (UID: \"cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776\") " pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm"
Oct 30 20:58:19.280600 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:19.280557    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776-tmp-dir\") pod \"distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm\" (UID: \"cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776\") " pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm"
Oct 30 20:58:19.280600 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:19.280587    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776-dshm\") pod \"distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm\" (UID: \"cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776\") " pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm"
Oct 30 20:58:19.280770 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:19.280756    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7j6k\" (UniqueName: \"kubernetes.io/projected/cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776-kube-api-access-r7j6k\") pod \"distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm\" (UID: \"cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776\") " pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm"
Oct 30 20:58:19.280849 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:19.280781    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776-home\") pod \"distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm\" (UID: \"cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776\") " pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm"
Oct 30 20:58:19.382304 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:19.382265    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7j6k\" (UniqueName: \"kubernetes.io/projected/cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776-kube-api-access-r7j6k\") pod \"distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm\" (UID: \"cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776\") " pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm"
Oct 30 20:58:19.382304 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:19.382310    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776-home\") pod \"distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm\" (UID: \"cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776\") " pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm"
Oct 30 20:58:19.382572 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:19.382331    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776-tls-certs\") pod \"distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm\" (UID: \"cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776\") " pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm"
Oct 30 20:58:19.382572 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:19.382346    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776-model-cache\") pod \"distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm\" (UID: \"cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776\") " pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm"
Oct 30 20:58:19.382572 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:19.382422    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776-tmp-dir\") pod \"distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm\" (UID: \"cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776\") " pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm"
Oct 30 20:58:19.382572 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:19.382438    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776-dshm\") pod \"distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm\" (UID: \"cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776\") " pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm"
Oct 30 20:58:19.382841 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:19.382774    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776-home\") pod \"distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm\" (UID: \"cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776\") " pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm"
Oct 30 20:58:19.382910 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:19.382849    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776-tmp-dir\") pod \"distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm\" (UID: \"cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776\") " pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm"
Oct 30 20:58:19.382910 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:19.382822    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776-model-cache\") pod \"distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm\" (UID: \"cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776\") " pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm"
Oct 30 20:58:19.384963 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:19.384942    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776-dshm\") pod \"distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm\" (UID: \"cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776\") " pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm"
Oct 30 20:58:19.385224 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:19.385208    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776-tls-certs\") pod \"distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm\" (UID: \"cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776\") " pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm"
Oct 30 20:58:19.440293 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:19.440267    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7j6k\" (UniqueName: \"kubernetes.io/projected/cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776-kube-api-access-r7j6k\") pod \"distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm\" (UID: \"cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776\") " pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm"
Oct 30 20:58:19.472658 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:19.472622    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm"
Oct 30 20:58:19.863383 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:19.863353    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-8fa2a7/distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm"]
Oct 30 20:58:19.864931 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:58:19.864904    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd7dd1ea_85b8_4601_bb8e_0ceaa24f5776.slice/crio-61b16e2cd5ab53974a9be6a718148498fe26781fe2280858aa3c6f0886f82af1 WatchSource:0}: Error finding container 61b16e2cd5ab53974a9be6a718148498fe26781fe2280858aa3c6f0886f82af1: Status 404 returned error can't find the container with id 61b16e2cd5ab53974a9be6a718148498fe26781fe2280858aa3c6f0886f82af1
Oct 30 20:58:20.004543 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:20.004505    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm" event={"ID":"cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776","Type":"ContainerStarted","Data":"317fcab0854178e48daa348f091f4a7abaf1653ccf67bfae69aa14db368026ad"}
Oct 30 20:58:20.004720 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:20.004559    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm" event={"ID":"cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776","Type":"ContainerStarted","Data":"61b16e2cd5ab53974a9be6a718148498fe26781fe2280858aa3c6f0886f82af1"}
Oct 30 20:58:20.123909 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:20.123775    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm" podStartSLOduration=1.123760522 podStartE2EDuration="1.123760522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:58:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:58:20.120956165 +0000 UTC m=+1183.185732564" watchObservedRunningTime="2026-10-30 20:58:20.123760522 +0000 UTC m=+1183.188536920"
Oct 30 20:58:20.472913 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:20.472877    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm"
Oct 30 20:58:24.459738 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:58:24.459700    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 20:58:29.472996 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:29.472960    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm"
Oct 30 20:58:29.486327 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:29.486301    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm"
Oct 30 20:58:30.113353 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:30.113323    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm"
Oct 30 20:58:35.350409 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:35.350375    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d562e94-59xbh"]
Oct 30 20:58:35.354370 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:35.354352    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d562e94-59xbh"
Oct 30 20:58:35.406609 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:35.406572    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:58:35.406754 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:35.406621    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:58:35.456510 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:35.456477    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:58:35.456710 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:35.456552    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 20:58:35.460460 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:35.460433    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d562e94-59xbh"]
Oct 30 20:58:35.482131 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:35.482105    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e7f7cf4-9ed5-40d4-aa69-2e086b1854d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d562e94-59xbh\" (UID: \"7e7f7cf4-9ed5-40d4-aa69-2e086b1854d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d562e94-59xbh"
Oct 30 20:58:35.482276 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:35.482153    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkjlt\" (UniqueName: \"kubernetes.io/projected/7e7f7cf4-9ed5-40d4-aa69-2e086b1854d1-kube-api-access-zkjlt\") pod \"maas-api-revoke-sub-3d562e94-59xbh\" (UID: \"7e7f7cf4-9ed5-40d4-aa69-2e086b1854d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d562e94-59xbh"
Oct 30 20:58:35.482276 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:35.482177    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e7f7cf4-9ed5-40d4-aa69-2e086b1854d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d562e94-59xbh\" (UID: \"7e7f7cf4-9ed5-40d4-aa69-2e086b1854d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d562e94-59xbh"
Oct 30 20:58:35.583099 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:35.583067    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e7f7cf4-9ed5-40d4-aa69-2e086b1854d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d562e94-59xbh\" (UID: \"7e7f7cf4-9ed5-40d4-aa69-2e086b1854d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d562e94-59xbh"
Oct 30 20:58:35.583276 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:35.583117    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkjlt\" (UniqueName: \"kubernetes.io/projected/7e7f7cf4-9ed5-40d4-aa69-2e086b1854d1-kube-api-access-zkjlt\") pod \"maas-api-revoke-sub-3d562e94-59xbh\" (UID: \"7e7f7cf4-9ed5-40d4-aa69-2e086b1854d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d562e94-59xbh"
Oct 30 20:58:35.583276 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:35.583145    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e7f7cf4-9ed5-40d4-aa69-2e086b1854d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d562e94-59xbh\" (UID: \"7e7f7cf4-9ed5-40d4-aa69-2e086b1854d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d562e94-59xbh"
Oct 30 20:58:35.583731 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:35.583709    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e7f7cf4-9ed5-40d4-aa69-2e086b1854d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d562e94-59xbh\" (UID: \"7e7f7cf4-9ed5-40d4-aa69-2e086b1854d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d562e94-59xbh"
Oct 30 20:58:35.585683 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:35.585663    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e7f7cf4-9ed5-40d4-aa69-2e086b1854d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d562e94-59xbh\" (UID: \"7e7f7cf4-9ed5-40d4-aa69-2e086b1854d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d562e94-59xbh"
Oct 30 20:58:35.640512 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:35.640443    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkjlt\" (UniqueName: \"kubernetes.io/projected/7e7f7cf4-9ed5-40d4-aa69-2e086b1854d1-kube-api-access-zkjlt\") pod \"maas-api-revoke-sub-3d562e94-59xbh\" (UID: \"7e7f7cf4-9ed5-40d4-aa69-2e086b1854d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d562e94-59xbh"
Oct 30 20:58:35.668586 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:35.668553    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d562e94-59xbh"
Oct 30 20:58:36.049034 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:36.049005    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d562e94-59xbh"]
Oct 30 20:58:36.049969 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:58:36.049943    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e7f7cf4_9ed5_40d4_aa69_2e086b1854d1.slice/crio-cd9636079829c768b3d6ef3c1be8bdffd322c45c0fa750a592a1a2f93a9d039f WatchSource:0}: Error finding container cd9636079829c768b3d6ef3c1be8bdffd322c45c0fa750a592a1a2f93a9d039f: Status 404 returned error can't find the container with id cd9636079829c768b3d6ef3c1be8bdffd322c45c0fa750a592a1a2f93a9d039f
Oct 30 20:58:36.058216 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:36.058192    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd0c8b7d-wcdtn"]
Oct 30 20:58:36.098894 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:36.098863    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd0c8b7d-wcdtn"]
Oct 30 20:58:36.148712 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:36.148679    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d562e94-59xbh" event={"ID":"7e7f7cf4-9ed5-40d4-aa69-2e086b1854d1","Type":"ContainerStarted","Data":"cd9636079829c768b3d6ef3c1be8bdffd322c45c0fa750a592a1a2f93a9d039f"}
Oct 30 20:58:37.086053 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:37.085998    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed2d8620-1d01-48d1-9491-3bacfd9214a3" path="/var/lib/kubelet/pods/ed2d8620-1d01-48d1-9491-3bacfd9214a3/volumes"
Oct 30 20:58:37.228503 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:37.228460    5713 generic.go:356] "Generic (PLEG): container finished" podID="7e7f7cf4-9ed5-40d4-aa69-2e086b1854d1" containerID="9bf01375c663cfbd6cd8ad936488452bcc18b49e6e426eaada02b733ca2e2c7f" exitCode=0
Oct 30 20:58:37.228700 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:37.228679    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d562e94-59xbh" event={"ID":"7e7f7cf4-9ed5-40d4-aa69-2e086b1854d1","Type":"ContainerDied","Data":"9bf01375c663cfbd6cd8ad936488452bcc18b49e6e426eaada02b733ca2e2c7f"}
Oct 30 20:58:37.280212 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:37.280183    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d4597073-8ff0-4a4e-933c-c64e934fe977/prometheus/0.log"
Oct 30 20:58:37.281033 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:37.281014    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d4597073-8ff0-4a4e-933c-c64e934fe977/prometheus/0.log"
Oct 30 20:58:37.305774 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:37.305753    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r7lpf_764461d5-e0b2-4856-a97e-0c5d3d34a497/ovn-acl-logging/0.log"
Oct 30 20:58:37.306158 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:37.306136    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r7lpf_764461d5-e0b2-4856-a97e-0c5d3d34a497/ovn-acl-logging/0.log"
Oct 30 20:58:38.421735 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:38.421705    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d562e94-59xbh"
Oct 30 20:58:38.519307 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:38.519209    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e7f7cf4-9ed5-40d4-aa69-2e086b1854d1-kube-api-access-zkjlt\" (UniqueName: \"kubernetes.io/projected/7e7f7cf4-9ed5-40d4-aa69-2e086b1854d1-kube-api-access-zkjlt\") pod \"7e7f7cf4-9ed5-40d4-aa69-2e086b1854d1\" (UID: \"7e7f7cf4-9ed5-40d4-aa69-2e086b1854d1\") "
Oct 30 20:58:38.519307 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:38.519299    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e7f7cf4-9ed5-40d4-aa69-2e086b1854d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e7f7cf4-9ed5-40d4-aa69-2e086b1854d1-maas-api-xks-ca\") pod \"7e7f7cf4-9ed5-40d4-aa69-2e086b1854d1\" (UID: \"7e7f7cf4-9ed5-40d4-aa69-2e086b1854d1\") "
Oct 30 20:58:38.519534 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:38.519431    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e7f7cf4-9ed5-40d4-aa69-2e086b1854d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e7f7cf4-9ed5-40d4-aa69-2e086b1854d1-maas-api-openshift-ca\") pod \"7e7f7cf4-9ed5-40d4-aa69-2e086b1854d1\" (UID: \"7e7f7cf4-9ed5-40d4-aa69-2e086b1854d1\") "
Oct 30 20:58:38.519782 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:38.519757    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e7f7cf4-9ed5-40d4-aa69-2e086b1854d1-maas-api-openshift-ca" pod "7e7f7cf4-9ed5-40d4-aa69-2e086b1854d1" (UID: "7e7f7cf4-9ed5-40d4-aa69-2e086b1854d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:58:38.521451 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:38.521422    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e7f7cf4-9ed5-40d4-aa69-2e086b1854d1-kube-api-access-zkjlt" pod "7e7f7cf4-9ed5-40d4-aa69-2e086b1854d1" (UID: "7e7f7cf4-9ed5-40d4-aa69-2e086b1854d1"). InnerVolumeSpecName "kube-api-access-zkjlt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:58:38.521556 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:38.521425    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e7f7cf4-9ed5-40d4-aa69-2e086b1854d1-maas-api-xks-ca" pod "7e7f7cf4-9ed5-40d4-aa69-2e086b1854d1" (UID: "7e7f7cf4-9ed5-40d4-aa69-2e086b1854d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:58:38.621096 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:38.621060    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkjlt\" (UniqueName: \"kubernetes.io/projected/7e7f7cf4-9ed5-40d4-aa69-2e086b1854d1-kube-api-access-zkjlt\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:38.621096 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:38.621092    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e7f7cf4-9ed5-40d4-aa69-2e086b1854d1-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:38.621096 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:38.621103    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e7f7cf4-9ed5-40d4-aa69-2e086b1854d1-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:39.244847 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:39.244817    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d562e94-59xbh"
Oct 30 20:58:39.245028 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:39.244817    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d562e94-59xbh" event={"ID":"7e7f7cf4-9ed5-40d4-aa69-2e086b1854d1","Type":"ContainerDied","Data":"cd9636079829c768b3d6ef3c1be8bdffd322c45c0fa750a592a1a2f93a9d039f"}
Oct 30 20:58:39.245028 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:39.244946    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd9636079829c768b3d6ef3c1be8bdffd322c45c0fa750a592a1a2f93a9d039f"
Oct 30 20:58:44.459453 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:44.459413    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-2b425d/distinct-2-w5-2b425d-kserve-6694867c68-bzpvn"]
Oct 30 20:58:44.460268 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:44.460248    5713 cpu_manager.go:414] "containerMap: removing container" podUID="7e7f7cf4-9ed5-40d4-aa69-2e086b1854d1" containerName="revoke-keys"
Oct 30 20:58:44.460368 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:44.460269    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e7f7cf4-9ed5-40d4-aa69-2e086b1854d1" containerName="revoke-keys"
Oct 30 20:58:44.460458 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:44.460444    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e7f7cf4-9ed5-40d4-aa69-2e086b1854d1" containerName="revoke-keys"
Oct 30 20:58:44.465449 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:44.464768    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2b425d/distinct-2-w5-2b425d-kserve-6694867c68-bzpvn"
Oct 30 20:58:44.521333 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:44.521299    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-2b425d/distinct-2-w5-2b425d-kserve-6694867c68-bzpvn"]
Oct 30 20:58:44.566538 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:44.566510    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-2b425d\"/\"distinct-2-w5-2b425d-kserve-self-signed-certs\""
Oct 30 20:58:44.593121 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:44.593085    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9d391e6-de6f-476d-a926-d1b333474109-tls-certs\") pod \"distinct-2-w5-2b425d-kserve-6694867c68-bzpvn\" (UID: \"b9d391e6-de6f-476d-a926-d1b333474109\") " pod="e2e-models-e2e-worker-w5-2b425d/distinct-2-w5-2b425d-kserve-6694867c68-bzpvn"
Oct 30 20:58:44.593269 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:44.593130    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9d391e6-de6f-476d-a926-d1b333474109-home\") pod \"distinct-2-w5-2b425d-kserve-6694867c68-bzpvn\" (UID: \"b9d391e6-de6f-476d-a926-d1b333474109\") " pod="e2e-models-e2e-worker-w5-2b425d/distinct-2-w5-2b425d-kserve-6694867c68-bzpvn"
Oct 30 20:58:44.593269 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:44.593256    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9d391e6-de6f-476d-a926-d1b333474109-dshm\") pod \"distinct-2-w5-2b425d-kserve-6694867c68-bzpvn\" (UID: \"b9d391e6-de6f-476d-a926-d1b333474109\") " pod="e2e-models-e2e-worker-w5-2b425d/distinct-2-w5-2b425d-kserve-6694867c68-bzpvn"
Oct 30 20:58:44.593360 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:44.593288    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9d391e6-de6f-476d-a926-d1b333474109-tmp-dir\") pod \"distinct-2-w5-2b425d-kserve-6694867c68-bzpvn\" (UID: \"b9d391e6-de6f-476d-a926-d1b333474109\") " pod="e2e-models-e2e-worker-w5-2b425d/distinct-2-w5-2b425d-kserve-6694867c68-bzpvn"
Oct 30 20:58:44.593400 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:44.593356    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpcth\" (UniqueName: \"kubernetes.io/projected/b9d391e6-de6f-476d-a926-d1b333474109-kube-api-access-bpcth\") pod \"distinct-2-w5-2b425d-kserve-6694867c68-bzpvn\" (UID: \"b9d391e6-de6f-476d-a926-d1b333474109\") " pod="e2e-models-e2e-worker-w5-2b425d/distinct-2-w5-2b425d-kserve-6694867c68-bzpvn"
Oct 30 20:58:44.593400 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:44.593395    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9d391e6-de6f-476d-a926-d1b333474109-model-cache\") pod \"distinct-2-w5-2b425d-kserve-6694867c68-bzpvn\" (UID: \"b9d391e6-de6f-476d-a926-d1b333474109\") " pod="e2e-models-e2e-worker-w5-2b425d/distinct-2-w5-2b425d-kserve-6694867c68-bzpvn"
Oct 30 20:58:44.694546 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:44.694512    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9d391e6-de6f-476d-a926-d1b333474109-dshm\") pod \"distinct-2-w5-2b425d-kserve-6694867c68-bzpvn\" (UID: \"b9d391e6-de6f-476d-a926-d1b333474109\") " pod="e2e-models-e2e-worker-w5-2b425d/distinct-2-w5-2b425d-kserve-6694867c68-bzpvn"
Oct 30 20:58:44.694756 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:44.694560    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9d391e6-de6f-476d-a926-d1b333474109-tmp-dir\") pod \"distinct-2-w5-2b425d-kserve-6694867c68-bzpvn\" (UID: \"b9d391e6-de6f-476d-a926-d1b333474109\") " pod="e2e-models-e2e-worker-w5-2b425d/distinct-2-w5-2b425d-kserve-6694867c68-bzpvn"
Oct 30 20:58:44.694756 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:44.694599    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpcth\" (UniqueName: \"kubernetes.io/projected/b9d391e6-de6f-476d-a926-d1b333474109-kube-api-access-bpcth\") pod \"distinct-2-w5-2b425d-kserve-6694867c68-bzpvn\" (UID: \"b9d391e6-de6f-476d-a926-d1b333474109\") " pod="e2e-models-e2e-worker-w5-2b425d/distinct-2-w5-2b425d-kserve-6694867c68-bzpvn"
Oct 30 20:58:44.694756 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:44.694634    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9d391e6-de6f-476d-a926-d1b333474109-model-cache\") pod \"distinct-2-w5-2b425d-kserve-6694867c68-bzpvn\" (UID: \"b9d391e6-de6f-476d-a926-d1b333474109\") " pod="e2e-models-e2e-worker-w5-2b425d/distinct-2-w5-2b425d-kserve-6694867c68-bzpvn"
Oct 30 20:58:44.694756 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:44.694686    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9d391e6-de6f-476d-a926-d1b333474109-tls-certs\") pod \"distinct-2-w5-2b425d-kserve-6694867c68-bzpvn\" (UID: \"b9d391e6-de6f-476d-a926-d1b333474109\") " pod="e2e-models-e2e-worker-w5-2b425d/distinct-2-w5-2b425d-kserve-6694867c68-bzpvn"
Oct 30 20:58:44.694756 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:44.694710    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9d391e6-de6f-476d-a926-d1b333474109-home\") pod \"distinct-2-w5-2b425d-kserve-6694867c68-bzpvn\" (UID: \"b9d391e6-de6f-476d-a926-d1b333474109\") " pod="e2e-models-e2e-worker-w5-2b425d/distinct-2-w5-2b425d-kserve-6694867c68-bzpvn"
Oct 30 20:58:44.695112 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:44.695092    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9d391e6-de6f-476d-a926-d1b333474109-tmp-dir\") pod \"distinct-2-w5-2b425d-kserve-6694867c68-bzpvn\" (UID: \"b9d391e6-de6f-476d-a926-d1b333474109\") " pod="e2e-models-e2e-worker-w5-2b425d/distinct-2-w5-2b425d-kserve-6694867c68-bzpvn"
Oct 30 20:58:44.695168 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:44.695137    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9d391e6-de6f-476d-a926-d1b333474109-home\") pod \"distinct-2-w5-2b425d-kserve-6694867c68-bzpvn\" (UID: \"b9d391e6-de6f-476d-a926-d1b333474109\") " pod="e2e-models-e2e-worker-w5-2b425d/distinct-2-w5-2b425d-kserve-6694867c68-bzpvn"
Oct 30 20:58:44.695204 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:44.695160    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9d391e6-de6f-476d-a926-d1b333474109-model-cache\") pod \"distinct-2-w5-2b425d-kserve-6694867c68-bzpvn\" (UID: \"b9d391e6-de6f-476d-a926-d1b333474109\") " pod="e2e-models-e2e-worker-w5-2b425d/distinct-2-w5-2b425d-kserve-6694867c68-bzpvn"
Oct 30 20:58:44.697022 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:44.697001    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9d391e6-de6f-476d-a926-d1b333474109-dshm\") pod \"distinct-2-w5-2b425d-kserve-6694867c68-bzpvn\" (UID: \"b9d391e6-de6f-476d-a926-d1b333474109\") " pod="e2e-models-e2e-worker-w5-2b425d/distinct-2-w5-2b425d-kserve-6694867c68-bzpvn"
Oct 30 20:58:44.697280 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:44.697262    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9d391e6-de6f-476d-a926-d1b333474109-tls-certs\") pod \"distinct-2-w5-2b425d-kserve-6694867c68-bzpvn\" (UID: \"b9d391e6-de6f-476d-a926-d1b333474109\") " pod="e2e-models-e2e-worker-w5-2b425d/distinct-2-w5-2b425d-kserve-6694867c68-bzpvn"
Oct 30 20:58:44.752727 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:44.752651    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpcth\" (UniqueName: \"kubernetes.io/projected/b9d391e6-de6f-476d-a926-d1b333474109-kube-api-access-bpcth\") pod \"distinct-2-w5-2b425d-kserve-6694867c68-bzpvn\" (UID: \"b9d391e6-de6f-476d-a926-d1b333474109\") " pod="e2e-models-e2e-worker-w5-2b425d/distinct-2-w5-2b425d-kserve-6694867c68-bzpvn"
Oct 30 20:58:44.782648 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:44.782458    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2b425d/distinct-2-w5-2b425d-kserve-6694867c68-bzpvn"
Oct 30 20:58:45.181283 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:45.181250    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-2b425d/distinct-2-w5-2b425d-kserve-6694867c68-bzpvn"]
Oct 30 20:58:45.181668 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:58:45.181647    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb9d391e6_de6f_476d_a926_d1b333474109.slice/crio-835bc0c9e5f8445d3fa402e10fd8cdf342781b161f039a6caa147a5dfc542781 WatchSource:0}: Error finding container 835bc0c9e5f8445d3fa402e10fd8cdf342781b161f039a6caa147a5dfc542781: Status 404 returned error can't find the container with id 835bc0c9e5f8445d3fa402e10fd8cdf342781b161f039a6caa147a5dfc542781
Oct 30 20:58:45.288854 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:45.288821    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2b425d/distinct-2-w5-2b425d-kserve-6694867c68-bzpvn" event={"ID":"b9d391e6-de6f-476d-a926-d1b333474109","Type":"ContainerStarted","Data":"9c10bf76a8d288d886bd48cea2c7614218fd4d9ea813f155b3155eab278a110e"}
Oct 30 20:58:45.289039 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:45.288864    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2b425d/distinct-2-w5-2b425d-kserve-6694867c68-bzpvn" event={"ID":"b9d391e6-de6f-476d-a926-d1b333474109","Type":"ContainerStarted","Data":"835bc0c9e5f8445d3fa402e10fd8cdf342781b161f039a6caa147a5dfc542781"}
Oct 30 20:58:45.410606 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:45.410546    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-2b425d/distinct-2-w5-2b425d-kserve-6694867c68-bzpvn" podStartSLOduration=1.41052702 podStartE2EDuration="1.41052702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:58:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:58:45.405339553 +0000 UTC m=+1208.470115952" watchObservedRunningTime="2026-10-30 20:58:45.41052702 +0000 UTC m=+1208.475303419"
Oct 30 20:58:45.782629 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:45.782594    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2b425d/distinct-2-w5-2b425d-kserve-6694867c68-bzpvn"
Oct 30 20:58:48.046769 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:48.046737    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-247596eb-6mdlp"]
Oct 30 20:58:48.096864 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:48.096831    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-247596eb-6mdlp"]
Oct 30 20:58:49.063419 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:49.063388    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4674926-5f7f-4869-b1bd-274bd996d3e7" path="/var/lib/kubelet/pods/b4674926-5f7f-4869-b1bd-274bd996d3e7/volumes"
Oct 30 20:58:49.549633 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:49.549595    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb31ab64-57b5l"]
Oct 30 20:58:49.553908 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:49.553891    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb31ab64-57b5l"
Oct 30 20:58:49.606189 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:49.606157    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:58:49.655942 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:49.655906    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:58:49.655942 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:49.655939    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:58:49.656180 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:49.655963    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 20:58:49.659622 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:49.659601    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb31ab64-57b5l"]
Oct 30 20:58:49.665061 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:49.665034    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b4cefda-3fd3-4394-aeac-e6594ce167a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb31ab64-57b5l\" (UID: \"7b4cefda-3fd3-4394-aeac-e6594ce167a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb31ab64-57b5l"
Oct 30 20:58:49.665180 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:49.665166    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5f4f\" (UniqueName: \"kubernetes.io/projected/7b4cefda-3fd3-4394-aeac-e6594ce167a0-kube-api-access-s5f4f\") pod \"maas-api-revoke-sub-eb31ab64-57b5l\" (UID: \"7b4cefda-3fd3-4394-aeac-e6594ce167a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb31ab64-57b5l"
Oct 30 20:58:49.665252 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:49.665237    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b4cefda-3fd3-4394-aeac-e6594ce167a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb31ab64-57b5l\" (UID: \"7b4cefda-3fd3-4394-aeac-e6594ce167a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb31ab64-57b5l"
Oct 30 20:58:49.766841 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:49.766770    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5f4f\" (UniqueName: \"kubernetes.io/projected/7b4cefda-3fd3-4394-aeac-e6594ce167a0-kube-api-access-s5f4f\") pod \"maas-api-revoke-sub-eb31ab64-57b5l\" (UID: \"7b4cefda-3fd3-4394-aeac-e6594ce167a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb31ab64-57b5l"
Oct 30 20:58:49.767052 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:49.766881    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b4cefda-3fd3-4394-aeac-e6594ce167a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb31ab64-57b5l\" (UID: \"7b4cefda-3fd3-4394-aeac-e6594ce167a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb31ab64-57b5l"
Oct 30 20:58:49.767126 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:49.767060    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b4cefda-3fd3-4394-aeac-e6594ce167a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb31ab64-57b5l\" (UID: \"7b4cefda-3fd3-4394-aeac-e6594ce167a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb31ab64-57b5l"
Oct 30 20:58:49.767681 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:49.767649    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b4cefda-3fd3-4394-aeac-e6594ce167a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb31ab64-57b5l\" (UID: \"7b4cefda-3fd3-4394-aeac-e6594ce167a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb31ab64-57b5l"
Oct 30 20:58:49.769415 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:49.769391    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b4cefda-3fd3-4394-aeac-e6594ce167a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb31ab64-57b5l\" (UID: \"7b4cefda-3fd3-4394-aeac-e6594ce167a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb31ab64-57b5l"
Oct 30 20:58:49.824738 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:49.824663    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5f4f\" (UniqueName: \"kubernetes.io/projected/7b4cefda-3fd3-4394-aeac-e6594ce167a0-kube-api-access-s5f4f\") pod \"maas-api-revoke-sub-eb31ab64-57b5l\" (UID: \"7b4cefda-3fd3-4394-aeac-e6594ce167a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb31ab64-57b5l"
Oct 30 20:58:49.868352 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:49.868311    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb31ab64-57b5l"
Oct 30 20:58:50.246424 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:50.246395    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb31ab64-57b5l"]
Oct 30 20:58:50.246878 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:58:50.246711    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b4cefda_3fd3_4394_aeac_e6594ce167a0.slice/crio-5bd56d8b0ef703b3e55a9256fb088bdbb3ace8208a13af70686a28c0e2fd5d00 WatchSource:0}: Error finding container 5bd56d8b0ef703b3e55a9256fb088bdbb3ace8208a13af70686a28c0e2fd5d00: Status 404 returned error can't find the container with id 5bd56d8b0ef703b3e55a9256fb088bdbb3ace8208a13af70686a28c0e2fd5d00
Oct 30 20:58:50.327699 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:50.327667    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb31ab64-57b5l" event={"ID":"7b4cefda-3fd3-4394-aeac-e6594ce167a0","Type":"ContainerStarted","Data":"5bd56d8b0ef703b3e55a9256fb088bdbb3ace8208a13af70686a28c0e2fd5d00"}
Oct 30 20:58:51.335995 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:51.335956    5713 generic.go:356] "Generic (PLEG): container finished" podID="7b4cefda-3fd3-4394-aeac-e6594ce167a0" containerID="6f2db1d55aa80062081ace611bd2d24c1d85946ec87078eb57615843ef9d34c8" exitCode=0
Oct 30 20:58:51.336423 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:51.336042    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb31ab64-57b5l" event={"ID":"7b4cefda-3fd3-4394-aeac-e6594ce167a0","Type":"ContainerDied","Data":"6f2db1d55aa80062081ace611bd2d24c1d85946ec87078eb57615843ef9d34c8"}
Oct 30 20:58:52.503313 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:52.503291    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb31ab64-57b5l"
Oct 30 20:58:52.601037 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:52.600956    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b4cefda-3fd3-4394-aeac-e6594ce167a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b4cefda-3fd3-4394-aeac-e6594ce167a0-maas-api-openshift-ca\") pod \"7b4cefda-3fd3-4394-aeac-e6594ce167a0\" (UID: \"7b4cefda-3fd3-4394-aeac-e6594ce167a0\") "
Oct 30 20:58:52.601183 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:52.601117    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b4cefda-3fd3-4394-aeac-e6594ce167a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b4cefda-3fd3-4394-aeac-e6594ce167a0-maas-api-xks-ca\") pod \"7b4cefda-3fd3-4394-aeac-e6594ce167a0\" (UID: \"7b4cefda-3fd3-4394-aeac-e6594ce167a0\") "
Oct 30 20:58:52.601265 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:52.601248    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b4cefda-3fd3-4394-aeac-e6594ce167a0-kube-api-access-s5f4f\" (UniqueName: \"kubernetes.io/projected/7b4cefda-3fd3-4394-aeac-e6594ce167a0-kube-api-access-s5f4f\") pod \"7b4cefda-3fd3-4394-aeac-e6594ce167a0\" (UID: \"7b4cefda-3fd3-4394-aeac-e6594ce167a0\") "
Oct 30 20:58:52.601380 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:52.601360    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b4cefda-3fd3-4394-aeac-e6594ce167a0-maas-api-openshift-ca" pod "7b4cefda-3fd3-4394-aeac-e6594ce167a0" (UID: "7b4cefda-3fd3-4394-aeac-e6594ce167a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:58:52.601695 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:52.601680    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b4cefda-3fd3-4394-aeac-e6594ce167a0-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:52.603341 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:52.603323    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b4cefda-3fd3-4394-aeac-e6594ce167a0-maas-api-xks-ca" pod "7b4cefda-3fd3-4394-aeac-e6594ce167a0" (UID: "7b4cefda-3fd3-4394-aeac-e6594ce167a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:58:52.603471 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:52.603449    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b4cefda-3fd3-4394-aeac-e6594ce167a0-kube-api-access-s5f4f" pod "7b4cefda-3fd3-4394-aeac-e6594ce167a0" (UID: "7b4cefda-3fd3-4394-aeac-e6594ce167a0"). InnerVolumeSpecName "kube-api-access-s5f4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:58:52.703165 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:52.703135    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5f4f\" (UniqueName: \"kubernetes.io/projected/7b4cefda-3fd3-4394-aeac-e6594ce167a0-kube-api-access-s5f4f\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:52.703165 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:52.703163    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b4cefda-3fd3-4394-aeac-e6594ce167a0-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:58:53.352721 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:53.352641    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb31ab64-57b5l"
Oct 30 20:58:53.352721 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:53.352683    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb31ab64-57b5l" event={"ID":"7b4cefda-3fd3-4394-aeac-e6594ce167a0","Type":"ContainerDied","Data":"5bd56d8b0ef703b3e55a9256fb088bdbb3ace8208a13af70686a28c0e2fd5d00"}
Oct 30 20:58:53.352942 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:53.352734    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5bd56d8b0ef703b3e55a9256fb088bdbb3ace8208a13af70686a28c0e2fd5d00"
Oct 30 20:58:54.459007 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:58:54.458973    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 20:58:54.783038 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:54.782944    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-2b425d/distinct-2-w5-2b425d-kserve-6694867c68-bzpvn"
Oct 30 20:58:54.795946 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:54.795915    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-2b425d/distinct-2-w5-2b425d-kserve-6694867c68-bzpvn"
Oct 30 20:58:55.378025 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:58:55.377997    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2b425d/distinct-2-w5-2b425d-kserve-6694867c68-bzpvn"
Oct 30 20:59:00.316402 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:00.316373    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ce05cc9-dtkj5"]
Oct 30 20:59:00.316943 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:00.316930    5713 cpu_manager.go:414] "containerMap: removing container" podUID="7b4cefda-3fd3-4394-aeac-e6594ce167a0" containerName="revoke-keys"
Oct 30 20:59:00.316993 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:00.316944    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b4cefda-3fd3-4394-aeac-e6594ce167a0" containerName="revoke-keys"
Oct 30 20:59:00.317067 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:00.317058    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b4cefda-3fd3-4394-aeac-e6594ce167a0" containerName="revoke-keys"
Oct 30 20:59:00.320491 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:00.320477    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce05cc9-dtkj5"
Oct 30 20:59:00.377372 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:00.377343    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ce05cc9-dtkj5"]
Oct 30 20:59:00.405544 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:00.405505    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81c89aa9-2696-4e30-aa16-a4af8c4235c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ce05cc9-dtkj5\" (UID: \"81c89aa9-2696-4e30-aa16-a4af8c4235c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce05cc9-dtkj5"
Oct 30 20:59:00.405722 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:00.405658    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swx6t\" (UniqueName: \"kubernetes.io/projected/81c89aa9-2696-4e30-aa16-a4af8c4235c5-kube-api-access-swx6t\") pod \"maas-api-revoke-sub-1ce05cc9-dtkj5\" (UID: \"81c89aa9-2696-4e30-aa16-a4af8c4235c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce05cc9-dtkj5"
Oct 30 20:59:00.405821 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:00.405733    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81c89aa9-2696-4e30-aa16-a4af8c4235c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ce05cc9-dtkj5\" (UID: \"81c89aa9-2696-4e30-aa16-a4af8c4235c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce05cc9-dtkj5"
Oct 30 20:59:00.422344 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:00.422318    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:59:00.422344 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:00.422330    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:59:00.422548 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:00.422350    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:59:00.422688 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:00.422673    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 20:59:00.507122 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:00.507089    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swx6t\" (UniqueName: \"kubernetes.io/projected/81c89aa9-2696-4e30-aa16-a4af8c4235c5-kube-api-access-swx6t\") pod \"maas-api-revoke-sub-1ce05cc9-dtkj5\" (UID: \"81c89aa9-2696-4e30-aa16-a4af8c4235c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce05cc9-dtkj5"
Oct 30 20:59:00.507282 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:00.507161    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81c89aa9-2696-4e30-aa16-a4af8c4235c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ce05cc9-dtkj5\" (UID: \"81c89aa9-2696-4e30-aa16-a4af8c4235c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce05cc9-dtkj5"
Oct 30 20:59:00.507335 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:00.507294    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81c89aa9-2696-4e30-aa16-a4af8c4235c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ce05cc9-dtkj5\" (UID: \"81c89aa9-2696-4e30-aa16-a4af8c4235c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce05cc9-dtkj5"
Oct 30 20:59:00.507871 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:00.507847    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81c89aa9-2696-4e30-aa16-a4af8c4235c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ce05cc9-dtkj5\" (UID: \"81c89aa9-2696-4e30-aa16-a4af8c4235c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce05cc9-dtkj5"
Oct 30 20:59:00.509721 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:00.509703    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81c89aa9-2696-4e30-aa16-a4af8c4235c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ce05cc9-dtkj5\" (UID: \"81c89aa9-2696-4e30-aa16-a4af8c4235c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce05cc9-dtkj5"
Oct 30 20:59:00.563876 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:00.563851    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swx6t\" (UniqueName: \"kubernetes.io/projected/81c89aa9-2696-4e30-aa16-a4af8c4235c5-kube-api-access-swx6t\") pod \"maas-api-revoke-sub-1ce05cc9-dtkj5\" (UID: \"81c89aa9-2696-4e30-aa16-a4af8c4235c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce05cc9-dtkj5"
Oct 30 20:59:00.633680 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:00.633599    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce05cc9-dtkj5"
Oct 30 20:59:01.013345 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:01.013312    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ce05cc9-dtkj5"]
Oct 30 20:59:01.014019 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:59:01.013985    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81c89aa9_2696_4e30_aa16_a4af8c4235c5.slice/crio-db234b558235543758b2af6438676c7ab52fb72016f2f436a960dc279a2b4e4b WatchSource:0}: Error finding container db234b558235543758b2af6438676c7ab52fb72016f2f436a960dc279a2b4e4b: Status 404 returned error can't find the container with id db234b558235543758b2af6438676c7ab52fb72016f2f436a960dc279a2b4e4b
Oct 30 20:59:01.420846 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:01.420803    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce05cc9-dtkj5" event={"ID":"81c89aa9-2696-4e30-aa16-a4af8c4235c5","Type":"ContainerStarted","Data":"12d2ee9d8baa72033da22dcff21970f476b48440d1e1cc69236d2471ba567fc5"}
Oct 30 20:59:01.421239 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:01.420860    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce05cc9-dtkj5" event={"ID":"81c89aa9-2696-4e30-aa16-a4af8c4235c5","Type":"ContainerStarted","Data":"db234b558235543758b2af6438676c7ab52fb72016f2f436a960dc279a2b4e4b"}
Oct 30 20:59:01.535625 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:01.535575    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce05cc9-dtkj5" podStartSLOduration=1.535560958 podStartE2EDuration="1.535560958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:59:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:59:01.534952415 +0000 UTC m=+1224.599728814" watchObservedRunningTime="2026-10-30 20:59:01.535560958 +0000 UTC m=+1224.600337356"
Oct 30 20:59:02.432647 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:02.432617    5713 generic.go:356] "Generic (PLEG): container finished" podID="81c89aa9-2696-4e30-aa16-a4af8c4235c5" containerID="12d2ee9d8baa72033da22dcff21970f476b48440d1e1cc69236d2471ba567fc5" exitCode=0
Oct 30 20:59:02.433054 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:02.432723    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce05cc9-dtkj5" event={"ID":"81c89aa9-2696-4e30-aa16-a4af8c4235c5","Type":"ContainerDied","Data":"12d2ee9d8baa72033da22dcff21970f476b48440d1e1cc69236d2471ba567fc5"}
Oct 30 20:59:03.047589 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:03.047555    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a561575-s9frn"]
Oct 30 20:59:03.051206 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:03.051185    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a561575-s9frn"]
Oct 30 20:59:03.064883 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:03.064851    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b76f110a-b354-400d-b8ff-637690977e28" path="/var/lib/kubelet/pods/b76f110a-b354-400d-b8ff-637690977e28/volumes"
Oct 30 20:59:03.585525 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:03.585498    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce05cc9-dtkj5"
Oct 30 20:59:03.644383 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:03.644356    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81c89aa9-2696-4e30-aa16-a4af8c4235c5-kube-api-access-swx6t\" (UniqueName: \"kubernetes.io/projected/81c89aa9-2696-4e30-aa16-a4af8c4235c5-kube-api-access-swx6t\") pod \"81c89aa9-2696-4e30-aa16-a4af8c4235c5\" (UID: \"81c89aa9-2696-4e30-aa16-a4af8c4235c5\") "
Oct 30 20:59:03.644552 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:03.644539    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81c89aa9-2696-4e30-aa16-a4af8c4235c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81c89aa9-2696-4e30-aa16-a4af8c4235c5-maas-api-xks-ca\") pod \"81c89aa9-2696-4e30-aa16-a4af8c4235c5\" (UID: \"81c89aa9-2696-4e30-aa16-a4af8c4235c5\") "
Oct 30 20:59:03.644615 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:03.644602    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81c89aa9-2696-4e30-aa16-a4af8c4235c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81c89aa9-2696-4e30-aa16-a4af8c4235c5-maas-api-openshift-ca\") pod \"81c89aa9-2696-4e30-aa16-a4af8c4235c5\" (UID: \"81c89aa9-2696-4e30-aa16-a4af8c4235c5\") "
Oct 30 20:59:03.644994 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:03.644969    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81c89aa9-2696-4e30-aa16-a4af8c4235c5-maas-api-openshift-ca" pod "81c89aa9-2696-4e30-aa16-a4af8c4235c5" (UID: "81c89aa9-2696-4e30-aa16-a4af8c4235c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:59:03.646641 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:03.646618    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81c89aa9-2696-4e30-aa16-a4af8c4235c5-kube-api-access-swx6t" pod "81c89aa9-2696-4e30-aa16-a4af8c4235c5" (UID: "81c89aa9-2696-4e30-aa16-a4af8c4235c5"). InnerVolumeSpecName "kube-api-access-swx6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:59:03.646869 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:03.646845    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81c89aa9-2696-4e30-aa16-a4af8c4235c5-maas-api-xks-ca" pod "81c89aa9-2696-4e30-aa16-a4af8c4235c5" (UID: "81c89aa9-2696-4e30-aa16-a4af8c4235c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:59:03.746492 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:03.746412    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81c89aa9-2696-4e30-aa16-a4af8c4235c5-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:03.746492 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:03.746441    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81c89aa9-2696-4e30-aa16-a4af8c4235c5-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:03.746492 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:03.746450    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swx6t\" (UniqueName: \"kubernetes.io/projected/81c89aa9-2696-4e30-aa16-a4af8c4235c5-kube-api-access-swx6t\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:04.449050 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:04.449016    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce05cc9-dtkj5" event={"ID":"81c89aa9-2696-4e30-aa16-a4af8c4235c5","Type":"ContainerDied","Data":"db234b558235543758b2af6438676c7ab52fb72016f2f436a960dc279a2b4e4b"}
Oct 30 20:59:04.449050 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:04.449038    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ce05cc9-dtkj5"
Oct 30 20:59:04.449245 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:04.449065    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db234b558235543758b2af6438676c7ab52fb72016f2f436a960dc279a2b4e4b"
Oct 30 20:59:07.539469 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:07.539435    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-682999/distinct-2-w3-682999-kserve-7bc687885b-gq5x7"]
Oct 30 20:59:07.540199 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:07.540179    5713 cpu_manager.go:414] "containerMap: removing container" podUID="81c89aa9-2696-4e30-aa16-a4af8c4235c5" containerName="revoke-keys"
Oct 30 20:59:07.540288 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:07.540200    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81c89aa9-2696-4e30-aa16-a4af8c4235c5" containerName="revoke-keys"
Oct 30 20:59:07.540402 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:07.540388    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="81c89aa9-2696-4e30-aa16-a4af8c4235c5" containerName="revoke-keys"
Oct 30 20:59:07.543965 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:07.543946    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-682999/distinct-2-w3-682999-kserve-7bc687885b-gq5x7"
Oct 30 20:59:07.593358 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:07.593323    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftnnk\" (UniqueName: \"kubernetes.io/projected/0dcf0422-081f-46d7-b246-7a49b074c88c-kube-api-access-ftnnk\") pod \"distinct-2-w3-682999-kserve-7bc687885b-gq5x7\" (UID: \"0dcf0422-081f-46d7-b246-7a49b074c88c\") " pod="e2e-models-e2e-worker-w3-682999/distinct-2-w3-682999-kserve-7bc687885b-gq5x7"
Oct 30 20:59:07.593625 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:07.593599    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0dcf0422-081f-46d7-b246-7a49b074c88c-home\") pod \"distinct-2-w3-682999-kserve-7bc687885b-gq5x7\" (UID: \"0dcf0422-081f-46d7-b246-7a49b074c88c\") " pod="e2e-models-e2e-worker-w3-682999/distinct-2-w3-682999-kserve-7bc687885b-gq5x7"
Oct 30 20:59:07.593763 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:07.593746    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dcf0422-081f-46d7-b246-7a49b074c88c-model-cache\") pod \"distinct-2-w3-682999-kserve-7bc687885b-gq5x7\" (UID: \"0dcf0422-081f-46d7-b246-7a49b074c88c\") " pod="e2e-models-e2e-worker-w3-682999/distinct-2-w3-682999-kserve-7bc687885b-gq5x7"
Oct 30 20:59:07.593890 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:07.593863    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dcf0422-081f-46d7-b246-7a49b074c88c-tmp-dir\") pod \"distinct-2-w3-682999-kserve-7bc687885b-gq5x7\" (UID: \"0dcf0422-081f-46d7-b246-7a49b074c88c\") " pod="e2e-models-e2e-worker-w3-682999/distinct-2-w3-682999-kserve-7bc687885b-gq5x7"
Oct 30 20:59:07.594010 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:07.593995    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dcf0422-081f-46d7-b246-7a49b074c88c-dshm\") pod \"distinct-2-w3-682999-kserve-7bc687885b-gq5x7\" (UID: \"0dcf0422-081f-46d7-b246-7a49b074c88c\") " pod="e2e-models-e2e-worker-w3-682999/distinct-2-w3-682999-kserve-7bc687885b-gq5x7"
Oct 30 20:59:07.594125 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:07.594027    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dcf0422-081f-46d7-b246-7a49b074c88c-tls-certs\") pod \"distinct-2-w3-682999-kserve-7bc687885b-gq5x7\" (UID: \"0dcf0422-081f-46d7-b246-7a49b074c88c\") " pod="e2e-models-e2e-worker-w3-682999/distinct-2-w3-682999-kserve-7bc687885b-gq5x7"
Oct 30 20:59:07.596646 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:07.596622    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-682999\"/\"distinct-2-w3-682999-kserve-self-signed-certs\""
Oct 30 20:59:07.639499 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:07.639468    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-682999/distinct-2-w3-682999-kserve-7bc687885b-gq5x7"]
Oct 30 20:59:07.695415 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:07.695376    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0dcf0422-081f-46d7-b246-7a49b074c88c-home\") pod \"distinct-2-w3-682999-kserve-7bc687885b-gq5x7\" (UID: \"0dcf0422-081f-46d7-b246-7a49b074c88c\") " pod="e2e-models-e2e-worker-w3-682999/distinct-2-w3-682999-kserve-7bc687885b-gq5x7"
Oct 30 20:59:07.695589 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:07.695459    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dcf0422-081f-46d7-b246-7a49b074c88c-model-cache\") pod \"distinct-2-w3-682999-kserve-7bc687885b-gq5x7\" (UID: \"0dcf0422-081f-46d7-b246-7a49b074c88c\") " pod="e2e-models-e2e-worker-w3-682999/distinct-2-w3-682999-kserve-7bc687885b-gq5x7"
Oct 30 20:59:07.695589 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:07.695569    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dcf0422-081f-46d7-b246-7a49b074c88c-tmp-dir\") pod \"distinct-2-w3-682999-kserve-7bc687885b-gq5x7\" (UID: \"0dcf0422-081f-46d7-b246-7a49b074c88c\") " pod="e2e-models-e2e-worker-w3-682999/distinct-2-w3-682999-kserve-7bc687885b-gq5x7"
Oct 30 20:59:07.695697 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:07.695681    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dcf0422-081f-46d7-b246-7a49b074c88c-dshm\") pod \"distinct-2-w3-682999-kserve-7bc687885b-gq5x7\" (UID: \"0dcf0422-081f-46d7-b246-7a49b074c88c\") " pod="e2e-models-e2e-worker-w3-682999/distinct-2-w3-682999-kserve-7bc687885b-gq5x7"
Oct 30 20:59:07.695732 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:07.695709    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dcf0422-081f-46d7-b246-7a49b074c88c-tls-certs\") pod \"distinct-2-w3-682999-kserve-7bc687885b-gq5x7\" (UID: \"0dcf0422-081f-46d7-b246-7a49b074c88c\") " pod="e2e-models-e2e-worker-w3-682999/distinct-2-w3-682999-kserve-7bc687885b-gq5x7"
Oct 30 20:59:07.695783 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:07.695741    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftnnk\" (UniqueName: \"kubernetes.io/projected/0dcf0422-081f-46d7-b246-7a49b074c88c-kube-api-access-ftnnk\") pod \"distinct-2-w3-682999-kserve-7bc687885b-gq5x7\" (UID: \"0dcf0422-081f-46d7-b246-7a49b074c88c\") " pod="e2e-models-e2e-worker-w3-682999/distinct-2-w3-682999-kserve-7bc687885b-gq5x7"
Oct 30 20:59:07.695783 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:07.695754    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0dcf0422-081f-46d7-b246-7a49b074c88c-home\") pod \"distinct-2-w3-682999-kserve-7bc687885b-gq5x7\" (UID: \"0dcf0422-081f-46d7-b246-7a49b074c88c\") " pod="e2e-models-e2e-worker-w3-682999/distinct-2-w3-682999-kserve-7bc687885b-gq5x7"
Oct 30 20:59:07.695900 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:07.695860    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dcf0422-081f-46d7-b246-7a49b074c88c-model-cache\") pod \"distinct-2-w3-682999-kserve-7bc687885b-gq5x7\" (UID: \"0dcf0422-081f-46d7-b246-7a49b074c88c\") " pod="e2e-models-e2e-worker-w3-682999/distinct-2-w3-682999-kserve-7bc687885b-gq5x7"
Oct 30 20:59:07.695938 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:07.695900    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dcf0422-081f-46d7-b246-7a49b074c88c-tmp-dir\") pod \"distinct-2-w3-682999-kserve-7bc687885b-gq5x7\" (UID: \"0dcf0422-081f-46d7-b246-7a49b074c88c\") " pod="e2e-models-e2e-worker-w3-682999/distinct-2-w3-682999-kserve-7bc687885b-gq5x7"
Oct 30 20:59:07.698247 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:07.698222    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dcf0422-081f-46d7-b246-7a49b074c88c-dshm\") pod \"distinct-2-w3-682999-kserve-7bc687885b-gq5x7\" (UID: \"0dcf0422-081f-46d7-b246-7a49b074c88c\") " pod="e2e-models-e2e-worker-w3-682999/distinct-2-w3-682999-kserve-7bc687885b-gq5x7"
Oct 30 20:59:07.698424 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:07.698405    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dcf0422-081f-46d7-b246-7a49b074c88c-tls-certs\") pod \"distinct-2-w3-682999-kserve-7bc687885b-gq5x7\" (UID: \"0dcf0422-081f-46d7-b246-7a49b074c88c\") " pod="e2e-models-e2e-worker-w3-682999/distinct-2-w3-682999-kserve-7bc687885b-gq5x7"
Oct 30 20:59:07.752696 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:07.752662    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftnnk\" (UniqueName: \"kubernetes.io/projected/0dcf0422-081f-46d7-b246-7a49b074c88c-kube-api-access-ftnnk\") pod \"distinct-2-w3-682999-kserve-7bc687885b-gq5x7\" (UID: \"0dcf0422-081f-46d7-b246-7a49b074c88c\") " pod="e2e-models-e2e-worker-w3-682999/distinct-2-w3-682999-kserve-7bc687885b-gq5x7"
Oct 30 20:59:07.856080 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:07.855995    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-682999/distinct-2-w3-682999-kserve-7bc687885b-gq5x7"
Oct 30 20:59:08.249837 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:08.249814    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-682999/distinct-2-w3-682999-kserve-7bc687885b-gq5x7"]
Oct 30 20:59:08.251341 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:59:08.251311    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0dcf0422_081f_46d7_b246_7a49b074c88c.slice/crio-c57fed04416968d812c9959d8194394e2c503825dc2f3b716c4713c8ada39581 WatchSource:0}: Error finding container c57fed04416968d812c9959d8194394e2c503825dc2f3b716c4713c8ada39581: Status 404 returned error can't find the container with id c57fed04416968d812c9959d8194394e2c503825dc2f3b716c4713c8ada39581
Oct 30 20:59:08.478641 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:08.478606    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-682999/distinct-2-w3-682999-kserve-7bc687885b-gq5x7" event={"ID":"0dcf0422-081f-46d7-b246-7a49b074c88c","Type":"ContainerStarted","Data":"d17cd1119ea5e3005d37fe27c968e8d570ddcd27e133b97288548a2b01fd3b3c"}
Oct 30 20:59:08.478641 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:08.478652    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-682999/distinct-2-w3-682999-kserve-7bc687885b-gq5x7" event={"ID":"0dcf0422-081f-46d7-b246-7a49b074c88c","Type":"ContainerStarted","Data":"c57fed04416968d812c9959d8194394e2c503825dc2f3b716c4713c8ada39581"}
Oct 30 20:59:08.594376 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:08.594272    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-682999/distinct-2-w3-682999-kserve-7bc687885b-gq5x7" podStartSLOduration=1.594257584 podStartE2EDuration="1.594257584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:59:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:59:08.592967943 +0000 UTC m=+1231.657744341" watchObservedRunningTime="2026-10-30 20:59:08.594257584 +0000 UTC m=+1231.659033982"
Oct 30 20:59:08.856980 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:08.856904    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-682999/distinct-2-w3-682999-kserve-7bc687885b-gq5x7"
Oct 30 20:59:14.050483 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:14.050444    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95279ef2-6mc22"]
Oct 30 20:59:14.099955 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:14.099924    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95279ef2-6mc22"]
Oct 30 20:59:15.063643 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:15.063605    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9dae508-b784-4f7d-b6a1-7146e62d51b2" path="/var/lib/kubelet/pods/f9dae508-b784-4f7d-b6a1-7146e62d51b2/volumes"
Oct 30 20:59:16.873552 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:16.873520    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8"]
Oct 30 20:59:16.878161 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:16.878141    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8"
Oct 30 20:59:16.987676 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:16.987635    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8"]
Oct 30 20:59:17.013133 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:17.013094    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03ef9f7c-175a-4012-a021-ec011e37928e-istiod-ca-cert\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8\" (UID: \"03ef9f7c-175a-4012-a021-ec011e37928e\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8"
Oct 30 20:59:17.013342 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:17.013154    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9rtk\" (UniqueName: \"kubernetes.io/projected/03ef9f7c-175a-4012-a021-ec011e37928e-kube-api-access-j9rtk\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8\" (UID: \"03ef9f7c-175a-4012-a021-ec011e37928e\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8"
Oct 30 20:59:17.013342 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:17.013235    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03ef9f7c-175a-4012-a021-ec011e37928e-istio-podinfo\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8\" (UID: \"03ef9f7c-175a-4012-a021-ec011e37928e\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8"
Oct 30 20:59:17.013462 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:17.013393    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ef9f7c-175a-4012-a021-ec011e37928e-workload-socket\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8\" (UID: \"03ef9f7c-175a-4012-a021-ec011e37928e\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8"
Oct 30 20:59:17.013462 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:17.013431    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ef9f7c-175a-4012-a021-ec011e37928e-credential-socket\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8\" (UID: \"03ef9f7c-175a-4012-a021-ec011e37928e\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8"
Oct 30 20:59:17.013573 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:17.013528    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03ef9f7c-175a-4012-a021-ec011e37928e-istio-token\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8\" (UID: \"03ef9f7c-175a-4012-a021-ec011e37928e\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8"
Oct 30 20:59:17.013672 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:17.013653    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03ef9f7c-175a-4012-a021-ec011e37928e-istio-envoy\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8\" (UID: \"03ef9f7c-175a-4012-a021-ec011e37928e\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8"
Oct 30 20:59:17.013716 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:17.013701    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03ef9f7c-175a-4012-a021-ec011e37928e-istio-data\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8\" (UID: \"03ef9f7c-175a-4012-a021-ec011e37928e\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8"
Oct 30 20:59:17.013806 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:17.013772    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03ef9f7c-175a-4012-a021-ec011e37928e-workload-certs\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8\" (UID: \"03ef9f7c-175a-4012-a021-ec011e37928e\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8"
Oct 30 20:59:17.114612 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:17.114579    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03ef9f7c-175a-4012-a021-ec011e37928e-istio-envoy\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8\" (UID: \"03ef9f7c-175a-4012-a021-ec011e37928e\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8"
Oct 30 20:59:17.114840 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:17.114635    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03ef9f7c-175a-4012-a021-ec011e37928e-istio-data\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8\" (UID: \"03ef9f7c-175a-4012-a021-ec011e37928e\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8"
Oct 30 20:59:17.114840 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:17.114658    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03ef9f7c-175a-4012-a021-ec011e37928e-workload-certs\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8\" (UID: \"03ef9f7c-175a-4012-a021-ec011e37928e\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8"
Oct 30 20:59:17.114840 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:17.114715    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03ef9f7c-175a-4012-a021-ec011e37928e-istiod-ca-cert\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8\" (UID: \"03ef9f7c-175a-4012-a021-ec011e37928e\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8"
Oct 30 20:59:17.114840 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:17.114738    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9rtk\" (UniqueName: \"kubernetes.io/projected/03ef9f7c-175a-4012-a021-ec011e37928e-kube-api-access-j9rtk\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8\" (UID: \"03ef9f7c-175a-4012-a021-ec011e37928e\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8"
Oct 30 20:59:17.114840 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:17.114761    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03ef9f7c-175a-4012-a021-ec011e37928e-istio-podinfo\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8\" (UID: \"03ef9f7c-175a-4012-a021-ec011e37928e\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8"
Oct 30 20:59:17.115127 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:17.114868    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ef9f7c-175a-4012-a021-ec011e37928e-workload-socket\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8\" (UID: \"03ef9f7c-175a-4012-a021-ec011e37928e\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8"
Oct 30 20:59:17.115127 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:17.114912    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ef9f7c-175a-4012-a021-ec011e37928e-credential-socket\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8\" (UID: \"03ef9f7c-175a-4012-a021-ec011e37928e\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8"
Oct 30 20:59:17.115127 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:17.114945    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03ef9f7c-175a-4012-a021-ec011e37928e-istio-token\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8\" (UID: \"03ef9f7c-175a-4012-a021-ec011e37928e\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8"
Oct 30 20:59:17.115288 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:17.115189    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03ef9f7c-175a-4012-a021-ec011e37928e-istio-data\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8\" (UID: \"03ef9f7c-175a-4012-a021-ec011e37928e\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8"
Oct 30 20:59:17.115288 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:17.115250    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ef9f7c-175a-4012-a021-ec011e37928e-workload-socket\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8\" (UID: \"03ef9f7c-175a-4012-a021-ec011e37928e\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8"
Oct 30 20:59:17.115456 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:17.115435    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03ef9f7c-175a-4012-a021-ec011e37928e-workload-certs\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8\" (UID: \"03ef9f7c-175a-4012-a021-ec011e37928e\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8"
Oct 30 20:59:17.115541 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:17.115442    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ef9f7c-175a-4012-a021-ec011e37928e-credential-socket\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8\" (UID: \"03ef9f7c-175a-4012-a021-ec011e37928e\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8"
Oct 30 20:59:17.115775 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:17.115751    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03ef9f7c-175a-4012-a021-ec011e37928e-istiod-ca-cert\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8\" (UID: \"03ef9f7c-175a-4012-a021-ec011e37928e\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8"
Oct 30 20:59:17.117089 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:17.117068    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03ef9f7c-175a-4012-a021-ec011e37928e-istio-envoy\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8\" (UID: \"03ef9f7c-175a-4012-a021-ec011e37928e\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8"
Oct 30 20:59:17.117610 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:17.117592    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03ef9f7c-175a-4012-a021-ec011e37928e-istio-podinfo\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8\" (UID: \"03ef9f7c-175a-4012-a021-ec011e37928e\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8"
Oct 30 20:59:17.174658 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:17.174566    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9rtk\" (UniqueName: \"kubernetes.io/projected/03ef9f7c-175a-4012-a021-ec011e37928e-kube-api-access-j9rtk\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8\" (UID: \"03ef9f7c-175a-4012-a021-ec011e37928e\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8"
Oct 30 20:59:17.174964 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:17.174942    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03ef9f7c-175a-4012-a021-ec011e37928e-istio-token\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8\" (UID: \"03ef9f7c-175a-4012-a021-ec011e37928e\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8"
Oct 30 20:59:17.192804 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:17.192766    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8"
Oct 30 20:59:17.634163 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:17.634128    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8"]
Oct 30 20:59:17.637025 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:17.636976    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 20:59:17.637155 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:17.637120    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 20:59:17.637227 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:17.637208    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 20:59:17.746672 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:17.746637    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk"]
Oct 30 20:59:17.750889 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:17.750866    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk"
Oct 30 20:59:17.752239 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:17.752211    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2"]
Oct 30 20:59:17.757053 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:17.757026    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2"
Oct 30 20:59:17.816489 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:17.816454    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk"]
Oct 30 20:59:17.838295 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:17.838264    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2"]
Oct 30 20:59:17.856267 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:17.856237    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-682999/distinct-2-w3-682999-kserve-7bc687885b-gq5x7"
Oct 30 20:59:17.871589 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:17.871559    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-682999/distinct-2-w3-682999-kserve-7bc687885b-gq5x7"
Oct 30 20:59:17.928531 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:17.928444    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a73cfa4-d614-4c39-b324-731fca0564c4-workload-socket\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk\" (UID: \"9a73cfa4-d614-4c39-b324-731fca0564c4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk"
Oct 30 20:59:17.928531 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:17.928494    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1255405-8724-4977-854d-11f13d8a847a-istio-data\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2\" (UID: \"e1255405-8724-4977-854d-11f13d8a847a\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2"
Oct 30 20:59:17.928531 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:17.928517    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a73cfa4-d614-4c39-b324-731fca0564c4-istio-token\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk\" (UID: \"9a73cfa4-d614-4c39-b324-731fca0564c4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk"
Oct 30 20:59:17.929111 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:17.928646    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1255405-8724-4977-854d-11f13d8a847a-istio-podinfo\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2\" (UID: \"e1255405-8724-4977-854d-11f13d8a847a\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2"
Oct 30 20:59:17.929111 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:17.928750    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1255405-8724-4977-854d-11f13d8a847a-istiod-ca-cert\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2\" (UID: \"e1255405-8724-4977-854d-11f13d8a847a\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2"
Oct 30 20:59:17.929111 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:17.928819    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1255405-8724-4977-854d-11f13d8a847a-workload-socket\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2\" (UID: \"e1255405-8724-4977-854d-11f13d8a847a\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2"
Oct 30 20:59:17.929111 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:17.928859    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf99v\" (UniqueName: \"kubernetes.io/projected/e1255405-8724-4977-854d-11f13d8a847a-kube-api-access-qf99v\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2\" (UID: \"e1255405-8724-4977-854d-11f13d8a847a\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2"
Oct 30 20:59:17.929111 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:17.928935    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a73cfa4-d614-4c39-b324-731fca0564c4-credential-socket\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk\" (UID: \"9a73cfa4-d614-4c39-b324-731fca0564c4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk"
Oct 30 20:59:17.929111 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:17.929010    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a73cfa4-d614-4c39-b324-731fca0564c4-istio-podinfo\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk\" (UID: \"9a73cfa4-d614-4c39-b324-731fca0564c4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk"
Oct 30 20:59:17.929111 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:17.929044    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a73cfa4-d614-4c39-b324-731fca0564c4-istio-envoy\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk\" (UID: \"9a73cfa4-d614-4c39-b324-731fca0564c4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk"
Oct 30 20:59:17.929111 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:17.929080    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1255405-8724-4977-854d-11f13d8a847a-istio-token\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2\" (UID: \"e1255405-8724-4977-854d-11f13d8a847a\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2"
Oct 30 20:59:17.929530 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:17.929138    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1255405-8724-4977-854d-11f13d8a847a-istio-envoy\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2\" (UID: \"e1255405-8724-4977-854d-11f13d8a847a\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2"
Oct 30 20:59:17.929530 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:17.929246    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6snnr\" (UniqueName: \"kubernetes.io/projected/9a73cfa4-d614-4c39-b324-731fca0564c4-kube-api-access-6snnr\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk\" (UID: \"9a73cfa4-d614-4c39-b324-731fca0564c4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk"
Oct 30 20:59:17.929530 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:17.929288    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a73cfa4-d614-4c39-b324-731fca0564c4-workload-certs\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk\" (UID: \"9a73cfa4-d614-4c39-b324-731fca0564c4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk"
Oct 30 20:59:17.929530 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:17.929311    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a73cfa4-d614-4c39-b324-731fca0564c4-istio-data\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk\" (UID: \"9a73cfa4-d614-4c39-b324-731fca0564c4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk"
Oct 30 20:59:17.929530 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:17.929446    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1255405-8724-4977-854d-11f13d8a847a-credential-socket\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2\" (UID: \"e1255405-8724-4977-854d-11f13d8a847a\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2"
Oct 30 20:59:17.929530 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:17.929479    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1255405-8724-4977-854d-11f13d8a847a-workload-certs\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2\" (UID: \"e1255405-8724-4977-854d-11f13d8a847a\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2"
Oct 30 20:59:17.929530 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:17.929519    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a73cfa4-d614-4c39-b324-731fca0564c4-istiod-ca-cert\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk\" (UID: \"9a73cfa4-d614-4c39-b324-731fca0564c4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk"
Oct 30 20:59:18.031342 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:18.031310    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1255405-8724-4977-854d-11f13d8a847a-istio-token\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2\" (UID: \"e1255405-8724-4977-854d-11f13d8a847a\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2"
Oct 30 20:59:18.031523 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:18.031384    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1255405-8724-4977-854d-11f13d8a847a-istio-envoy\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2\" (UID: \"e1255405-8724-4977-854d-11f13d8a847a\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2"
Oct 30 20:59:18.031523 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:18.031447    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6snnr\" (UniqueName: \"kubernetes.io/projected/9a73cfa4-d614-4c39-b324-731fca0564c4-kube-api-access-6snnr\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk\" (UID: \"9a73cfa4-d614-4c39-b324-731fca0564c4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk"
Oct 30 20:59:18.031523 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:18.031486    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a73cfa4-d614-4c39-b324-731fca0564c4-workload-certs\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk\" (UID: \"9a73cfa4-d614-4c39-b324-731fca0564c4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk"
Oct 30 20:59:18.031523 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:18.031510    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a73cfa4-d614-4c39-b324-731fca0564c4-istio-data\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk\" (UID: \"9a73cfa4-d614-4c39-b324-731fca0564c4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk"
Oct 30 20:59:18.031748 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:18.031601    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1255405-8724-4977-854d-11f13d8a847a-credential-socket\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2\" (UID: \"e1255405-8724-4977-854d-11f13d8a847a\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2"
Oct 30 20:59:18.031748 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:18.031626    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1255405-8724-4977-854d-11f13d8a847a-workload-certs\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2\" (UID: \"e1255405-8724-4977-854d-11f13d8a847a\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2"
Oct 30 20:59:18.031748 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:18.031667    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a73cfa4-d614-4c39-b324-731fca0564c4-istiod-ca-cert\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk\" (UID: \"9a73cfa4-d614-4c39-b324-731fca0564c4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk"
Oct 30 20:59:18.031748 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:18.031725    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a73cfa4-d614-4c39-b324-731fca0564c4-workload-socket\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk\" (UID: \"9a73cfa4-d614-4c39-b324-731fca0564c4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk"
Oct 30 20:59:18.031988 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:18.031760    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1255405-8724-4977-854d-11f13d8a847a-istio-data\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2\" (UID: \"e1255405-8724-4977-854d-11f13d8a847a\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2"
Oct 30 20:59:18.031988 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:18.031813    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a73cfa4-d614-4c39-b324-731fca0564c4-istio-token\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk\" (UID: \"9a73cfa4-d614-4c39-b324-731fca0564c4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk"
Oct 30 20:59:18.031988 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:18.031881    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1255405-8724-4977-854d-11f13d8a847a-istio-podinfo\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2\" (UID: \"e1255405-8724-4977-854d-11f13d8a847a\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2"
Oct 30 20:59:18.031988 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:18.031918    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1255405-8724-4977-854d-11f13d8a847a-istiod-ca-cert\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2\" (UID: \"e1255405-8724-4977-854d-11f13d8a847a\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2"
Oct 30 20:59:18.031988 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:18.031948    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1255405-8724-4977-854d-11f13d8a847a-workload-socket\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2\" (UID: \"e1255405-8724-4977-854d-11f13d8a847a\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2"
Oct 30 20:59:18.031988 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:18.031952    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a73cfa4-d614-4c39-b324-731fca0564c4-workload-certs\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk\" (UID: \"9a73cfa4-d614-4c39-b324-731fca0564c4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk"
Oct 30 20:59:18.031988 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:18.031978    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf99v\" (UniqueName: \"kubernetes.io/projected/e1255405-8724-4977-854d-11f13d8a847a-kube-api-access-qf99v\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2\" (UID: \"e1255405-8724-4977-854d-11f13d8a847a\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2"
Oct 30 20:59:18.032360 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:18.032051    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a73cfa4-d614-4c39-b324-731fca0564c4-credential-socket\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk\" (UID: \"9a73cfa4-d614-4c39-b324-731fca0564c4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk"
Oct 30 20:59:18.032360 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:18.032088    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a73cfa4-d614-4c39-b324-731fca0564c4-istio-podinfo\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk\" (UID: \"9a73cfa4-d614-4c39-b324-731fca0564c4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk"
Oct 30 20:59:18.032360 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:18.032122    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a73cfa4-d614-4c39-b324-731fca0564c4-istio-envoy\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk\" (UID: \"9a73cfa4-d614-4c39-b324-731fca0564c4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk"
Oct 30 20:59:18.032360 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:18.032155    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a73cfa4-d614-4c39-b324-731fca0564c4-istio-data\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk\" (UID: \"9a73cfa4-d614-4c39-b324-731fca0564c4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk"
Oct 30 20:59:18.032360 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:18.032258    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1255405-8724-4977-854d-11f13d8a847a-credential-socket\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2\" (UID: \"e1255405-8724-4977-854d-11f13d8a847a\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2"
Oct 30 20:59:18.032775 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:18.032485    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a73cfa4-d614-4c39-b324-731fca0564c4-credential-socket\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk\" (UID: \"9a73cfa4-d614-4c39-b324-731fca0564c4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk"
Oct 30 20:59:18.032775 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:18.032757    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a73cfa4-d614-4c39-b324-731fca0564c4-istiod-ca-cert\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk\" (UID: \"9a73cfa4-d614-4c39-b324-731fca0564c4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk"
Oct 30 20:59:18.033195 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:18.033107    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1255405-8724-4977-854d-11f13d8a847a-istio-data\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2\" (UID: \"e1255405-8724-4977-854d-11f13d8a847a\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2"
Oct 30 20:59:18.033195 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:18.033033    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a73cfa4-d614-4c39-b324-731fca0564c4-workload-socket\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk\" (UID: \"9a73cfa4-d614-4c39-b324-731fca0564c4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk"
Oct 30 20:59:18.033438 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:18.033346    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1255405-8724-4977-854d-11f13d8a847a-workload-socket\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2\" (UID: \"e1255405-8724-4977-854d-11f13d8a847a\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2"
Oct 30 20:59:18.033851 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:18.033828    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1255405-8724-4977-854d-11f13d8a847a-istiod-ca-cert\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2\" (UID: \"e1255405-8724-4977-854d-11f13d8a847a\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2"
Oct 30 20:59:18.033962 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:18.033943    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1255405-8724-4977-854d-11f13d8a847a-workload-certs\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2\" (UID: \"e1255405-8724-4977-854d-11f13d8a847a\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2"
Oct 30 20:59:18.034620 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:18.034600    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1255405-8724-4977-854d-11f13d8a847a-istio-envoy\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2\" (UID: \"e1255405-8724-4977-854d-11f13d8a847a\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2"
Oct 30 20:59:18.035336 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:18.035317    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a73cfa4-d614-4c39-b324-731fca0564c4-istio-podinfo\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk\" (UID: \"9a73cfa4-d614-4c39-b324-731fca0564c4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk"
Oct 30 20:59:18.035587 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:18.035571    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a73cfa4-d614-4c39-b324-731fca0564c4-istio-envoy\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk\" (UID: \"9a73cfa4-d614-4c39-b324-731fca0564c4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk"
Oct 30 20:59:18.035769 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:18.035749    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1255405-8724-4977-854d-11f13d8a847a-istio-podinfo\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2\" (UID: \"e1255405-8724-4977-854d-11f13d8a847a\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2"
Oct 30 20:59:18.089447 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:18.089413    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf99v\" (UniqueName: \"kubernetes.io/projected/e1255405-8724-4977-854d-11f13d8a847a-kube-api-access-qf99v\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2\" (UID: \"e1255405-8724-4977-854d-11f13d8a847a\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2"
Oct 30 20:59:18.089771 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:18.089751    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1255405-8724-4977-854d-11f13d8a847a-istio-token\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2\" (UID: \"e1255405-8724-4977-854d-11f13d8a847a\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2"
Oct 30 20:59:18.089989 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:18.089966    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a73cfa4-d614-4c39-b324-731fca0564c4-istio-token\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk\" (UID: \"9a73cfa4-d614-4c39-b324-731fca0564c4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk"
Oct 30 20:59:18.090520 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:18.090492    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6snnr\" (UniqueName: \"kubernetes.io/projected/9a73cfa4-d614-4c39-b324-731fca0564c4-kube-api-access-6snnr\") pod \"e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk\" (UID: \"9a73cfa4-d614-4c39-b324-731fca0564c4\") " pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk"
Oct 30 20:59:18.366920 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:18.366884    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk"
Oct 30 20:59:18.378273 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:18.378248    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2"
Oct 30 20:59:18.579106 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:18.579078    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8" event={"ID":"03ef9f7c-175a-4012-a021-ec011e37928e","Type":"ContainerStarted","Data":"669acbc53462750b4744dab1559955a86540a29cb8282e3f2f3b70ec1f5b2798"}
Oct 30 20:59:18.579264 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:18.579123    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8" event={"ID":"03ef9f7c-175a-4012-a021-ec011e37928e","Type":"ContainerStarted","Data":"202d8bfcd23e19f1db504b4333fb6ec8dc586082820ae5f4f7d6a006e9d8fc25"}
Oct 30 20:59:18.592997 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:18.592969    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-682999/distinct-2-w3-682999-kserve-7bc687885b-gq5x7"
Oct 30 20:59:18.631958 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:18.631884    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e61ebc76-d7p4p"]
Oct 30 20:59:18.658810 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:18.656890    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e61ebc76-d7p4p"
Oct 30 20:59:18.691190 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:18.691157    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e61ebc76-d7p4p"]
Oct 30 20:59:18.710160 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:18.710123    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:59:18.710352 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:18.710191    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 20:59:18.710352 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:18.710320    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:59:18.756036 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:18.755184    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8" podStartSLOduration=2.7551639530000003 podStartE2EDuration="2.755163953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:59:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:59:18.752605532 +0000 UTC m=+1241.817381931" watchObservedRunningTime="2026-10-30 20:59:18.755163953 +0000 UTC m=+1241.819940352"
Oct 30 20:59:18.767137 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:18.767105    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdc5287e-e62c-41b0-9ffe-c9f9d27f0cdd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e61ebc76-d7p4p\" (UID: \"fdc5287e-e62c-41b0-9ffe-c9f9d27f0cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e61ebc76-d7p4p"
Oct 30 20:59:18.767334 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:18.767275    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdc5287e-e62c-41b0-9ffe-c9f9d27f0cdd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e61ebc76-d7p4p\" (UID: \"fdc5287e-e62c-41b0-9ffe-c9f9d27f0cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e61ebc76-d7p4p"
Oct 30 20:59:18.767386 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:18.767344    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlpfm\" (UniqueName: \"kubernetes.io/projected/fdc5287e-e62c-41b0-9ffe-c9f9d27f0cdd-kube-api-access-tlpfm\") pod \"maas-api-revoke-sub-e61ebc76-d7p4p\" (UID: \"fdc5287e-e62c-41b0-9ffe-c9f9d27f0cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e61ebc76-d7p4p"
Oct 30 20:59:18.810382 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:18.810349    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:59:18.827067 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:18.827042    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk"]
Oct 30 20:59:18.827381 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:59:18.827357    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a73cfa4_d614_4c39_b324_731fca0564c4.slice/crio-7a1036f5812038a461d162457f0be1a35c1fa7d5219ea6130ab8395109b3ab1c WatchSource:0}: Error finding container 7a1036f5812038a461d162457f0be1a35c1fa7d5219ea6130ab8395109b3ab1c: Status 404 returned error can't find the container with id 7a1036f5812038a461d162457f0be1a35c1fa7d5219ea6130ab8395109b3ab1c
Oct 30 20:59:18.830052 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:18.830023    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 20:59:18.830142 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:18.830082    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 20:59:18.830142 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:18.830114    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 20:59:18.868584 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:18.868552    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdc5287e-e62c-41b0-9ffe-c9f9d27f0cdd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e61ebc76-d7p4p\" (UID: \"fdc5287e-e62c-41b0-9ffe-c9f9d27f0cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e61ebc76-d7p4p"
Oct 30 20:59:18.868720 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:18.868701    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdc5287e-e62c-41b0-9ffe-c9f9d27f0cdd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e61ebc76-d7p4p\" (UID: \"fdc5287e-e62c-41b0-9ffe-c9f9d27f0cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e61ebc76-d7p4p"
Oct 30 20:59:18.868840 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:18.868824    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlpfm\" (UniqueName: \"kubernetes.io/projected/fdc5287e-e62c-41b0-9ffe-c9f9d27f0cdd-kube-api-access-tlpfm\") pod \"maas-api-revoke-sub-e61ebc76-d7p4p\" (UID: \"fdc5287e-e62c-41b0-9ffe-c9f9d27f0cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e61ebc76-d7p4p"
Oct 30 20:59:18.869384 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:18.869360    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2"]
Oct 30 20:59:18.869685 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:18.869661    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdc5287e-e62c-41b0-9ffe-c9f9d27f0cdd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e61ebc76-d7p4p\" (UID: \"fdc5287e-e62c-41b0-9ffe-c9f9d27f0cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e61ebc76-d7p4p"
Oct 30 20:59:18.870761 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:59:18.870690    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1255405_8724_4977_854d_11f13d8a847a.slice/crio-71765f9523506cf9462cf503da06c5129da2a0b5a13074bd125a1b0e32d1c723 WatchSource:0}: Error finding container 71765f9523506cf9462cf503da06c5129da2a0b5a13074bd125a1b0e32d1c723: Status 404 returned error can't find the container with id 71765f9523506cf9462cf503da06c5129da2a0b5a13074bd125a1b0e32d1c723
Oct 30 20:59:18.872366 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:18.872345    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdc5287e-e62c-41b0-9ffe-c9f9d27f0cdd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e61ebc76-d7p4p\" (UID: \"fdc5287e-e62c-41b0-9ffe-c9f9d27f0cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e61ebc76-d7p4p"
Oct 30 20:59:18.873192 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:18.873160    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 20:59:18.873481 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:18.873460    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 20:59:18.874222 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:18.874197    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 20:59:18.926337 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:18.926311    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlpfm\" (UniqueName: \"kubernetes.io/projected/fdc5287e-e62c-41b0-9ffe-c9f9d27f0cdd-kube-api-access-tlpfm\") pod \"maas-api-revoke-sub-e61ebc76-d7p4p\" (UID: \"fdc5287e-e62c-41b0-9ffe-c9f9d27f0cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e61ebc76-d7p4p"
Oct 30 20:59:18.975033 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:18.974999    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e61ebc76-d7p4p"
Oct 30 20:59:19.193781 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:19.193743    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8"
Oct 30 20:59:19.195331 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:19.195287    5713 patch_prober.go:29] interesting pod/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 30 20:59:19.195472 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:19.195365    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8" podUID="03ef9f7c-175a-4012-a021-ec011e37928e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 30 20:59:19.420696 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:19.420673    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e61ebc76-d7p4p"]
Oct 30 20:59:19.423823 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:59:19.423772    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdc5287e_e62c_41b0_9ffe_c9f9d27f0cdd.slice/crio-b1423ecde4f96bad4527f3f48759fb59e6f041289e3d7aa9403d2463cf1684e4 WatchSource:0}: Error finding container b1423ecde4f96bad4527f3f48759fb59e6f041289e3d7aa9403d2463cf1684e4: Status 404 returned error can't find the container with id b1423ecde4f96bad4527f3f48759fb59e6f041289e3d7aa9403d2463cf1684e4
Oct 30 20:59:19.593683 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:19.593643    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk" event={"ID":"9a73cfa4-d614-4c39-b324-731fca0564c4","Type":"ContainerStarted","Data":"c1ec5df9fca78a32bc25d54802c6a63c1a397a64889906056dfba49017d9ad30"}
Oct 30 20:59:19.593912 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:19.593715    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk" event={"ID":"9a73cfa4-d614-4c39-b324-731fca0564c4","Type":"ContainerStarted","Data":"7a1036f5812038a461d162457f0be1a35c1fa7d5219ea6130ab8395109b3ab1c"}
Oct 30 20:59:19.595850 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:19.595822    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2" event={"ID":"e1255405-8724-4977-854d-11f13d8a847a","Type":"ContainerStarted","Data":"eceac2f023bbcb511f2a9a8ce3078ed19b64ce3e11aa5701a7509788dcc1b859"}
Oct 30 20:59:19.595964 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:19.595873    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2" event={"ID":"e1255405-8724-4977-854d-11f13d8a847a","Type":"ContainerStarted","Data":"71765f9523506cf9462cf503da06c5129da2a0b5a13074bd125a1b0e32d1c723"}
Oct 30 20:59:19.597260 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:19.597227    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e61ebc76-d7p4p" event={"ID":"fdc5287e-e62c-41b0-9ffe-c9f9d27f0cdd","Type":"ContainerStarted","Data":"b1423ecde4f96bad4527f3f48759fb59e6f041289e3d7aa9403d2463cf1684e4"}
Oct 30 20:59:19.728875 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:19.727230    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk" podStartSLOduration=2.727212003 podStartE2EDuration="2.727212003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:59:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:59:19.717502751 +0000 UTC m=+1242.782279151" watchObservedRunningTime="2026-10-30 20:59:19.727212003 +0000 UTC m=+1242.791988402"
Oct 30 20:59:19.855333 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:19.855266    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2" podStartSLOduration=2.8552428560000003 podStartE2EDuration="2.855242856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:59:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:59:19.834127804 +0000 UTC m=+1242.898904203" watchObservedRunningTime="2026-10-30 20:59:19.855242856 +0000 UTC m=+1242.920019255"
Oct 30 20:59:20.207815 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:20.195234    5713 patch_prober.go:29] interesting pod/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 30 20:59:20.207815 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:20.195299    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8" podUID="03ef9f7c-175a-4012-a021-ec011e37928e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 30 20:59:20.370814 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:20.367559    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk"
Oct 30 20:59:20.370814 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:20.369833    5713 patch_prober.go:29] interesting pod/e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 30 20:59:20.370814 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:20.369900    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk" podUID="9a73cfa4-d614-4c39-b324-731fca0564c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 30 20:59:20.383817 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:20.382178    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2"
Oct 30 20:59:20.383817 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:20.383704    5713 patch_prober.go:29] interesting pod/e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 30 20:59:20.383817 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:20.383758    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2" podUID="e1255405-8724-4977-854d-11f13d8a847a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 30 20:59:20.628101 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:20.628010    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e61ebc76-d7p4p" event={"ID":"fdc5287e-e62c-41b0-9ffe-c9f9d27f0cdd","Type":"ContainerStarted","Data":"29d2eef40f757d550bce5da59435377a159730d5c0e008b928238bc3ec97d1cc"}
Oct 30 20:59:21.194233 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:21.194191    5713 patch_prober.go:29] interesting pod/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 30 20:59:21.194517 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:21.194487    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8" podUID="03ef9f7c-175a-4012-a021-ec011e37928e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 30 20:59:21.370996 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:21.370963    5713 patch_prober.go:29] interesting pod/e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 30 20:59:21.371579 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:21.371547    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk" podUID="9a73cfa4-d614-4c39-b324-731fca0564c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 30 20:59:21.385194 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:21.385157    5713 patch_prober.go:29] interesting pod/e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 30 20:59:21.385436 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:21.385405    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2" podUID="e1255405-8724-4977-854d-11f13d8a847a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 30 20:59:21.669876 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:21.669487    5713 generic.go:356] "Generic (PLEG): container finished" podID="fdc5287e-e62c-41b0-9ffe-c9f9d27f0cdd" containerID="29d2eef40f757d550bce5da59435377a159730d5c0e008b928238bc3ec97d1cc" exitCode=0
Oct 30 20:59:21.673812 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:21.670492    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e61ebc76-d7p4p" event={"ID":"fdc5287e-e62c-41b0-9ffe-c9f9d27f0cdd","Type":"ContainerDied","Data":"29d2eef40f757d550bce5da59435377a159730d5c0e008b928238bc3ec97d1cc"}
Oct 30 20:59:22.194267 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:22.194227    5713 patch_prober.go:29] interesting pod/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 30 20:59:22.194569 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:22.194540    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8" podUID="03ef9f7c-175a-4012-a021-ec011e37928e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 30 20:59:22.369629 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:22.369581    5713 patch_prober.go:29] interesting pod/e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 30 20:59:22.369964 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:22.369910    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk" podUID="9a73cfa4-d614-4c39-b324-731fca0564c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 30 20:59:22.379227 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:22.379194    5713 patch_prober.go:29] interesting pod/e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 30 20:59:22.379851 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:22.379820    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2" podUID="e1255405-8724-4977-854d-11f13d8a847a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 30 20:59:23.216833 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:23.215124    5713 patch_prober.go:29] interesting pod/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 30 20:59:23.216833 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:23.215188    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8" podUID="03ef9f7c-175a-4012-a021-ec011e37928e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 30 20:59:23.222560 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:23.222259    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e61ebc76-d7p4p"
Oct 30 20:59:23.330084 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:23.330051    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdc5287e-e62c-41b0-9ffe-c9f9d27f0cdd-kube-api-access-tlpfm\" (UniqueName: \"kubernetes.io/projected/fdc5287e-e62c-41b0-9ffe-c9f9d27f0cdd-kube-api-access-tlpfm\") pod \"fdc5287e-e62c-41b0-9ffe-c9f9d27f0cdd\" (UID: \"fdc5287e-e62c-41b0-9ffe-c9f9d27f0cdd\") "
Oct 30 20:59:23.330313 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:23.330138    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdc5287e-e62c-41b0-9ffe-c9f9d27f0cdd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdc5287e-e62c-41b0-9ffe-c9f9d27f0cdd-maas-api-openshift-ca\") pod \"fdc5287e-e62c-41b0-9ffe-c9f9d27f0cdd\" (UID: \"fdc5287e-e62c-41b0-9ffe-c9f9d27f0cdd\") "
Oct 30 20:59:23.330313 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:23.330276    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdc5287e-e62c-41b0-9ffe-c9f9d27f0cdd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdc5287e-e62c-41b0-9ffe-c9f9d27f0cdd-maas-api-xks-ca\") pod \"fdc5287e-e62c-41b0-9ffe-c9f9d27f0cdd\" (UID: \"fdc5287e-e62c-41b0-9ffe-c9f9d27f0cdd\") "
Oct 30 20:59:23.330448 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:23.330433    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdc5287e-e62c-41b0-9ffe-c9f9d27f0cdd-maas-api-openshift-ca" pod "fdc5287e-e62c-41b0-9ffe-c9f9d27f0cdd" (UID: "fdc5287e-e62c-41b0-9ffe-c9f9d27f0cdd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:59:23.330805 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:23.330771    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdc5287e-e62c-41b0-9ffe-c9f9d27f0cdd-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:23.332671 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:23.332650    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdc5287e-e62c-41b0-9ffe-c9f9d27f0cdd-kube-api-access-tlpfm" pod "fdc5287e-e62c-41b0-9ffe-c9f9d27f0cdd" (UID: "fdc5287e-e62c-41b0-9ffe-c9f9d27f0cdd"). InnerVolumeSpecName "kube-api-access-tlpfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:59:23.333183 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:23.333161    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdc5287e-e62c-41b0-9ffe-c9f9d27f0cdd-maas-api-xks-ca" pod "fdc5287e-e62c-41b0-9ffe-c9f9d27f0cdd" (UID: "fdc5287e-e62c-41b0-9ffe-c9f9d27f0cdd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:59:23.378820 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:23.371897    5713 patch_prober.go:29] interesting pod/e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 30 20:59:23.378820 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:23.371961    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk" podUID="9a73cfa4-d614-4c39-b324-731fca0564c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 30 20:59:23.382858 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:23.379146    5713 patch_prober.go:29] interesting pod/e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 30 20:59:23.382858 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:23.379199    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2" podUID="e1255405-8724-4977-854d-11f13d8a847a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 30 20:59:23.441850 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:23.432983    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlpfm\" (UniqueName: \"kubernetes.io/projected/fdc5287e-e62c-41b0-9ffe-c9f9d27f0cdd-kube-api-access-tlpfm\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:23.441850 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:23.433020    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdc5287e-e62c-41b0-9ffe-c9f9d27f0cdd-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:23.723568 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:23.722406    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e61ebc76-d7p4p"
Oct 30 20:59:23.723568 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:23.722407    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e61ebc76-d7p4p" event={"ID":"fdc5287e-e62c-41b0-9ffe-c9f9d27f0cdd","Type":"ContainerDied","Data":"b1423ecde4f96bad4527f3f48759fb59e6f041289e3d7aa9403d2463cf1684e4"}
Oct 30 20:59:23.723568 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:23.722489    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1423ecde4f96bad4527f3f48759fb59e6f041289e3d7aa9403d2463cf1684e4"
Oct 30 20:59:24.194010 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:24.193837    5713 patch_prober.go:29] interesting pod/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 30 20:59:24.194010 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:24.193895    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8" podUID="03ef9f7c-175a-4012-a021-ec011e37928e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 30 20:59:24.369815 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:24.368876    5713 patch_prober.go:29] interesting pod/e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 30 20:59:24.369815 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:24.368934    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk" podUID="9a73cfa4-d614-4c39-b324-731fca0564c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 30 20:59:24.380814 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:24.378660    5713 patch_prober.go:29] interesting pod/e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 30 20:59:24.380814 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:24.378721    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2" podUID="e1255405-8724-4977-854d-11f13d8a847a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 30 20:59:24.506240 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:59:24.506149    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 20:59:25.193617 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:25.193558    5713 patch_prober.go:29] interesting pod/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 30 20:59:25.193981 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:25.193937    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8" podUID="03ef9f7c-175a-4012-a021-ec011e37928e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 30 20:59:25.369151 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:25.369109    5713 patch_prober.go:29] interesting pod/e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 30 20:59:25.369332 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:25.369177    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk" podUID="9a73cfa4-d614-4c39-b324-731fca0564c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 30 20:59:25.387815 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:25.384895    5713 patch_prober.go:29] interesting pod/e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 30 20:59:25.387815 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:25.384959    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2" podUID="e1255405-8724-4977-854d-11f13d8a847a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 30 20:59:26.199541 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:26.199508    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8"
Oct 30 20:59:26.200109 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:26.199847    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8"
Oct 30 20:59:26.200873 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:26.200851    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8"
Oct 30 20:59:26.367483 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:26.367446    5713 patch_prober.go:29] interesting pod/e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 30 20:59:26.367655 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:26.367508    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk" podUID="9a73cfa4-d614-4c39-b324-731fca0564c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 30 20:59:26.920168 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:26.920132    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2"
Oct 30 20:59:26.921203 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:26.921179    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2"
Oct 30 20:59:26.925708 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:26.925610    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2"
Oct 30 20:59:27.371932 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:27.371902    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk"
Oct 30 20:59:27.372372 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:27.372354    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk"
Oct 30 20:59:27.373008 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:27.372990    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk"
Oct 30 20:59:30.929816 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:30.926310    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9424c8f8-jqnc2"]
Oct 30 20:59:30.958537 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:30.957031    5713 cpu_manager.go:414] "containerMap: removing container" podUID="fdc5287e-e62c-41b0-9ffe-c9f9d27f0cdd" containerName="revoke-keys"
Oct 30 20:59:30.958537 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:30.957259    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdc5287e-e62c-41b0-9ffe-c9f9d27f0cdd" containerName="revoke-keys"
Oct 30 20:59:31.104970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:31.104920    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l" podUID="fc9510c5-2180-41d6-90bc-9acdff0f3616" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:59:31.147913 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:31.147866    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7" podUID="b2193213-ebae-498a-911a-6056a8e94312" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded"
Oct 30 20:59:31.168656 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:31.168623    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdc5287e-e62c-41b0-9ffe-c9f9d27f0cdd" containerName="revoke-keys"
Oct 30 20:59:31.268300 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:31.268213    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9424c8f8-jqnc2"]
Oct 30 20:59:31.279388 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:31.276026    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9424c8f8-jqnc2"
Oct 30 20:59:31.329531 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:31.329490    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:59:31.330083 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:31.330064    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:59:31.330366 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:31.330350    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 20:59:31.330593 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:31.330578    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:59:31.337149 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:31.337126    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8cb8d7c-784pk"]
Oct 30 20:59:31.337341 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:31.337325    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8cb8d7c-784pk"]
Oct 30 20:59:31.468149 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:31.468106    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bba59f9-e753-4e47-8c53-3e59fc35a780-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9424c8f8-jqnc2\" (UID: \"4bba59f9-e753-4e47-8c53-3e59fc35a780\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9424c8f8-jqnc2"
Oct 30 20:59:31.468584 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:31.468558    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwp6z\" (UniqueName: \"kubernetes.io/projected/4bba59f9-e753-4e47-8c53-3e59fc35a780-kube-api-access-zwp6z\") pod \"maas-api-revoke-sub-9424c8f8-jqnc2\" (UID: \"4bba59f9-e753-4e47-8c53-3e59fc35a780\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9424c8f8-jqnc2"
Oct 30 20:59:31.468870 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:31.468849    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bba59f9-e753-4e47-8c53-3e59fc35a780-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9424c8f8-jqnc2\" (UID: \"4bba59f9-e753-4e47-8c53-3e59fc35a780\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9424c8f8-jqnc2"
Oct 30 20:59:31.571225 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:31.571123    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bba59f9-e753-4e47-8c53-3e59fc35a780-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9424c8f8-jqnc2\" (UID: \"4bba59f9-e753-4e47-8c53-3e59fc35a780\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9424c8f8-jqnc2"
Oct 30 20:59:31.572404 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:31.572378    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bba59f9-e753-4e47-8c53-3e59fc35a780-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9424c8f8-jqnc2\" (UID: \"4bba59f9-e753-4e47-8c53-3e59fc35a780\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9424c8f8-jqnc2"
Oct 30 20:59:31.576110 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:31.576088    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwp6z\" (UniqueName: \"kubernetes.io/projected/4bba59f9-e753-4e47-8c53-3e59fc35a780-kube-api-access-zwp6z\") pod \"maas-api-revoke-sub-9424c8f8-jqnc2\" (UID: \"4bba59f9-e753-4e47-8c53-3e59fc35a780\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9424c8f8-jqnc2"
Oct 30 20:59:31.576729 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:31.572237    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bba59f9-e753-4e47-8c53-3e59fc35a780-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9424c8f8-jqnc2\" (UID: \"4bba59f9-e753-4e47-8c53-3e59fc35a780\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9424c8f8-jqnc2"
Oct 30 20:59:31.590275 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:31.590247    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bba59f9-e753-4e47-8c53-3e59fc35a780-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9424c8f8-jqnc2\" (UID: \"4bba59f9-e753-4e47-8c53-3e59fc35a780\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9424c8f8-jqnc2"
Oct 30 20:59:31.658814 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:31.652970    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwp6z\" (UniqueName: \"kubernetes.io/projected/4bba59f9-e753-4e47-8c53-3e59fc35a780-kube-api-access-zwp6z\") pod \"maas-api-revoke-sub-9424c8f8-jqnc2\" (UID: \"4bba59f9-e753-4e47-8c53-3e59fc35a780\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9424c8f8-jqnc2"
Oct 30 20:59:31.658814 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:31.653199    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9424c8f8-jqnc2"
Oct 30 20:59:32.690564 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:32.690536    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9424c8f8-jqnc2"]
Oct 30 20:59:32.691036 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:59:32.690576    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bba59f9_e753_4e47_8c53_3e59fc35a780.slice/crio-e9067195c7df4b1565bbe03d4f5e4915f1b180e64bb5d04dda5b88caec7bc4ec WatchSource:0}: Error finding container e9067195c7df4b1565bbe03d4f5e4915f1b180e64bb5d04dda5b88caec7bc4ec: Status 404 returned error can't find the container with id e9067195c7df4b1565bbe03d4f5e4915f1b180e64bb5d04dda5b88caec7bc4ec
Oct 30 20:59:32.906583 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:32.906552    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9424c8f8-jqnc2" event={"ID":"4bba59f9-e753-4e47-8c53-3e59fc35a780","Type":"ContainerStarted","Data":"e9067195c7df4b1565bbe03d4f5e4915f1b180e64bb5d04dda5b88caec7bc4ec"}
Oct 30 20:59:33.066832 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:33.066776    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a300bba-c70c-4ff1-97d6-7517fa970477" path="/var/lib/kubelet/pods/1a300bba-c70c-4ff1-97d6-7517fa970477/volumes"
Oct 30 20:59:33.916263 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:33.916162    5713 generic.go:356] "Generic (PLEG): container finished" podID="4bba59f9-e753-4e47-8c53-3e59fc35a780" containerID="6045a0c58a6b856e0ec0931bde8a2dcad3f0e191f85d4fc4830426f026b07ddb" exitCode=0
Oct 30 20:59:33.916263 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:33.916245    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9424c8f8-jqnc2" event={"ID":"4bba59f9-e753-4e47-8c53-3e59fc35a780","Type":"ContainerDied","Data":"6045a0c58a6b856e0ec0931bde8a2dcad3f0e191f85d4fc4830426f026b07ddb"}
Oct 30 20:59:34.103495 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:34.103468    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ac02efe-kkzwt"]
Oct 30 20:59:34.107532 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:34.107515    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ac02efe-kkzwt"
Oct 30 20:59:34.144941 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:34.144892    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-684h9\" (UniqueName: \"kubernetes.io/projected/5346bed6-4f88-420e-a76f-b98fc50d1c2c-kube-api-access-684h9\") pod \"maas-api-revoke-sub-3ac02efe-kkzwt\" (UID: \"5346bed6-4f88-420e-a76f-b98fc50d1c2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ac02efe-kkzwt"
Oct 30 20:59:34.145135 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:34.145106    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5346bed6-4f88-420e-a76f-b98fc50d1c2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ac02efe-kkzwt\" (UID: \"5346bed6-4f88-420e-a76f-b98fc50d1c2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ac02efe-kkzwt"
Oct 30 20:59:34.145361 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:34.145334    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5346bed6-4f88-420e-a76f-b98fc50d1c2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ac02efe-kkzwt\" (UID: \"5346bed6-4f88-420e-a76f-b98fc50d1c2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ac02efe-kkzwt"
Oct 30 20:59:34.165742 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:34.165712    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ac02efe-kkzwt"]
Oct 30 20:59:34.246869 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:34.246834    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-684h9\" (UniqueName: \"kubernetes.io/projected/5346bed6-4f88-420e-a76f-b98fc50d1c2c-kube-api-access-684h9\") pod \"maas-api-revoke-sub-3ac02efe-kkzwt\" (UID: \"5346bed6-4f88-420e-a76f-b98fc50d1c2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ac02efe-kkzwt"
Oct 30 20:59:34.247027 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:34.246914    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5346bed6-4f88-420e-a76f-b98fc50d1c2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ac02efe-kkzwt\" (UID: \"5346bed6-4f88-420e-a76f-b98fc50d1c2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ac02efe-kkzwt"
Oct 30 20:59:34.247027 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:34.246963    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5346bed6-4f88-420e-a76f-b98fc50d1c2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ac02efe-kkzwt\" (UID: \"5346bed6-4f88-420e-a76f-b98fc50d1c2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ac02efe-kkzwt"
Oct 30 20:59:34.247765 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:34.247739    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5346bed6-4f88-420e-a76f-b98fc50d1c2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ac02efe-kkzwt\" (UID: \"5346bed6-4f88-420e-a76f-b98fc50d1c2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ac02efe-kkzwt"
Oct 30 20:59:34.252306 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:34.252274    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5346bed6-4f88-420e-a76f-b98fc50d1c2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ac02efe-kkzwt\" (UID: \"5346bed6-4f88-420e-a76f-b98fc50d1c2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ac02efe-kkzwt"
Oct 30 20:59:34.305648 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:34.305620    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-684h9\" (UniqueName: \"kubernetes.io/projected/5346bed6-4f88-420e-a76f-b98fc50d1c2c-kube-api-access-684h9\") pod \"maas-api-revoke-sub-3ac02efe-kkzwt\" (UID: \"5346bed6-4f88-420e-a76f-b98fc50d1c2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ac02efe-kkzwt"
Oct 30 20:59:34.421984 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:34.421949    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ac02efe-kkzwt"
Oct 30 20:59:34.826533 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:34.826507    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ac02efe-kkzwt"]
Oct 30 20:59:34.827219 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:59:34.827192    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5346bed6_4f88_420e_a76f_b98fc50d1c2c.slice/crio-b7821645149f554057ab723a339d8fe641b41b2272b8ac9432c0ae5ef371215c WatchSource:0}: Error finding container b7821645149f554057ab723a339d8fe641b41b2272b8ac9432c0ae5ef371215c: Status 404 returned error can't find the container with id b7821645149f554057ab723a339d8fe641b41b2272b8ac9432c0ae5ef371215c
Oct 30 20:59:34.925896 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:34.925852    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ac02efe-kkzwt" event={"ID":"5346bed6-4f88-420e-a76f-b98fc50d1c2c","Type":"ContainerStarted","Data":"b7821645149f554057ab723a339d8fe641b41b2272b8ac9432c0ae5ef371215c"}
Oct 30 20:59:35.056087 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:35.056059    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9424c8f8-jqnc2"
Oct 30 20:59:35.157118 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:35.157093    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bba59f9-e753-4e47-8c53-3e59fc35a780-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bba59f9-e753-4e47-8c53-3e59fc35a780-maas-api-xks-ca\") pod \"4bba59f9-e753-4e47-8c53-3e59fc35a780\" (UID: \"4bba59f9-e753-4e47-8c53-3e59fc35a780\") "
Oct 30 20:59:35.157239 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:35.157143    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bba59f9-e753-4e47-8c53-3e59fc35a780-kube-api-access-zwp6z\" (UniqueName: \"kubernetes.io/projected/4bba59f9-e753-4e47-8c53-3e59fc35a780-kube-api-access-zwp6z\") pod \"4bba59f9-e753-4e47-8c53-3e59fc35a780\" (UID: \"4bba59f9-e753-4e47-8c53-3e59fc35a780\") "
Oct 30 20:59:35.157239 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:35.157182    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bba59f9-e753-4e47-8c53-3e59fc35a780-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bba59f9-e753-4e47-8c53-3e59fc35a780-maas-api-openshift-ca\") pod \"4bba59f9-e753-4e47-8c53-3e59fc35a780\" (UID: \"4bba59f9-e753-4e47-8c53-3e59fc35a780\") "
Oct 30 20:59:35.157617 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:35.157594    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bba59f9-e753-4e47-8c53-3e59fc35a780-maas-api-openshift-ca" pod "4bba59f9-e753-4e47-8c53-3e59fc35a780" (UID: "4bba59f9-e753-4e47-8c53-3e59fc35a780"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:59:35.157914 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:35.157893    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bba59f9-e753-4e47-8c53-3e59fc35a780-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:35.159327 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:35.159305    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bba59f9-e753-4e47-8c53-3e59fc35a780-kube-api-access-zwp6z" pod "4bba59f9-e753-4e47-8c53-3e59fc35a780" (UID: "4bba59f9-e753-4e47-8c53-3e59fc35a780"). InnerVolumeSpecName "kube-api-access-zwp6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:59:35.159403 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:35.159343    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bba59f9-e753-4e47-8c53-3e59fc35a780-maas-api-xks-ca" pod "4bba59f9-e753-4e47-8c53-3e59fc35a780" (UID: "4bba59f9-e753-4e47-8c53-3e59fc35a780"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:59:35.258976 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:35.258933    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bba59f9-e753-4e47-8c53-3e59fc35a780-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:35.258976 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:35.258967    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwp6z\" (UniqueName: \"kubernetes.io/projected/4bba59f9-e753-4e47-8c53-3e59fc35a780-kube-api-access-zwp6z\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:35.936982 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:35.936948    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9424c8f8-jqnc2"
Oct 30 20:59:35.937452 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:35.936946    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9424c8f8-jqnc2" event={"ID":"4bba59f9-e753-4e47-8c53-3e59fc35a780","Type":"ContainerDied","Data":"e9067195c7df4b1565bbe03d4f5e4915f1b180e64bb5d04dda5b88caec7bc4ec"}
Oct 30 20:59:35.937452 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:35.937100    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9067195c7df4b1565bbe03d4f5e4915f1b180e64bb5d04dda5b88caec7bc4ec"
Oct 30 20:59:35.938853 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:35.938821    5713 generic.go:356] "Generic (PLEG): container finished" podID="5346bed6-4f88-420e-a76f-b98fc50d1c2c" containerID="57936174c0bbdc452f0ba8250b4ba5501322f3fe448b4365427a7a2720b7c72f" exitCode=0
Oct 30 20:59:35.938970 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:35.938863    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ac02efe-kkzwt" event={"ID":"5346bed6-4f88-420e-a76f-b98fc50d1c2c","Type":"ContainerDied","Data":"57936174c0bbdc452f0ba8250b4ba5501322f3fe448b4365427a7a2720b7c72f"}
Oct 30 20:59:36.550850 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:36.550820    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c76f5bd-z4wm4"]
Oct 30 20:59:36.551397 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:36.551384    5713 cpu_manager.go:414] "containerMap: removing container" podUID="4bba59f9-e753-4e47-8c53-3e59fc35a780" containerName="revoke-keys"
Oct 30 20:59:36.551446 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:36.551399    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bba59f9-e753-4e47-8c53-3e59fc35a780" containerName="revoke-keys"
Oct 30 20:59:36.551540 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:36.551530    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bba59f9-e753-4e47-8c53-3e59fc35a780" containerName="revoke-keys"
Oct 30 20:59:36.554997 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:36.554982    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c76f5bd-z4wm4"
Oct 30 20:59:36.575345 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:36.575311    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a541157-cbb6-442a-98cf-02dd5345cb8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c76f5bd-z4wm4\" (UID: \"5a541157-cbb6-442a-98cf-02dd5345cb8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c76f5bd-z4wm4"
Oct 30 20:59:36.575511 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:36.575406    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a541157-cbb6-442a-98cf-02dd5345cb8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c76f5bd-z4wm4\" (UID: \"5a541157-cbb6-442a-98cf-02dd5345cb8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c76f5bd-z4wm4"
Oct 30 20:59:36.575574 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:36.575512    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqhzb\" (UniqueName: \"kubernetes.io/projected/5a541157-cbb6-442a-98cf-02dd5345cb8e-kube-api-access-wqhzb\") pod \"maas-api-revoke-sub-8c76f5bd-z4wm4\" (UID: \"5a541157-cbb6-442a-98cf-02dd5345cb8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c76f5bd-z4wm4"
Oct 30 20:59:36.611379 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:36.611352    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c76f5bd-z4wm4"]
Oct 30 20:59:36.676915 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:36.676870    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a541157-cbb6-442a-98cf-02dd5345cb8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c76f5bd-z4wm4\" (UID: \"5a541157-cbb6-442a-98cf-02dd5345cb8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c76f5bd-z4wm4"
Oct 30 20:59:36.677125 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:36.676996    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqhzb\" (UniqueName: \"kubernetes.io/projected/5a541157-cbb6-442a-98cf-02dd5345cb8e-kube-api-access-wqhzb\") pod \"maas-api-revoke-sub-8c76f5bd-z4wm4\" (UID: \"5a541157-cbb6-442a-98cf-02dd5345cb8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c76f5bd-z4wm4"
Oct 30 20:59:36.677233 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:36.677217    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a541157-cbb6-442a-98cf-02dd5345cb8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c76f5bd-z4wm4\" (UID: \"5a541157-cbb6-442a-98cf-02dd5345cb8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c76f5bd-z4wm4"
Oct 30 20:59:36.677534 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:36.677515    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a541157-cbb6-442a-98cf-02dd5345cb8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c76f5bd-z4wm4\" (UID: \"5a541157-cbb6-442a-98cf-02dd5345cb8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c76f5bd-z4wm4"
Oct 30 20:59:36.679716 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:36.679695    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a541157-cbb6-442a-98cf-02dd5345cb8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c76f5bd-z4wm4\" (UID: \"5a541157-cbb6-442a-98cf-02dd5345cb8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c76f5bd-z4wm4"
Oct 30 20:59:36.735274 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:36.735242    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqhzb\" (UniqueName: \"kubernetes.io/projected/5a541157-cbb6-442a-98cf-02dd5345cb8e-kube-api-access-wqhzb\") pod \"maas-api-revoke-sub-8c76f5bd-z4wm4\" (UID: \"5a541157-cbb6-442a-98cf-02dd5345cb8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c76f5bd-z4wm4"
Oct 30 20:59:36.868484 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:36.868396    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c76f5bd-z4wm4"
Oct 30 20:59:37.114022 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:37.113993    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ac02efe-kkzwt"
Oct 30 20:59:37.184700 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:37.184666    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5346bed6-4f88-420e-a76f-b98fc50d1c2c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5346bed6-4f88-420e-a76f-b98fc50d1c2c-maas-api-openshift-ca\") pod \"5346bed6-4f88-420e-a76f-b98fc50d1c2c\" (UID: \"5346bed6-4f88-420e-a76f-b98fc50d1c2c\") "
Oct 30 20:59:37.184927 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:37.184717    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5346bed6-4f88-420e-a76f-b98fc50d1c2c-kube-api-access-684h9\" (UniqueName: \"kubernetes.io/projected/5346bed6-4f88-420e-a76f-b98fc50d1c2c-kube-api-access-684h9\") pod \"5346bed6-4f88-420e-a76f-b98fc50d1c2c\" (UID: \"5346bed6-4f88-420e-a76f-b98fc50d1c2c\") "
Oct 30 20:59:37.184927 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:37.184742    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5346bed6-4f88-420e-a76f-b98fc50d1c2c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5346bed6-4f88-420e-a76f-b98fc50d1c2c-maas-api-xks-ca\") pod \"5346bed6-4f88-420e-a76f-b98fc50d1c2c\" (UID: \"5346bed6-4f88-420e-a76f-b98fc50d1c2c\") "
Oct 30 20:59:37.185140 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:37.185106    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5346bed6-4f88-420e-a76f-b98fc50d1c2c-maas-api-openshift-ca" pod "5346bed6-4f88-420e-a76f-b98fc50d1c2c" (UID: "5346bed6-4f88-420e-a76f-b98fc50d1c2c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:59:37.185712 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:37.185693    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5346bed6-4f88-420e-a76f-b98fc50d1c2c-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:37.187167 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:37.187147    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5346bed6-4f88-420e-a76f-b98fc50d1c2c-kube-api-access-684h9" pod "5346bed6-4f88-420e-a76f-b98fc50d1c2c" (UID: "5346bed6-4f88-420e-a76f-b98fc50d1c2c"). InnerVolumeSpecName "kube-api-access-684h9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:59:37.187492 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:37.187475    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5346bed6-4f88-420e-a76f-b98fc50d1c2c-maas-api-xks-ca" pod "5346bed6-4f88-420e-a76f-b98fc50d1c2c" (UID: "5346bed6-4f88-420e-a76f-b98fc50d1c2c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:59:37.244278 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:37.244247    5713 scope.go:122] "RemoveContainer" containerID="3a8d769a8a55f38b6fbaac74ccc8ef912347a6865c0e492d322d83f640de4c9a"
Oct 30 20:59:37.251705 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:37.251677    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c76f5bd-z4wm4"]
Oct 30 20:59:37.252015 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:59:37.251987    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a541157_cbb6_442a_98cf_02dd5345cb8e.slice/crio-6992f667bf00a631cafc5018166ca0d8772163b4913e53a5b9495d0da16d86af WatchSource:0}: Error finding container 6992f667bf00a631cafc5018166ca0d8772163b4913e53a5b9495d0da16d86af: Status 404 returned error can't find the container with id 6992f667bf00a631cafc5018166ca0d8772163b4913e53a5b9495d0da16d86af
Oct 30 20:59:37.286545 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:37.286516    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-684h9\" (UniqueName: \"kubernetes.io/projected/5346bed6-4f88-420e-a76f-b98fc50d1c2c-kube-api-access-684h9\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:37.286545 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:37.286547    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5346bed6-4f88-420e-a76f-b98fc50d1c2c-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:37.478576 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:37.478547    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ed62713-2ff75"]
Oct 30 20:59:37.479224 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:37.479206    5713 cpu_manager.go:414] "containerMap: removing container" podUID="5346bed6-4f88-420e-a76f-b98fc50d1c2c" containerName="revoke-keys"
Oct 30 20:59:37.479305 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:37.479226    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5346bed6-4f88-420e-a76f-b98fc50d1c2c" containerName="revoke-keys"
Oct 30 20:59:37.479344 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:37.479322    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="5346bed6-4f88-420e-a76f-b98fc50d1c2c" containerName="revoke-keys"
Oct 30 20:59:37.484308 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:37.484287    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ed62713-2ff75"
Oct 30 20:59:37.540130 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:37.540096    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ed62713-2ff75"]
Oct 30 20:59:37.590151 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:37.590122    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da4c03af-ef11-46ae-bc29-e8e607cfc505-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ed62713-2ff75\" (UID: \"da4c03af-ef11-46ae-bc29-e8e607cfc505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ed62713-2ff75"
Oct 30 20:59:37.590342 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:37.590249    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da4c03af-ef11-46ae-bc29-e8e607cfc505-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ed62713-2ff75\" (UID: \"da4c03af-ef11-46ae-bc29-e8e607cfc505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ed62713-2ff75"
Oct 30 20:59:37.590342 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:37.590296    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhd8l\" (UniqueName: \"kubernetes.io/projected/da4c03af-ef11-46ae-bc29-e8e607cfc505-kube-api-access-nhd8l\") pod \"maas-api-revoke-sub-4ed62713-2ff75\" (UID: \"da4c03af-ef11-46ae-bc29-e8e607cfc505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ed62713-2ff75"
Oct 30 20:59:37.691755 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:37.691719    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da4c03af-ef11-46ae-bc29-e8e607cfc505-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ed62713-2ff75\" (UID: \"da4c03af-ef11-46ae-bc29-e8e607cfc505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ed62713-2ff75"
Oct 30 20:59:37.691941 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:37.691880    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da4c03af-ef11-46ae-bc29-e8e607cfc505-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ed62713-2ff75\" (UID: \"da4c03af-ef11-46ae-bc29-e8e607cfc505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ed62713-2ff75"
Oct 30 20:59:37.692009 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:37.691942    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhd8l\" (UniqueName: \"kubernetes.io/projected/da4c03af-ef11-46ae-bc29-e8e607cfc505-kube-api-access-nhd8l\") pod \"maas-api-revoke-sub-4ed62713-2ff75\" (UID: \"da4c03af-ef11-46ae-bc29-e8e607cfc505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ed62713-2ff75"
Oct 30 20:59:37.692751 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:37.692726    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da4c03af-ef11-46ae-bc29-e8e607cfc505-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ed62713-2ff75\" (UID: \"da4c03af-ef11-46ae-bc29-e8e607cfc505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ed62713-2ff75"
Oct 30 20:59:37.694506 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:37.694487    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da4c03af-ef11-46ae-bc29-e8e607cfc505-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ed62713-2ff75\" (UID: \"da4c03af-ef11-46ae-bc29-e8e607cfc505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ed62713-2ff75"
Oct 30 20:59:37.749296 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:37.749230    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhd8l\" (UniqueName: \"kubernetes.io/projected/da4c03af-ef11-46ae-bc29-e8e607cfc505-kube-api-access-nhd8l\") pod \"maas-api-revoke-sub-4ed62713-2ff75\" (UID: \"da4c03af-ef11-46ae-bc29-e8e607cfc505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ed62713-2ff75"
Oct 30 20:59:37.799018 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:37.798981    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ed62713-2ff75"
Oct 30 20:59:37.960855 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:37.960811    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c76f5bd-z4wm4" event={"ID":"5a541157-cbb6-442a-98cf-02dd5345cb8e","Type":"ContainerStarted","Data":"c24e3bb2ee11c3d413ab1f10968dc4032925964860c1bc5e91fff88fb1abff5a"}
Oct 30 20:59:37.961054 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:37.960878    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c76f5bd-z4wm4" event={"ID":"5a541157-cbb6-442a-98cf-02dd5345cb8e","Type":"ContainerStarted","Data":"6992f667bf00a631cafc5018166ca0d8772163b4913e53a5b9495d0da16d86af"}
Oct 30 20:59:37.965486 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:37.963465    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ac02efe-kkzwt" event={"ID":"5346bed6-4f88-420e-a76f-b98fc50d1c2c","Type":"ContainerDied","Data":"b7821645149f554057ab723a339d8fe641b41b2272b8ac9432c0ae5ef371215c"}
Oct 30 20:59:37.965486 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:37.963521    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7821645149f554057ab723a339d8fe641b41b2272b8ac9432c0ae5ef371215c"
Oct 30 20:59:37.965486 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:37.963582    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ac02efe-kkzwt"
Oct 30 20:59:38.078342 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:38.078248    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c76f5bd-z4wm4" podStartSLOduration=2.07823339 podStartE2EDuration="2.07823339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:59:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:59:38.076482214 +0000 UTC m=+1261.141258615" watchObservedRunningTime="2026-10-30 20:59:38.07823339 +0000 UTC m=+1261.143009825"
Oct 30 20:59:38.185603 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:38.185572    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ed62713-2ff75"]
Oct 30 20:59:38.186032 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:59:38.185727    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda4c03af_ef11_46ae_bc29_e8e607cfc505.slice/crio-21bdd1163243ad41d997760f422850762e13ae47dd6f74a9a6b69a54e6823d8b WatchSource:0}: Error finding container 21bdd1163243ad41d997760f422850762e13ae47dd6f74a9a6b69a54e6823d8b: Status 404 returned error can't find the container with id 21bdd1163243ad41d997760f422850762e13ae47dd6f74a9a6b69a54e6823d8b
Oct 30 20:59:38.974272 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:38.974234    5713 generic.go:356] "Generic (PLEG): container finished" podID="5a541157-cbb6-442a-98cf-02dd5345cb8e" containerID="c24e3bb2ee11c3d413ab1f10968dc4032925964860c1bc5e91fff88fb1abff5a" exitCode=0
Oct 30 20:59:38.974455 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:38.974313    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c76f5bd-z4wm4" event={"ID":"5a541157-cbb6-442a-98cf-02dd5345cb8e","Type":"ContainerDied","Data":"c24e3bb2ee11c3d413ab1f10968dc4032925964860c1bc5e91fff88fb1abff5a"}
Oct 30 20:59:38.976215 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:38.976188    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ed62713-2ff75" event={"ID":"da4c03af-ef11-46ae-bc29-e8e607cfc505","Type":"ContainerStarted","Data":"2105fc4b9d195aa05730e6eaf81c681e9e23c019099d26c8d6a7852be37b3015"}
Oct 30 20:59:38.976344 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:38.976241    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ed62713-2ff75" event={"ID":"da4c03af-ef11-46ae-bc29-e8e607cfc505","Type":"ContainerStarted","Data":"21bdd1163243ad41d997760f422850762e13ae47dd6f74a9a6b69a54e6823d8b"}
Oct 30 20:59:39.080993 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:39.080953    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67ce5c56-b5hz9"]
Oct 30 20:59:39.085100 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:39.085071    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67ce5c56-b5hz9"]
Oct 30 20:59:39.211553 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:39.211502    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ed62713-2ff75" podStartSLOduration=2.211488177 podStartE2EDuration="2.211488177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:59:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:59:39.207893649 +0000 UTC m=+1262.272670047" watchObservedRunningTime="2026-10-30 20:59:39.211488177 +0000 UTC m=+1262.276264575"
Oct 30 20:59:39.986655 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:39.986613    5713 generic.go:356] "Generic (PLEG): container finished" podID="da4c03af-ef11-46ae-bc29-e8e607cfc505" containerID="2105fc4b9d195aa05730e6eaf81c681e9e23c019099d26c8d6a7852be37b3015" exitCode=0
Oct 30 20:59:39.986849 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:39.986693    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ed62713-2ff75" event={"ID":"da4c03af-ef11-46ae-bc29-e8e607cfc505","Type":"ContainerDied","Data":"2105fc4b9d195aa05730e6eaf81c681e9e23c019099d26c8d6a7852be37b3015"}
Oct 30 20:59:40.164302 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:40.164276    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c76f5bd-z4wm4"
Oct 30 20:59:40.229356 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:40.229325    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a541157-cbb6-442a-98cf-02dd5345cb8e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a541157-cbb6-442a-98cf-02dd5345cb8e-maas-api-xks-ca\") pod \"5a541157-cbb6-442a-98cf-02dd5345cb8e\" (UID: \"5a541157-cbb6-442a-98cf-02dd5345cb8e\") "
Oct 30 20:59:40.229831 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:40.229438    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a541157-cbb6-442a-98cf-02dd5345cb8e-kube-api-access-wqhzb\" (UniqueName: \"kubernetes.io/projected/5a541157-cbb6-442a-98cf-02dd5345cb8e-kube-api-access-wqhzb\") pod \"5a541157-cbb6-442a-98cf-02dd5345cb8e\" (UID: \"5a541157-cbb6-442a-98cf-02dd5345cb8e\") "
Oct 30 20:59:40.229831 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:40.229504    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a541157-cbb6-442a-98cf-02dd5345cb8e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a541157-cbb6-442a-98cf-02dd5345cb8e-maas-api-openshift-ca\") pod \"5a541157-cbb6-442a-98cf-02dd5345cb8e\" (UID: \"5a541157-cbb6-442a-98cf-02dd5345cb8e\") "
Oct 30 20:59:40.230009 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:40.229982    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a541157-cbb6-442a-98cf-02dd5345cb8e-maas-api-openshift-ca" pod "5a541157-cbb6-442a-98cf-02dd5345cb8e" (UID: "5a541157-cbb6-442a-98cf-02dd5345cb8e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:59:40.230185 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:40.230105    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a541157-cbb6-442a-98cf-02dd5345cb8e-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:40.231614 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:40.231588    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a541157-cbb6-442a-98cf-02dd5345cb8e-maas-api-xks-ca" pod "5a541157-cbb6-442a-98cf-02dd5345cb8e" (UID: "5a541157-cbb6-442a-98cf-02dd5345cb8e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:59:40.231899 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:40.231879    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a541157-cbb6-442a-98cf-02dd5345cb8e-kube-api-access-wqhzb" pod "5a541157-cbb6-442a-98cf-02dd5345cb8e" (UID: "5a541157-cbb6-442a-98cf-02dd5345cb8e"). InnerVolumeSpecName "kube-api-access-wqhzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:59:40.331429 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:40.331399    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a541157-cbb6-442a-98cf-02dd5345cb8e-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:40.331429 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:40.331428    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqhzb\" (UniqueName: \"kubernetes.io/projected/5a541157-cbb6-442a-98cf-02dd5345cb8e-kube-api-access-wqhzb\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:40.996857 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:40.996779    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c76f5bd-z4wm4"
Oct 30 20:59:40.996857 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:40.996830    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c76f5bd-z4wm4" event={"ID":"5a541157-cbb6-442a-98cf-02dd5345cb8e","Type":"ContainerDied","Data":"6992f667bf00a631cafc5018166ca0d8772163b4913e53a5b9495d0da16d86af"}
Oct 30 20:59:40.997106 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:40.996890    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6992f667bf00a631cafc5018166ca0d8772163b4913e53a5b9495d0da16d86af"
Oct 30 20:59:41.070747 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:41.070708    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b1e7680-458d-4ad9-a82a-af10b825e27f" path="/var/lib/kubelet/pods/3b1e7680-458d-4ad9-a82a-af10b825e27f/volumes"
Oct 30 20:59:41.154665 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:41.154639    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ed62713-2ff75"
Oct 30 20:59:41.243177 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:41.243141    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da4c03af-ef11-46ae-bc29-e8e607cfc505-kube-api-access-nhd8l\" (UniqueName: \"kubernetes.io/projected/da4c03af-ef11-46ae-bc29-e8e607cfc505-kube-api-access-nhd8l\") pod \"da4c03af-ef11-46ae-bc29-e8e607cfc505\" (UID: \"da4c03af-ef11-46ae-bc29-e8e607cfc505\") "
Oct 30 20:59:41.243655 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:41.243204    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da4c03af-ef11-46ae-bc29-e8e607cfc505-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da4c03af-ef11-46ae-bc29-e8e607cfc505-maas-api-xks-ca\") pod \"da4c03af-ef11-46ae-bc29-e8e607cfc505\" (UID: \"da4c03af-ef11-46ae-bc29-e8e607cfc505\") "
Oct 30 20:59:41.243655 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:41.243438    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da4c03af-ef11-46ae-bc29-e8e607cfc505-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da4c03af-ef11-46ae-bc29-e8e607cfc505-maas-api-openshift-ca\") pod \"da4c03af-ef11-46ae-bc29-e8e607cfc505\" (UID: \"da4c03af-ef11-46ae-bc29-e8e607cfc505\") "
Oct 30 20:59:41.243764 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:41.243745    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da4c03af-ef11-46ae-bc29-e8e607cfc505-maas-api-openshift-ca" pod "da4c03af-ef11-46ae-bc29-e8e607cfc505" (UID: "da4c03af-ef11-46ae-bc29-e8e607cfc505"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:59:41.244009 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:41.243989    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da4c03af-ef11-46ae-bc29-e8e607cfc505-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:41.245993 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:41.245967    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da4c03af-ef11-46ae-bc29-e8e607cfc505-maas-api-xks-ca" pod "da4c03af-ef11-46ae-bc29-e8e607cfc505" (UID: "da4c03af-ef11-46ae-bc29-e8e607cfc505"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:59:41.245993 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:41.245977    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da4c03af-ef11-46ae-bc29-e8e607cfc505-kube-api-access-nhd8l" pod "da4c03af-ef11-46ae-bc29-e8e607cfc505" (UID: "da4c03af-ef11-46ae-bc29-e8e607cfc505"). InnerVolumeSpecName "kube-api-access-nhd8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:59:41.345303 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:41.345265    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhd8l\" (UniqueName: \"kubernetes.io/projected/da4c03af-ef11-46ae-bc29-e8e607cfc505-kube-api-access-nhd8l\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:41.345303 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:41.345298    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da4c03af-ef11-46ae-bc29-e8e607cfc505-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:42.005409 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:42.005383    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ed62713-2ff75"
Oct 30 20:59:42.005582 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:42.005425    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ed62713-2ff75" event={"ID":"da4c03af-ef11-46ae-bc29-e8e607cfc505","Type":"ContainerDied","Data":"21bdd1163243ad41d997760f422850762e13ae47dd6f74a9a6b69a54e6823d8b"}
Oct 30 20:59:42.005582 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:42.005493    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21bdd1163243ad41d997760f422850762e13ae47dd6f74a9a6b69a54e6823d8b"
Oct 30 20:59:42.124416 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:42.124384    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-sjpxz"]
Oct 30 20:59:42.124592 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:42.124571    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-sjpxz" podUID="f543e9a0-a5a6-417a-8f00-b0627a5c1776" containerName="istio-proxy" containerID="cri-o://639cde1a6ec57812a7185a4df695ef7d15ae78553fa1c7389e2c179598726ec7" gracePeriod=30
Oct 30 20:59:45.459358 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:45.459320    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef5fd58f-j4dmr"]
Oct 30 20:59:45.459931 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:45.459915    5713 cpu_manager.go:414] "containerMap: removing container" podUID="5a541157-cbb6-442a-98cf-02dd5345cb8e" containerName="revoke-keys"
Oct 30 20:59:45.459990 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:45.459932    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a541157-cbb6-442a-98cf-02dd5345cb8e" containerName="revoke-keys"
Oct 30 20:59:45.459990 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:45.459955    5713 cpu_manager.go:414] "containerMap: removing container" podUID="da4c03af-ef11-46ae-bc29-e8e607cfc505" containerName="revoke-keys"
Oct 30 20:59:45.459990 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:45.459960    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da4c03af-ef11-46ae-bc29-e8e607cfc505" containerName="revoke-keys"
Oct 30 20:59:45.460092 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:45.460082    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a541157-cbb6-442a-98cf-02dd5345cb8e" containerName="revoke-keys"
Oct 30 20:59:45.460128 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:45.460096    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="da4c03af-ef11-46ae-bc29-e8e607cfc505" containerName="revoke-keys"
Oct 30 20:59:45.463779 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:45.463762    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5fd58f-j4dmr"
Oct 30 20:59:45.525193 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:45.525148    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef5fd58f-j4dmr"]
Oct 30 20:59:45.568343 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:45.568305    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:59:45.568343 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:45.568309    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:59:45.568343 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:45.568337    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:59:45.568585 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:45.568367    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 20:59:45.598474 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:45.598439    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90aaa4a2-48e3-4baa-bdc9-27b7c308ec89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef5fd58f-j4dmr\" (UID: \"90aaa4a2-48e3-4baa-bdc9-27b7c308ec89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5fd58f-j4dmr"
Oct 30 20:59:45.598629 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:45.598497    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90aaa4a2-48e3-4baa-bdc9-27b7c308ec89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef5fd58f-j4dmr\" (UID: \"90aaa4a2-48e3-4baa-bdc9-27b7c308ec89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5fd58f-j4dmr"
Oct 30 20:59:45.598753 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:45.598731    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrvv2\" (UniqueName: \"kubernetes.io/projected/90aaa4a2-48e3-4baa-bdc9-27b7c308ec89-kube-api-access-xrvv2\") pod \"maas-api-revoke-sub-ef5fd58f-j4dmr\" (UID: \"90aaa4a2-48e3-4baa-bdc9-27b7c308ec89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5fd58f-j4dmr"
Oct 30 20:59:45.700689 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:45.700649    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrvv2\" (UniqueName: \"kubernetes.io/projected/90aaa4a2-48e3-4baa-bdc9-27b7c308ec89-kube-api-access-xrvv2\") pod \"maas-api-revoke-sub-ef5fd58f-j4dmr\" (UID: \"90aaa4a2-48e3-4baa-bdc9-27b7c308ec89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5fd58f-j4dmr"
Oct 30 20:59:45.700891 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:45.700766    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90aaa4a2-48e3-4baa-bdc9-27b7c308ec89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef5fd58f-j4dmr\" (UID: \"90aaa4a2-48e3-4baa-bdc9-27b7c308ec89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5fd58f-j4dmr"
Oct 30 20:59:45.700891 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:45.700818    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90aaa4a2-48e3-4baa-bdc9-27b7c308ec89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef5fd58f-j4dmr\" (UID: \"90aaa4a2-48e3-4baa-bdc9-27b7c308ec89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5fd58f-j4dmr"
Oct 30 20:59:45.701449 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:45.701421    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90aaa4a2-48e3-4baa-bdc9-27b7c308ec89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef5fd58f-j4dmr\" (UID: \"90aaa4a2-48e3-4baa-bdc9-27b7c308ec89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5fd58f-j4dmr"
Oct 30 20:59:45.703428 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:45.703402    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90aaa4a2-48e3-4baa-bdc9-27b7c308ec89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef5fd58f-j4dmr\" (UID: \"90aaa4a2-48e3-4baa-bdc9-27b7c308ec89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5fd58f-j4dmr"
Oct 30 20:59:45.758805 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:45.758717    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrvv2\" (UniqueName: \"kubernetes.io/projected/90aaa4a2-48e3-4baa-bdc9-27b7c308ec89-kube-api-access-xrvv2\") pod \"maas-api-revoke-sub-ef5fd58f-j4dmr\" (UID: \"90aaa4a2-48e3-4baa-bdc9-27b7c308ec89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5fd58f-j4dmr"
Oct 30 20:59:45.780839 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:45.780809    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5fd58f-j4dmr"
Oct 30 20:59:46.070239 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:46.070153    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61484a2b-4qggj"]
Oct 30 20:59:46.075930 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:46.075901    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61484a2b-4qggj"]
Oct 30 20:59:46.189670 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:46.189640    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef5fd58f-j4dmr"]
Oct 30 20:59:46.190309 ip-10-0-128-165 kubenswrapper[5713]: W1030 20:59:46.190277    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90aaa4a2_48e3_4baa_bdc9_27b7c308ec89.slice/crio-96ea8989daf5a0e1744669578dd2f51179ea6a672f472a192e5115aeadb8c2e6 WatchSource:0}: Error finding container 96ea8989daf5a0e1744669578dd2f51179ea6a672f472a192e5115aeadb8c2e6: Status 404 returned error can't find the container with id 96ea8989daf5a0e1744669578dd2f51179ea6a672f472a192e5115aeadb8c2e6
Oct 30 20:59:46.666350 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:46.666246    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-sjpxz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:59:46.666350 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:46.666303    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-sjpxz" podUID="f543e9a0-a5a6-417a-8f00-b0627a5c1776" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:59:47.049098 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:47.049039    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5fd58f-j4dmr" event={"ID":"90aaa4a2-48e3-4baa-bdc9-27b7c308ec89","Type":"ContainerStarted","Data":"4c1464ea69aee92331245ff41b7a3e4efc3b5be5d7afc28e580a22a2a58344a6"}
Oct 30 20:59:47.049276 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:47.049126    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5fd58f-j4dmr" event={"ID":"90aaa4a2-48e3-4baa-bdc9-27b7c308ec89","Type":"ContainerStarted","Data":"96ea8989daf5a0e1744669578dd2f51179ea6a672f472a192e5115aeadb8c2e6"}
Oct 30 20:59:47.064625 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:47.064594    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="922e26e1-9fc8-4e31-9897-1c80dcfe2cb9" path="/var/lib/kubelet/pods/922e26e1-9fc8-4e31-9897-1c80dcfe2cb9/volumes"
Oct 30 20:59:47.171472 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:47.171420    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5fd58f-j4dmr" podStartSLOduration=2.171404892 podStartE2EDuration="2.171404892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:59:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:59:47.168690656 +0000 UTC m=+1270.233467054" watchObservedRunningTime="2026-10-30 20:59:47.171404892 +0000 UTC m=+1270.236181290"
Oct 30 20:59:47.405609 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:47.405580    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-sjpxz"
Oct 30 20:59:47.530162 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:47.526207    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f543e9a0-a5a6-417a-8f00-b0627a5c1776-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f543e9a0-a5a6-417a-8f00-b0627a5c1776-istiod-ca-cert\") pod \"f543e9a0-a5a6-417a-8f00-b0627a5c1776\" (UID: \"f543e9a0-a5a6-417a-8f00-b0627a5c1776\") "
Oct 30 20:59:47.530162 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:47.526270    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f543e9a0-a5a6-417a-8f00-b0627a5c1776-istio-token\" (UniqueName: \"kubernetes.io/projected/f543e9a0-a5a6-417a-8f00-b0627a5c1776-istio-token\") pod \"f543e9a0-a5a6-417a-8f00-b0627a5c1776\" (UID: \"f543e9a0-a5a6-417a-8f00-b0627a5c1776\") "
Oct 30 20:59:47.530162 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:47.526306    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f543e9a0-a5a6-417a-8f00-b0627a5c1776-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f543e9a0-a5a6-417a-8f00-b0627a5c1776-credential-socket\") pod \"f543e9a0-a5a6-417a-8f00-b0627a5c1776\" (UID: \"f543e9a0-a5a6-417a-8f00-b0627a5c1776\") "
Oct 30 20:59:47.530162 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:47.526333    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f543e9a0-a5a6-417a-8f00-b0627a5c1776-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f543e9a0-a5a6-417a-8f00-b0627a5c1776-workload-socket\") pod \"f543e9a0-a5a6-417a-8f00-b0627a5c1776\" (UID: \"f543e9a0-a5a6-417a-8f00-b0627a5c1776\") "
Oct 30 20:59:47.530162 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:47.526379    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f543e9a0-a5a6-417a-8f00-b0627a5c1776-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f543e9a0-a5a6-417a-8f00-b0627a5c1776-istio-data\") pod \"f543e9a0-a5a6-417a-8f00-b0627a5c1776\" (UID: \"f543e9a0-a5a6-417a-8f00-b0627a5c1776\") "
Oct 30 20:59:47.530162 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:47.526429    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f543e9a0-a5a6-417a-8f00-b0627a5c1776-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f543e9a0-a5a6-417a-8f00-b0627a5c1776-workload-certs\") pod \"f543e9a0-a5a6-417a-8f00-b0627a5c1776\" (UID: \"f543e9a0-a5a6-417a-8f00-b0627a5c1776\") "
Oct 30 20:59:47.530162 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:47.526483    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f543e9a0-a5a6-417a-8f00-b0627a5c1776-kube-api-access-4ddbn\" (UniqueName: \"kubernetes.io/projected/f543e9a0-a5a6-417a-8f00-b0627a5c1776-kube-api-access-4ddbn\") pod \"f543e9a0-a5a6-417a-8f00-b0627a5c1776\" (UID: \"f543e9a0-a5a6-417a-8f00-b0627a5c1776\") "
Oct 30 20:59:47.530162 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:47.526609    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f543e9a0-a5a6-417a-8f00-b0627a5c1776-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f543e9a0-a5a6-417a-8f00-b0627a5c1776-istio-podinfo\") pod \"f543e9a0-a5a6-417a-8f00-b0627a5c1776\" (UID: \"f543e9a0-a5a6-417a-8f00-b0627a5c1776\") "
Oct 30 20:59:47.530162 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:47.526667    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f543e9a0-a5a6-417a-8f00-b0627a5c1776-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f543e9a0-a5a6-417a-8f00-b0627a5c1776-istio-envoy\") pod \"f543e9a0-a5a6-417a-8f00-b0627a5c1776\" (UID: \"f543e9a0-a5a6-417a-8f00-b0627a5c1776\") "
Oct 30 20:59:47.530162 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:47.526600    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f543e9a0-a5a6-417a-8f00-b0627a5c1776-istiod-ca-cert" pod "f543e9a0-a5a6-417a-8f00-b0627a5c1776" (UID: "f543e9a0-a5a6-417a-8f00-b0627a5c1776"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:59:47.530162 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:47.527505    5713 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f543e9a0-a5a6-417a-8f00-b0627a5c1776-istiod-ca-cert\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:47.530162 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:47.526884    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f543e9a0-a5a6-417a-8f00-b0627a5c1776-workload-certs" pod "f543e9a0-a5a6-417a-8f00-b0627a5c1776" (UID: "f543e9a0-a5a6-417a-8f00-b0627a5c1776"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:59:47.530162 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:47.527060    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f543e9a0-a5a6-417a-8f00-b0627a5c1776-istio-data" pod "f543e9a0-a5a6-417a-8f00-b0627a5c1776" (UID: "f543e9a0-a5a6-417a-8f00-b0627a5c1776"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:59:47.530162 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:47.527593    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f543e9a0-a5a6-417a-8f00-b0627a5c1776-credential-socket" pod "f543e9a0-a5a6-417a-8f00-b0627a5c1776" (UID: "f543e9a0-a5a6-417a-8f00-b0627a5c1776"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:59:47.530162 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:47.528269    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f543e9a0-a5a6-417a-8f00-b0627a5c1776-workload-socket" pod "f543e9a0-a5a6-417a-8f00-b0627a5c1776" (UID: "f543e9a0-a5a6-417a-8f00-b0627a5c1776"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:59:47.530162 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:47.529732    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f543e9a0-a5a6-417a-8f00-b0627a5c1776-istio-envoy" pod "f543e9a0-a5a6-417a-8f00-b0627a5c1776" (UID: "f543e9a0-a5a6-417a-8f00-b0627a5c1776"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:59:47.531053 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:47.530421    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f543e9a0-a5a6-417a-8f00-b0627a5c1776-istio-podinfo" pod "f543e9a0-a5a6-417a-8f00-b0627a5c1776" (UID: "f543e9a0-a5a6-417a-8f00-b0627a5c1776"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:59:47.531053 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:47.531001    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f543e9a0-a5a6-417a-8f00-b0627a5c1776-istio-token" pod "f543e9a0-a5a6-417a-8f00-b0627a5c1776" (UID: "f543e9a0-a5a6-417a-8f00-b0627a5c1776"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:59:47.532213 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:47.532153    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f543e9a0-a5a6-417a-8f00-b0627a5c1776-kube-api-access-4ddbn" pod "f543e9a0-a5a6-417a-8f00-b0627a5c1776" (UID: "f543e9a0-a5a6-417a-8f00-b0627a5c1776"). InnerVolumeSpecName "kube-api-access-4ddbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:59:47.628925 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:47.628843    5713 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f543e9a0-a5a6-417a-8f00-b0627a5c1776-istio-podinfo\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:47.628925 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:47.628873    5713 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f543e9a0-a5a6-417a-8f00-b0627a5c1776-istio-envoy\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:47.628925 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:47.628884    5713 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f543e9a0-a5a6-417a-8f00-b0627a5c1776-istio-token\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:47.628925 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:47.628894    5713 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f543e9a0-a5a6-417a-8f00-b0627a5c1776-credential-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:47.628925 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:47.628904    5713 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f543e9a0-a5a6-417a-8f00-b0627a5c1776-workload-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:47.628925 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:47.628912    5713 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f543e9a0-a5a6-417a-8f00-b0627a5c1776-istio-data\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:47.628925 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:47.628923    5713 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f543e9a0-a5a6-417a-8f00-b0627a5c1776-workload-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:47.629324 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:47.628936    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ddbn\" (UniqueName: \"kubernetes.io/projected/f543e9a0-a5a6-417a-8f00-b0627a5c1776-kube-api-access-4ddbn\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:48.059276 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:48.059243    5713 generic.go:356] "Generic (PLEG): container finished" podID="90aaa4a2-48e3-4baa-bdc9-27b7c308ec89" containerID="4c1464ea69aee92331245ff41b7a3e4efc3b5be5d7afc28e580a22a2a58344a6" exitCode=0
Oct 30 20:59:48.059881 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:48.059328    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5fd58f-j4dmr" event={"ID":"90aaa4a2-48e3-4baa-bdc9-27b7c308ec89","Type":"ContainerDied","Data":"4c1464ea69aee92331245ff41b7a3e4efc3b5be5d7afc28e580a22a2a58344a6"}
Oct 30 20:59:48.061034 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:48.061009    5713 generic.go:356] "Generic (PLEG): container finished" podID="f543e9a0-a5a6-417a-8f00-b0627a5c1776" containerID="639cde1a6ec57812a7185a4df695ef7d15ae78553fa1c7389e2c179598726ec7" exitCode=0
Oct 30 20:59:48.061146 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:48.061099    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-sjpxz"
Oct 30 20:59:48.061205 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:48.061096    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-sjpxz" event={"ID":"f543e9a0-a5a6-417a-8f00-b0627a5c1776","Type":"ContainerDied","Data":"639cde1a6ec57812a7185a4df695ef7d15ae78553fa1c7389e2c179598726ec7"}
Oct 30 20:59:48.061267 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:48.061222    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-sjpxz" event={"ID":"f543e9a0-a5a6-417a-8f00-b0627a5c1776","Type":"ContainerDied","Data":"0d6a015e059c74dd84dc452a6009893c90eb22deb024b97e409784d8ca459e58"}
Oct 30 20:59:48.061267 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:48.061258    5713 scope.go:122] "RemoveContainer" containerID="639cde1a6ec57812a7185a4df695ef7d15ae78553fa1c7389e2c179598726ec7"
Oct 30 20:59:48.072654 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:48.072633    5713 scope.go:122] "RemoveContainer" containerID="639cde1a6ec57812a7185a4df695ef7d15ae78553fa1c7389e2c179598726ec7"
Oct 30 20:59:48.072999 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:59:48.072980    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"639cde1a6ec57812a7185a4df695ef7d15ae78553fa1c7389e2c179598726ec7\": container with ID starting with 639cde1a6ec57812a7185a4df695ef7d15ae78553fa1c7389e2c179598726ec7 not found: ID does not exist" containerID="639cde1a6ec57812a7185a4df695ef7d15ae78553fa1c7389e2c179598726ec7"
Oct 30 20:59:48.073082 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:48.073009    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"639cde1a6ec57812a7185a4df695ef7d15ae78553fa1c7389e2c179598726ec7"} err="failed to get container status \"639cde1a6ec57812a7185a4df695ef7d15ae78553fa1c7389e2c179598726ec7\": rpc error: code = NotFound desc = could not find container \"639cde1a6ec57812a7185a4df695ef7d15ae78553fa1c7389e2c179598726ec7\": container with ID starting with 639cde1a6ec57812a7185a4df695ef7d15ae78553fa1c7389e2c179598726ec7 not found: ID does not exist"
Oct 30 20:59:48.287722 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:48.287681    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-sjpxz"]
Oct 30 20:59:48.337729 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:48.337639    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-sjpxz"]
Oct 30 20:59:49.069006 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:49.068973    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f543e9a0-a5a6-417a-8f00-b0627a5c1776" path="/var/lib/kubelet/pods/f543e9a0-a5a6-417a-8f00-b0627a5c1776/volumes"
Oct 30 20:59:49.232571 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:49.232539    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5fd58f-j4dmr"
Oct 30 20:59:49.352112 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:49.352040    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90aaa4a2-48e3-4baa-bdc9-27b7c308ec89-kube-api-access-xrvv2\" (UniqueName: \"kubernetes.io/projected/90aaa4a2-48e3-4baa-bdc9-27b7c308ec89-kube-api-access-xrvv2\") pod \"90aaa4a2-48e3-4baa-bdc9-27b7c308ec89\" (UID: \"90aaa4a2-48e3-4baa-bdc9-27b7c308ec89\") "
Oct 30 20:59:49.352285 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:49.352169    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90aaa4a2-48e3-4baa-bdc9-27b7c308ec89-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90aaa4a2-48e3-4baa-bdc9-27b7c308ec89-maas-api-xks-ca\") pod \"90aaa4a2-48e3-4baa-bdc9-27b7c308ec89\" (UID: \"90aaa4a2-48e3-4baa-bdc9-27b7c308ec89\") "
Oct 30 20:59:49.352285 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:49.352240    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90aaa4a2-48e3-4baa-bdc9-27b7c308ec89-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90aaa4a2-48e3-4baa-bdc9-27b7c308ec89-maas-api-openshift-ca\") pod \"90aaa4a2-48e3-4baa-bdc9-27b7c308ec89\" (UID: \"90aaa4a2-48e3-4baa-bdc9-27b7c308ec89\") "
Oct 30 20:59:49.352633 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:49.352607    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90aaa4a2-48e3-4baa-bdc9-27b7c308ec89-maas-api-openshift-ca" pod "90aaa4a2-48e3-4baa-bdc9-27b7c308ec89" (UID: "90aaa4a2-48e3-4baa-bdc9-27b7c308ec89"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:59:49.352820 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:49.352802    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90aaa4a2-48e3-4baa-bdc9-27b7c308ec89-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:49.354374 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:49.354351    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90aaa4a2-48e3-4baa-bdc9-27b7c308ec89-maas-api-xks-ca" pod "90aaa4a2-48e3-4baa-bdc9-27b7c308ec89" (UID: "90aaa4a2-48e3-4baa-bdc9-27b7c308ec89"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:59:49.354636 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:49.354620    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90aaa4a2-48e3-4baa-bdc9-27b7c308ec89-kube-api-access-xrvv2" pod "90aaa4a2-48e3-4baa-bdc9-27b7c308ec89" (UID: "90aaa4a2-48e3-4baa-bdc9-27b7c308ec89"). InnerVolumeSpecName "kube-api-access-xrvv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:59:49.453771 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:49.453740    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrvv2\" (UniqueName: \"kubernetes.io/projected/90aaa4a2-48e3-4baa-bdc9-27b7c308ec89-kube-api-access-xrvv2\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:49.453771 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:49.453768    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90aaa4a2-48e3-4baa-bdc9-27b7c308ec89-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:49.840057 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:49.840026    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-824fa8f4-2jt6r"]
Oct 30 20:59:49.840618 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:49.840604    5713 cpu_manager.go:414] "containerMap: removing container" podUID="90aaa4a2-48e3-4baa-bdc9-27b7c308ec89" containerName="revoke-keys"
Oct 30 20:59:49.840667 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:49.840621    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90aaa4a2-48e3-4baa-bdc9-27b7c308ec89" containerName="revoke-keys"
Oct 30 20:59:49.840667 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:49.840647    5713 cpu_manager.go:414] "containerMap: removing container" podUID="f543e9a0-a5a6-417a-8f00-b0627a5c1776" containerName="istio-proxy"
Oct 30 20:59:49.840667 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:49.840652    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f543e9a0-a5a6-417a-8f00-b0627a5c1776" containerName="istio-proxy"
Oct 30 20:59:49.840765 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:49.840740    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="f543e9a0-a5a6-417a-8f00-b0627a5c1776" containerName="istio-proxy"
Oct 30 20:59:49.840765 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:49.840757    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="90aaa4a2-48e3-4baa-bdc9-27b7c308ec89" containerName="revoke-keys"
Oct 30 20:59:49.844432 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:49.844409    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-824fa8f4-2jt6r"
Oct 30 20:59:49.900517 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:49.900486    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-824fa8f4-2jt6r"]
Oct 30 20:59:49.960363 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:49.960330    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h85h\" (UniqueName: \"kubernetes.io/projected/2948b895-a1c8-426c-ba07-751bff16e075-kube-api-access-9h85h\") pod \"maas-api-revoke-sub-824fa8f4-2jt6r\" (UID: \"2948b895-a1c8-426c-ba07-751bff16e075\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-824fa8f4-2jt6r"
Oct 30 20:59:49.960568 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:49.960375    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2948b895-a1c8-426c-ba07-751bff16e075-maas-api-xks-ca\") pod \"maas-api-revoke-sub-824fa8f4-2jt6r\" (UID: \"2948b895-a1c8-426c-ba07-751bff16e075\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-824fa8f4-2jt6r"
Oct 30 20:59:49.960638 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:49.960563    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2948b895-a1c8-426c-ba07-751bff16e075-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-824fa8f4-2jt6r\" (UID: \"2948b895-a1c8-426c-ba07-751bff16e075\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-824fa8f4-2jt6r"
Oct 30 20:59:50.062339 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:50.062307    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h85h\" (UniqueName: \"kubernetes.io/projected/2948b895-a1c8-426c-ba07-751bff16e075-kube-api-access-9h85h\") pod \"maas-api-revoke-sub-824fa8f4-2jt6r\" (UID: \"2948b895-a1c8-426c-ba07-751bff16e075\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-824fa8f4-2jt6r"
Oct 30 20:59:50.062493 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:50.062350    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2948b895-a1c8-426c-ba07-751bff16e075-maas-api-xks-ca\") pod \"maas-api-revoke-sub-824fa8f4-2jt6r\" (UID: \"2948b895-a1c8-426c-ba07-751bff16e075\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-824fa8f4-2jt6r"
Oct 30 20:59:50.062493 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:50.062428    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2948b895-a1c8-426c-ba07-751bff16e075-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-824fa8f4-2jt6r\" (UID: \"2948b895-a1c8-426c-ba07-751bff16e075\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-824fa8f4-2jt6r"
Oct 30 20:59:50.063140 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:50.063110    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2948b895-a1c8-426c-ba07-751bff16e075-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-824fa8f4-2jt6r\" (UID: \"2948b895-a1c8-426c-ba07-751bff16e075\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-824fa8f4-2jt6r"
Oct 30 20:59:50.065182 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:50.065162    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2948b895-a1c8-426c-ba07-751bff16e075-maas-api-xks-ca\") pod \"maas-api-revoke-sub-824fa8f4-2jt6r\" (UID: \"2948b895-a1c8-426c-ba07-751bff16e075\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-824fa8f4-2jt6r"
Oct 30 20:59:50.084557 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:50.084532    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5fd58f-j4dmr"
Oct 30 20:59:50.084953 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:50.084556    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5fd58f-j4dmr" event={"ID":"90aaa4a2-48e3-4baa-bdc9-27b7c308ec89","Type":"ContainerDied","Data":"96ea8989daf5a0e1744669578dd2f51179ea6a672f472a192e5115aeadb8c2e6"}
Oct 30 20:59:50.084953 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:50.084604    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96ea8989daf5a0e1744669578dd2f51179ea6a672f472a192e5115aeadb8c2e6"
Oct 30 20:59:50.120551 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:50.120524    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h85h\" (UniqueName: \"kubernetes.io/projected/2948b895-a1c8-426c-ba07-751bff16e075-kube-api-access-9h85h\") pod \"maas-api-revoke-sub-824fa8f4-2jt6r\" (UID: \"2948b895-a1c8-426c-ba07-751bff16e075\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-824fa8f4-2jt6r"
Oct 30 20:59:50.159993 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:50.159967    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-824fa8f4-2jt6r"
Oct 30 20:59:50.547214 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:50.547175    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-824fa8f4-2jt6r"]
Oct 30 20:59:51.095166 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:51.095076    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-824fa8f4-2jt6r" event={"ID":"2948b895-a1c8-426c-ba07-751bff16e075","Type":"ContainerStarted","Data":"99bad37d32417706ad98f34bdd2bf0e71af26d86da4b38608b00d3988fbaf48a"}
Oct 30 20:59:51.095166 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:51.095140    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-824fa8f4-2jt6r" event={"ID":"2948b895-a1c8-426c-ba07-751bff16e075","Type":"ContainerStarted","Data":"0854a89dc6892b93cd95ed4239502520ace4263418724648b2c11a653e3f8d73"}
Oct 30 20:59:51.212608 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:51.212549    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-824fa8f4-2jt6r" podStartSLOduration=2.212536559 podStartE2EDuration="2.212536559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:59:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:59:51.210524895 +0000 UTC m=+1274.275301294" watchObservedRunningTime="2026-10-30 20:59:51.212536559 +0000 UTC m=+1274.277312957"
Oct 30 20:59:52.104453 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:52.104422    5713 generic.go:356] "Generic (PLEG): container finished" podID="2948b895-a1c8-426c-ba07-751bff16e075" containerID="99bad37d32417706ad98f34bdd2bf0e71af26d86da4b38608b00d3988fbaf48a" exitCode=0
Oct 30 20:59:52.104915 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:52.104511    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-824fa8f4-2jt6r" event={"ID":"2948b895-a1c8-426c-ba07-751bff16e075","Type":"ContainerDied","Data":"99bad37d32417706ad98f34bdd2bf0e71af26d86da4b38608b00d3988fbaf48a"}
Oct 30 20:59:53.277447 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:53.277421    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-824fa8f4-2jt6r"
Oct 30 20:59:53.403445 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:53.403365    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2948b895-a1c8-426c-ba07-751bff16e075-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2948b895-a1c8-426c-ba07-751bff16e075-maas-api-xks-ca\") pod \"2948b895-a1c8-426c-ba07-751bff16e075\" (UID: \"2948b895-a1c8-426c-ba07-751bff16e075\") "
Oct 30 20:59:53.403445 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:53.403418    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2948b895-a1c8-426c-ba07-751bff16e075-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2948b895-a1c8-426c-ba07-751bff16e075-maas-api-openshift-ca\") pod \"2948b895-a1c8-426c-ba07-751bff16e075\" (UID: \"2948b895-a1c8-426c-ba07-751bff16e075\") "
Oct 30 20:59:53.403739 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:53.403719    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2948b895-a1c8-426c-ba07-751bff16e075-kube-api-access-9h85h\" (UniqueName: \"kubernetes.io/projected/2948b895-a1c8-426c-ba07-751bff16e075-kube-api-access-9h85h\") pod \"2948b895-a1c8-426c-ba07-751bff16e075\" (UID: \"2948b895-a1c8-426c-ba07-751bff16e075\") "
Oct 30 20:59:53.403894 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:53.403861    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2948b895-a1c8-426c-ba07-751bff16e075-maas-api-openshift-ca" pod "2948b895-a1c8-426c-ba07-751bff16e075" (UID: "2948b895-a1c8-426c-ba07-751bff16e075"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:59:53.404345 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:53.404320    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2948b895-a1c8-426c-ba07-751bff16e075-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:53.405902 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:53.405870    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2948b895-a1c8-426c-ba07-751bff16e075-maas-api-xks-ca" pod "2948b895-a1c8-426c-ba07-751bff16e075" (UID: "2948b895-a1c8-426c-ba07-751bff16e075"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:59:53.406408 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:53.406381    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2948b895-a1c8-426c-ba07-751bff16e075-kube-api-access-9h85h" pod "2948b895-a1c8-426c-ba07-751bff16e075" (UID: "2948b895-a1c8-426c-ba07-751bff16e075"). InnerVolumeSpecName "kube-api-access-9h85h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:59:53.506171 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:53.506137    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9h85h\" (UniqueName: \"kubernetes.io/projected/2948b895-a1c8-426c-ba07-751bff16e075-kube-api-access-9h85h\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:53.506171 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:53.506168    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2948b895-a1c8-426c-ba07-751bff16e075-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:59:54.122957 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:54.122926    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-824fa8f4-2jt6r"
Oct 30 20:59:54.123146 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:54.122971    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-824fa8f4-2jt6r" event={"ID":"2948b895-a1c8-426c-ba07-751bff16e075","Type":"ContainerDied","Data":"0854a89dc6892b93cd95ed4239502520ace4263418724648b2c11a653e3f8d73"}
Oct 30 20:59:54.123146 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:54.123025    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0854a89dc6892b93cd95ed4239502520ace4263418724648b2c11a653e3f8d73"
Oct 30 20:59:54.459719 ip-10-0-128-165 kubenswrapper[5713]: E1030 20:59:54.459684    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 20:59:58.047367 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:58.047336    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69129130-79t4h"]
Oct 30 20:59:58.096988 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:58.096959    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69129130-79t4h"]
Oct 30 20:59:59.064319 ip-10-0-128-165 kubenswrapper[5713]: I1030 20:59:59.064284    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30683061-8eeb-4ade-bb56-23c3aac52094" path="/var/lib/kubelet/pods/30683061-8eeb-4ade-bb56-23c3aac52094/volumes"
Oct 30 21:00:00.180352 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:00.180317    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d061d0-29889900-sp86r"]
Oct 30 21:00:00.180927 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:00.180912    5713 cpu_manager.go:414] "containerMap: removing container" podUID="2948b895-a1c8-426c-ba07-751bff16e075" containerName="revoke-keys"
Oct 30 21:00:00.180977 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:00.180927    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2948b895-a1c8-426c-ba07-751bff16e075" containerName="revoke-keys"
Oct 30 21:00:00.181045 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:00.181035    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="2948b895-a1c8-426c-ba07-751bff16e075" containerName="revoke-keys"
Oct 30 21:00:00.184685 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:00.184668    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-781058-29889900-tfdx9"]
Oct 30 21:00:00.184831 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:00.184814    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d061d0-29889900-sp86r"
Oct 30 21:00:00.188657 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:00.188638    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-781058-29889900-tfdx9"
Oct 30 21:00:00.192869 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:00.192669    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b425d-29889900-sz9cc"]
Oct 30 21:00:00.197515 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:00.197498    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b425d-29889900-sz9cc"
Oct 30 21:00:00.238654 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:00.238625    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889900-dtz4k"]
Oct 30 21:00:00.258689 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:00.258635    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889900-hm4sk"]
Oct 30 21:00:00.258856 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:00.258768    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889900-dtz4k"
Oct 30 21:00:00.265374 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:00.265346    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d061d0-29889900-sp86r"]
Oct 30 21:00:00.265509 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:00.265477    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889900-hm4sk"
Oct 30 21:00:00.284173 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:00.284143    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:00:00.288272 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:00.288248    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:00:00.288418 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:00.288397    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:00:00.288819 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:00.288764    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-781058-29889900-tfdx9"]
Oct 30 21:00:00.295968 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:00.295305    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b425d-29889900-sz9cc"]
Oct 30 21:00:00.296917 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:00.296885    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq7hr\" (UniqueName: \"kubernetes.io/projected/10eec293-a840-4727-ade8-d2afa45e9f9a-kube-api-access-xq7hr\") pod \"maas-api-key-cleanup-e2e-worker-w0-781058-29889900-tfdx9\" (UID: \"10eec293-a840-4727-ade8-d2afa45e9f9a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-781058-29889900-tfdx9"
Oct 30 21:00:00.297031 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:00.296949    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtssh\" (UniqueName: \"kubernetes.io/projected/a8371114-ea5d-4cbc-b761-5aa5cdfef8fd-kube-api-access-rtssh\") pod \"maas-api-key-cleanup-e2e-worker-w2-d061d0-29889900-sp86r\" (UID: \"a8371114-ea5d-4cbc-b761-5aa5cdfef8fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d061d0-29889900-sp86r"
Oct 30 21:00:00.297031 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:00.297000    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hx59\" (UniqueName: \"kubernetes.io/projected/c0df799c-ee0f-45c0-bbe2-c7928deb0e3b-kube-api-access-8hx59\") pod \"maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889900-hm4sk\" (UID: \"c0df799c-ee0f-45c0-bbe2-c7928deb0e3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889900-hm4sk"
Oct 30 21:00:00.297133 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:00.297049    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8371114-ea5d-4cbc-b761-5aa5cdfef8fd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-d061d0-29889900-sp86r\" (UID: \"a8371114-ea5d-4cbc-b761-5aa5cdfef8fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d061d0-29889900-sp86r"
Oct 30 21:00:00.297182 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:00.297138    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0df799c-ee0f-45c0-bbe2-c7928deb0e3b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889900-hm4sk\" (UID: \"c0df799c-ee0f-45c0-bbe2-c7928deb0e3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889900-hm4sk"
Oct 30 21:00:00.297231 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:00.297190    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/280ab52d-2d8d-4fe7-b293-73f470d3cd28-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889900-dtz4k\" (UID: \"280ab52d-2d8d-4fe7-b293-73f470d3cd28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889900-dtz4k"
Oct 30 21:00:00.297231 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:00.297221    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv6wb\" (UniqueName: \"kubernetes.io/projected/76aed203-0d7c-410e-84bd-637f57a1620a-kube-api-access-gv6wb\") pod \"maas-api-key-cleanup-e2e-worker-w5-2b425d-29889900-sz9cc\" (UID: \"76aed203-0d7c-410e-84bd-637f57a1620a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b425d-29889900-sz9cc"
Oct 30 21:00:00.297323 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:00.297261    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76aed203-0d7c-410e-84bd-637f57a1620a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-2b425d-29889900-sz9cc\" (UID: \"76aed203-0d7c-410e-84bd-637f57a1620a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b425d-29889900-sz9cc"
Oct 30 21:00:00.297323 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:00.297300    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10eec293-a840-4727-ade8-d2afa45e9f9a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-781058-29889900-tfdx9\" (UID: \"10eec293-a840-4727-ade8-d2afa45e9f9a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-781058-29889900-tfdx9"
Oct 30 21:00:00.297439 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:00.297420    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9ljw\" (UniqueName: \"kubernetes.io/projected/280ab52d-2d8d-4fe7-b293-73f470d3cd28-kube-api-access-t9ljw\") pod \"maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889900-dtz4k\" (UID: \"280ab52d-2d8d-4fe7-b293-73f470d3cd28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889900-dtz4k"
Oct 30 21:00:00.302912 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:00.301003    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889900-dtz4k"]
Oct 30 21:00:00.346513 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:00.346474    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889900-hm4sk"]
Oct 30 21:00:00.398410 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:00.398364    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9ljw\" (UniqueName: \"kubernetes.io/projected/280ab52d-2d8d-4fe7-b293-73f470d3cd28-kube-api-access-t9ljw\") pod \"maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889900-dtz4k\" (UID: \"280ab52d-2d8d-4fe7-b293-73f470d3cd28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889900-dtz4k"
Oct 30 21:00:00.398410 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:00.398421    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq7hr\" (UniqueName: \"kubernetes.io/projected/10eec293-a840-4727-ade8-d2afa45e9f9a-kube-api-access-xq7hr\") pod \"maas-api-key-cleanup-e2e-worker-w0-781058-29889900-tfdx9\" (UID: \"10eec293-a840-4727-ade8-d2afa45e9f9a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-781058-29889900-tfdx9"
Oct 30 21:00:00.398636 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:00.398450    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtssh\" (UniqueName: \"kubernetes.io/projected/a8371114-ea5d-4cbc-b761-5aa5cdfef8fd-kube-api-access-rtssh\") pod \"maas-api-key-cleanup-e2e-worker-w2-d061d0-29889900-sp86r\" (UID: \"a8371114-ea5d-4cbc-b761-5aa5cdfef8fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d061d0-29889900-sp86r"
Oct 30 21:00:00.398636 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:00.398477    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hx59\" (UniqueName: \"kubernetes.io/projected/c0df799c-ee0f-45c0-bbe2-c7928deb0e3b-kube-api-access-8hx59\") pod \"maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889900-hm4sk\" (UID: \"c0df799c-ee0f-45c0-bbe2-c7928deb0e3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889900-hm4sk"
Oct 30 21:00:00.398636 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:00.398508    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8371114-ea5d-4cbc-b761-5aa5cdfef8fd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-d061d0-29889900-sp86r\" (UID: \"a8371114-ea5d-4cbc-b761-5aa5cdfef8fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d061d0-29889900-sp86r"
Oct 30 21:00:00.398636 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:00.398579    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0df799c-ee0f-45c0-bbe2-c7928deb0e3b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889900-hm4sk\" (UID: \"c0df799c-ee0f-45c0-bbe2-c7928deb0e3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889900-hm4sk"
Oct 30 21:00:00.398636 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:00.398628    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/280ab52d-2d8d-4fe7-b293-73f470d3cd28-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889900-dtz4k\" (UID: \"280ab52d-2d8d-4fe7-b293-73f470d3cd28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889900-dtz4k"
Oct 30 21:00:00.398890 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:00.398659    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv6wb\" (UniqueName: \"kubernetes.io/projected/76aed203-0d7c-410e-84bd-637f57a1620a-kube-api-access-gv6wb\") pod \"maas-api-key-cleanup-e2e-worker-w5-2b425d-29889900-sz9cc\" (UID: \"76aed203-0d7c-410e-84bd-637f57a1620a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b425d-29889900-sz9cc"
Oct 30 21:00:00.398890 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:00.398699    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76aed203-0d7c-410e-84bd-637f57a1620a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-2b425d-29889900-sz9cc\" (UID: \"76aed203-0d7c-410e-84bd-637f57a1620a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b425d-29889900-sz9cc"
Oct 30 21:00:00.398890 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:00.398741    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10eec293-a840-4727-ade8-d2afa45e9f9a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-781058-29889900-tfdx9\" (UID: \"10eec293-a840-4727-ade8-d2afa45e9f9a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-781058-29889900-tfdx9"
Oct 30 21:00:00.399418 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:00.399399    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8371114-ea5d-4cbc-b761-5aa5cdfef8fd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-d061d0-29889900-sp86r\" (UID: \"a8371114-ea5d-4cbc-b761-5aa5cdfef8fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d061d0-29889900-sp86r"
Oct 30 21:00:00.399494 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:00.399465    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/280ab52d-2d8d-4fe7-b293-73f470d3cd28-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889900-dtz4k\" (UID: \"280ab52d-2d8d-4fe7-b293-73f470d3cd28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889900-dtz4k"
Oct 30 21:00:00.399603 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:00.399469    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0df799c-ee0f-45c0-bbe2-c7928deb0e3b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889900-hm4sk\" (UID: \"c0df799c-ee0f-45c0-bbe2-c7928deb0e3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889900-hm4sk"
Oct 30 21:00:00.399603 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:00.399570    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10eec293-a840-4727-ade8-d2afa45e9f9a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-781058-29889900-tfdx9\" (UID: \"10eec293-a840-4727-ade8-d2afa45e9f9a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-781058-29889900-tfdx9"
Oct 30 21:00:00.399673 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:00.399632    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76aed203-0d7c-410e-84bd-637f57a1620a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-2b425d-29889900-sz9cc\" (UID: \"76aed203-0d7c-410e-84bd-637f57a1620a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b425d-29889900-sz9cc"
Oct 30 21:00:00.459375 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:00.459304    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv6wb\" (UniqueName: \"kubernetes.io/projected/76aed203-0d7c-410e-84bd-637f57a1620a-kube-api-access-gv6wb\") pod \"maas-api-key-cleanup-e2e-worker-w5-2b425d-29889900-sz9cc\" (UID: \"76aed203-0d7c-410e-84bd-637f57a1620a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b425d-29889900-sz9cc"
Oct 30 21:00:00.459501 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:00.459374    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq7hr\" (UniqueName: \"kubernetes.io/projected/10eec293-a840-4727-ade8-d2afa45e9f9a-kube-api-access-xq7hr\") pod \"maas-api-key-cleanup-e2e-worker-w0-781058-29889900-tfdx9\" (UID: \"10eec293-a840-4727-ade8-d2afa45e9f9a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-781058-29889900-tfdx9"
Oct 30 21:00:00.459501 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:00.459471    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9ljw\" (UniqueName: \"kubernetes.io/projected/280ab52d-2d8d-4fe7-b293-73f470d3cd28-kube-api-access-t9ljw\") pod \"maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889900-dtz4k\" (UID: \"280ab52d-2d8d-4fe7-b293-73f470d3cd28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889900-dtz4k"
Oct 30 21:00:00.459586 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:00.459479    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hx59\" (UniqueName: \"kubernetes.io/projected/c0df799c-ee0f-45c0-bbe2-c7928deb0e3b-kube-api-access-8hx59\") pod \"maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889900-hm4sk\" (UID: \"c0df799c-ee0f-45c0-bbe2-c7928deb0e3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889900-hm4sk"
Oct 30 21:00:00.461310 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:00.461291    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtssh\" (UniqueName: \"kubernetes.io/projected/a8371114-ea5d-4cbc-b761-5aa5cdfef8fd-kube-api-access-rtssh\") pod \"maas-api-key-cleanup-e2e-worker-w2-d061d0-29889900-sp86r\" (UID: \"a8371114-ea5d-4cbc-b761-5aa5cdfef8fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d061d0-29889900-sp86r"
Oct 30 21:00:00.500759 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:00.500719    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d061d0-29889900-sp86r"
Oct 30 21:00:00.510402 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:00.510380    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-781058-29889900-tfdx9"
Oct 30 21:00:00.517422 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:00.517399    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b425d-29889900-sz9cc"
Oct 30 21:00:00.577534 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:00.577326    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889900-dtz4k"
Oct 30 21:00:00.608115 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:00.589052    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889900-hm4sk"
Oct 30 21:00:00.897842 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:00.897814    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d061d0-29889900-sp86r"]
Oct 30 21:00:00.898836 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:00:00.898806    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8371114_ea5d_4cbc_b761_5aa5cdfef8fd.slice/crio-a77542ff4bb79aae3c9b74693038ce0be22eaa4cfb5ac08d946ea9e0cb6b8a9b WatchSource:0}: Error finding container a77542ff4bb79aae3c9b74693038ce0be22eaa4cfb5ac08d946ea9e0cb6b8a9b: Status 404 returned error can't find the container with id a77542ff4bb79aae3c9b74693038ce0be22eaa4cfb5ac08d946ea9e0cb6b8a9b
Oct 30 21:00:00.948766 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:00.948729    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-781058-29889900-tfdx9"]
Oct 30 21:00:00.952875 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:00:00.952845    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10eec293_a840_4727_ade8_d2afa45e9f9a.slice/crio-2d0bb4ea9bf183f21dba0c404d85b319f154d4bf6f2f5581bbeff6986018c4e7 WatchSource:0}: Error finding container 2d0bb4ea9bf183f21dba0c404d85b319f154d4bf6f2f5581bbeff6986018c4e7: Status 404 returned error can't find the container with id 2d0bb4ea9bf183f21dba0c404d85b319f154d4bf6f2f5581bbeff6986018c4e7
Oct 30 21:00:01.001450 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:00:01.001390    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76aed203_0d7c_410e_84bd_637f57a1620a.slice/crio-538733c7ac7b7c6b3919115f3ab35a0cea0464321e8588af565db8f87f5447f0 WatchSource:0}: Error finding container 538733c7ac7b7c6b3919115f3ab35a0cea0464321e8588af565db8f87f5447f0: Status 404 returned error can't find the container with id 538733c7ac7b7c6b3919115f3ab35a0cea0464321e8588af565db8f87f5447f0
Oct 30 21:00:01.001575 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:01.001513    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b425d-29889900-sz9cc"]
Oct 30 21:00:01.024515 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:01.024487    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889900-dtz4k"]
Oct 30 21:00:01.024628 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:00:01.024605    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod280ab52d_2d8d_4fe7_b293_73f470d3cd28.slice/crio-d68fb16915d3aed91265833d642a779cc035314244038bbe979a05f370680d41 WatchSource:0}: Error finding container d68fb16915d3aed91265833d642a779cc035314244038bbe979a05f370680d41: Status 404 returned error can't find the container with id d68fb16915d3aed91265833d642a779cc035314244038bbe979a05f370680d41
Oct 30 21:00:01.047392 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:00:01.047359    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0df799c_ee0f_45c0_bbe2_c7928deb0e3b.slice/crio-a87869e13eb741594ec32ff0637f1015bda349867ac14e1ee995263445a672b7 WatchSource:0}: Error finding container a87869e13eb741594ec32ff0637f1015bda349867ac14e1ee995263445a672b7: Status 404 returned error can't find the container with id a87869e13eb741594ec32ff0637f1015bda349867ac14e1ee995263445a672b7
Oct 30 21:00:01.048552 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:01.048533    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889900-hm4sk"]
Oct 30 21:00:01.065350 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:01.065316    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj"]
Oct 30 21:00:01.072089 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:01.072067    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj"
Oct 30 21:00:01.111078 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:01.111046    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abfc50b7-cb57-4beb-bed1-70204a890a8a-istio-podinfo\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj\" (UID: \"abfc50b7-cb57-4beb-bed1-70204a890a8a\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj"
Oct 30 21:00:01.111231 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:01.111083    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abfc50b7-cb57-4beb-bed1-70204a890a8a-istiod-ca-cert\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj\" (UID: \"abfc50b7-cb57-4beb-bed1-70204a890a8a\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj"
Oct 30 21:00:01.111231 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:01.111159    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abfc50b7-cb57-4beb-bed1-70204a890a8a-istio-data\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj\" (UID: \"abfc50b7-cb57-4beb-bed1-70204a890a8a\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj"
Oct 30 21:00:01.111326 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:01.111283    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abfc50b7-cb57-4beb-bed1-70204a890a8a-credential-socket\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj\" (UID: \"abfc50b7-cb57-4beb-bed1-70204a890a8a\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj"
Oct 30 21:00:01.111326 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:01.111312    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abfc50b7-cb57-4beb-bed1-70204a890a8a-istio-token\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj\" (UID: \"abfc50b7-cb57-4beb-bed1-70204a890a8a\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj"
Oct 30 21:00:01.111408 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:01.111345    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abfc50b7-cb57-4beb-bed1-70204a890a8a-istio-envoy\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj\" (UID: \"abfc50b7-cb57-4beb-bed1-70204a890a8a\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj"
Oct 30 21:00:01.111510 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:01.111488    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abfc50b7-cb57-4beb-bed1-70204a890a8a-workload-socket\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj\" (UID: \"abfc50b7-cb57-4beb-bed1-70204a890a8a\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj"
Oct 30 21:00:01.111565 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:01.111532    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abfc50b7-cb57-4beb-bed1-70204a890a8a-workload-certs\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj\" (UID: \"abfc50b7-cb57-4beb-bed1-70204a890a8a\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj"
Oct 30 21:00:01.111565 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:01.111549    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks45n\" (UniqueName: \"kubernetes.io/projected/abfc50b7-cb57-4beb-bed1-70204a890a8a-kube-api-access-ks45n\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj\" (UID: \"abfc50b7-cb57-4beb-bed1-70204a890a8a\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj"
Oct 30 21:00:01.122157 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:01.122123    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj"]
Oct 30 21:00:01.178047 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:01.178008    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b425d-29889900-sz9cc" event={"ID":"76aed203-0d7c-410e-84bd-637f57a1620a","Type":"ContainerStarted","Data":"538733c7ac7b7c6b3919115f3ab35a0cea0464321e8588af565db8f87f5447f0"}
Oct 30 21:00:01.179407 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:01.179383    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-781058-29889900-tfdx9" event={"ID":"10eec293-a840-4727-ade8-d2afa45e9f9a","Type":"ContainerStarted","Data":"2d0bb4ea9bf183f21dba0c404d85b319f154d4bf6f2f5581bbeff6986018c4e7"}
Oct 30 21:00:01.180702 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:01.180683    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889900-dtz4k" event={"ID":"280ab52d-2d8d-4fe7-b293-73f470d3cd28","Type":"ContainerStarted","Data":"d68fb16915d3aed91265833d642a779cc035314244038bbe979a05f370680d41"}
Oct 30 21:00:01.182065 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:01.182038    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889900-hm4sk" event={"ID":"c0df799c-ee0f-45c0-bbe2-c7928deb0e3b","Type":"ContainerStarted","Data":"a87869e13eb741594ec32ff0637f1015bda349867ac14e1ee995263445a672b7"}
Oct 30 21:00:01.183267 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:01.183247    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d061d0-29889900-sp86r" event={"ID":"a8371114-ea5d-4cbc-b761-5aa5cdfef8fd","Type":"ContainerStarted","Data":"a77542ff4bb79aae3c9b74693038ce0be22eaa4cfb5ac08d946ea9e0cb6b8a9b"}
Oct 30 21:00:01.212823 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:01.212749    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abfc50b7-cb57-4beb-bed1-70204a890a8a-workload-certs\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj\" (UID: \"abfc50b7-cb57-4beb-bed1-70204a890a8a\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj"
Oct 30 21:00:01.212823 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:01.212809    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks45n\" (UniqueName: \"kubernetes.io/projected/abfc50b7-cb57-4beb-bed1-70204a890a8a-kube-api-access-ks45n\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj\" (UID: \"abfc50b7-cb57-4beb-bed1-70204a890a8a\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj"
Oct 30 21:00:01.213063 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:01.212874    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abfc50b7-cb57-4beb-bed1-70204a890a8a-istio-podinfo\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj\" (UID: \"abfc50b7-cb57-4beb-bed1-70204a890a8a\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj"
Oct 30 21:00:01.213063 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:01.212890    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abfc50b7-cb57-4beb-bed1-70204a890a8a-istiod-ca-cert\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj\" (UID: \"abfc50b7-cb57-4beb-bed1-70204a890a8a\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj"
Oct 30 21:00:01.213063 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:01.212929    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abfc50b7-cb57-4beb-bed1-70204a890a8a-istio-data\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj\" (UID: \"abfc50b7-cb57-4beb-bed1-70204a890a8a\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj"
Oct 30 21:00:01.213063 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:01.213022    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abfc50b7-cb57-4beb-bed1-70204a890a8a-credential-socket\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj\" (UID: \"abfc50b7-cb57-4beb-bed1-70204a890a8a\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj"
Oct 30 21:00:01.213063 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:01.213049    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abfc50b7-cb57-4beb-bed1-70204a890a8a-istio-token\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj\" (UID: \"abfc50b7-cb57-4beb-bed1-70204a890a8a\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj"
Oct 30 21:00:01.213316 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:01.213088    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abfc50b7-cb57-4beb-bed1-70204a890a8a-istio-envoy\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj\" (UID: \"abfc50b7-cb57-4beb-bed1-70204a890a8a\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj"
Oct 30 21:00:01.213316 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:01.213208    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abfc50b7-cb57-4beb-bed1-70204a890a8a-workload-socket\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj\" (UID: \"abfc50b7-cb57-4beb-bed1-70204a890a8a\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj"
Oct 30 21:00:01.213316 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:01.213251    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abfc50b7-cb57-4beb-bed1-70204a890a8a-workload-certs\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj\" (UID: \"abfc50b7-cb57-4beb-bed1-70204a890a8a\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj"
Oct 30 21:00:01.213316 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:01.213296    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abfc50b7-cb57-4beb-bed1-70204a890a8a-istio-data\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj\" (UID: \"abfc50b7-cb57-4beb-bed1-70204a890a8a\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj"
Oct 30 21:00:01.213604 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:01.213582    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abfc50b7-cb57-4beb-bed1-70204a890a8a-workload-socket\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj\" (UID: \"abfc50b7-cb57-4beb-bed1-70204a890a8a\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj"
Oct 30 21:00:01.213659 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:01.213599    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abfc50b7-cb57-4beb-bed1-70204a890a8a-credential-socket\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj\" (UID: \"abfc50b7-cb57-4beb-bed1-70204a890a8a\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj"
Oct 30 21:00:01.213892 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:01.213875    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abfc50b7-cb57-4beb-bed1-70204a890a8a-istiod-ca-cert\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj\" (UID: \"abfc50b7-cb57-4beb-bed1-70204a890a8a\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj"
Oct 30 21:00:01.234850 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:01.234727    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abfc50b7-cb57-4beb-bed1-70204a890a8a-istio-envoy\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj\" (UID: \"abfc50b7-cb57-4beb-bed1-70204a890a8a\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj"
Oct 30 21:00:01.235026 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:01.235009    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abfc50b7-cb57-4beb-bed1-70204a890a8a-istio-podinfo\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj\" (UID: \"abfc50b7-cb57-4beb-bed1-70204a890a8a\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj"
Oct 30 21:00:01.336253 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:01.336032    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abfc50b7-cb57-4beb-bed1-70204a890a8a-istio-token\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj\" (UID: \"abfc50b7-cb57-4beb-bed1-70204a890a8a\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj"
Oct 30 21:00:01.336253 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:01.336184    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks45n\" (UniqueName: \"kubernetes.io/projected/abfc50b7-cb57-4beb-bed1-70204a890a8a-kube-api-access-ks45n\") pod \"e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj\" (UID: \"abfc50b7-cb57-4beb-bed1-70204a890a8a\") " pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj"
Oct 30 21:00:01.463979 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:01.463953    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj"
Oct 30 21:00:01.803490 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:01.803408    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff"]
Oct 30 21:00:01.808628 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:01.808604    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff"
Oct 30 21:00:01.858577 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:01.858552    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj"]
Oct 30 21:00:01.867495 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:01.867470    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff"]
Oct 30 21:00:01.888097 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:00:01.888065    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabfc50b7_cb57_4beb_bed1_70204a890a8a.slice/crio-4b95131a0a417dc9b14c2cd182e1b2d25a488090752affeced873d1ee5e2ff12 WatchSource:0}: Error finding container 4b95131a0a417dc9b14c2cd182e1b2d25a488090752affeced873d1ee5e2ff12: Status 404 returned error can't find the container with id 4b95131a0a417dc9b14c2cd182e1b2d25a488090752affeced873d1ee5e2ff12
Oct 30 21:00:01.891627 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:01.891590    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:00:01.891728 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:01.891682    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:00:01.891770 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:01.891728    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:00:01.930176 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:01.930140    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c58cfb06-4eaa-4e30-8382-ab66354eec5a-istio-podinfo\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff\" (UID: \"c58cfb06-4eaa-4e30-8382-ab66354eec5a\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff"
Oct 30 21:00:01.930325 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:01.930265    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c58cfb06-4eaa-4e30-8382-ab66354eec5a-workload-certs\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff\" (UID: \"c58cfb06-4eaa-4e30-8382-ab66354eec5a\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff"
Oct 30 21:00:01.930421 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:01.930401    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c58cfb06-4eaa-4e30-8382-ab66354eec5a-istio-envoy\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff\" (UID: \"c58cfb06-4eaa-4e30-8382-ab66354eec5a\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff"
Oct 30 21:00:01.930491 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:01.930477    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c58cfb06-4eaa-4e30-8382-ab66354eec5a-workload-socket\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff\" (UID: \"c58cfb06-4eaa-4e30-8382-ab66354eec5a\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff"
Oct 30 21:00:01.930528 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:01.930500    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4sm2\" (UniqueName: \"kubernetes.io/projected/c58cfb06-4eaa-4e30-8382-ab66354eec5a-kube-api-access-x4sm2\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff\" (UID: \"c58cfb06-4eaa-4e30-8382-ab66354eec5a\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff"
Oct 30 21:00:01.930562 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:01.930530    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c58cfb06-4eaa-4e30-8382-ab66354eec5a-istio-token\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff\" (UID: \"c58cfb06-4eaa-4e30-8382-ab66354eec5a\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff"
Oct 30 21:00:01.930596 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:01.930579    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c58cfb06-4eaa-4e30-8382-ab66354eec5a-istiod-ca-cert\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff\" (UID: \"c58cfb06-4eaa-4e30-8382-ab66354eec5a\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff"
Oct 30 21:00:01.930665 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:01.930654    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c58cfb06-4eaa-4e30-8382-ab66354eec5a-credential-socket\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff\" (UID: \"c58cfb06-4eaa-4e30-8382-ab66354eec5a\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff"
Oct 30 21:00:01.930701 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:01.930672    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c58cfb06-4eaa-4e30-8382-ab66354eec5a-istio-data\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff\" (UID: \"c58cfb06-4eaa-4e30-8382-ab66354eec5a\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff"
Oct 30 21:00:02.032297 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:02.032259    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c58cfb06-4eaa-4e30-8382-ab66354eec5a-workload-certs\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff\" (UID: \"c58cfb06-4eaa-4e30-8382-ab66354eec5a\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff"
Oct 30 21:00:02.032538 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:02.032374    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c58cfb06-4eaa-4e30-8382-ab66354eec5a-istio-envoy\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff\" (UID: \"c58cfb06-4eaa-4e30-8382-ab66354eec5a\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff"
Oct 30 21:00:02.032538 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:02.032456    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c58cfb06-4eaa-4e30-8382-ab66354eec5a-workload-socket\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff\" (UID: \"c58cfb06-4eaa-4e30-8382-ab66354eec5a\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff"
Oct 30 21:00:02.032538 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:02.032483    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4sm2\" (UniqueName: \"kubernetes.io/projected/c58cfb06-4eaa-4e30-8382-ab66354eec5a-kube-api-access-x4sm2\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff\" (UID: \"c58cfb06-4eaa-4e30-8382-ab66354eec5a\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff"
Oct 30 21:00:02.032538 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:02.032529    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c58cfb06-4eaa-4e30-8382-ab66354eec5a-istio-token\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff\" (UID: \"c58cfb06-4eaa-4e30-8382-ab66354eec5a\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff"
Oct 30 21:00:02.032857 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:02.032546    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c58cfb06-4eaa-4e30-8382-ab66354eec5a-istiod-ca-cert\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff\" (UID: \"c58cfb06-4eaa-4e30-8382-ab66354eec5a\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff"
Oct 30 21:00:02.032857 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:02.032599    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c58cfb06-4eaa-4e30-8382-ab66354eec5a-credential-socket\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff\" (UID: \"c58cfb06-4eaa-4e30-8382-ab66354eec5a\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff"
Oct 30 21:00:02.032857 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:02.032622    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c58cfb06-4eaa-4e30-8382-ab66354eec5a-istio-data\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff\" (UID: \"c58cfb06-4eaa-4e30-8382-ab66354eec5a\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff"
Oct 30 21:00:02.033027 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:02.032962    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c58cfb06-4eaa-4e30-8382-ab66354eec5a-istio-podinfo\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff\" (UID: \"c58cfb06-4eaa-4e30-8382-ab66354eec5a\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff"
Oct 30 21:00:02.033518 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:02.033164    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c58cfb06-4eaa-4e30-8382-ab66354eec5a-workload-certs\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff\" (UID: \"c58cfb06-4eaa-4e30-8382-ab66354eec5a\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff"
Oct 30 21:00:02.033518 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:02.033448    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c58cfb06-4eaa-4e30-8382-ab66354eec5a-istio-data\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff\" (UID: \"c58cfb06-4eaa-4e30-8382-ab66354eec5a\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff"
Oct 30 21:00:02.033518 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:02.033448    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c58cfb06-4eaa-4e30-8382-ab66354eec5a-istiod-ca-cert\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff\" (UID: \"c58cfb06-4eaa-4e30-8382-ab66354eec5a\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff"
Oct 30 21:00:02.033518 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:02.033440    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c58cfb06-4eaa-4e30-8382-ab66354eec5a-credential-socket\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff\" (UID: \"c58cfb06-4eaa-4e30-8382-ab66354eec5a\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff"
Oct 30 21:00:02.033832 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:02.033628    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c58cfb06-4eaa-4e30-8382-ab66354eec5a-workload-socket\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff\" (UID: \"c58cfb06-4eaa-4e30-8382-ab66354eec5a\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff"
Oct 30 21:00:02.039287 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:02.038749    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c58cfb06-4eaa-4e30-8382-ab66354eec5a-istio-podinfo\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff\" (UID: \"c58cfb06-4eaa-4e30-8382-ab66354eec5a\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff"
Oct 30 21:00:02.040531 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:02.040480    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c58cfb06-4eaa-4e30-8382-ab66354eec5a-istio-envoy\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff\" (UID: \"c58cfb06-4eaa-4e30-8382-ab66354eec5a\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff"
Oct 30 21:00:02.119815 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:02.115688    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4sm2\" (UniqueName: \"kubernetes.io/projected/c58cfb06-4eaa-4e30-8382-ab66354eec5a-kube-api-access-x4sm2\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff\" (UID: \"c58cfb06-4eaa-4e30-8382-ab66354eec5a\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff"
Oct 30 21:00:02.129645 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:02.126670    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c58cfb06-4eaa-4e30-8382-ab66354eec5a-istio-token\") pod \"e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff\" (UID: \"c58cfb06-4eaa-4e30-8382-ab66354eec5a\") " pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff"
Oct 30 21:00:02.196153 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:02.195436    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889900-hm4sk" event={"ID":"c0df799c-ee0f-45c0-bbe2-c7928deb0e3b","Type":"ContainerStarted","Data":"b306a5f351c8f9b5f67203274e5d5ab0f9e9553afd82647b49f7f60d5dde3417"}
Oct 30 21:00:02.197424 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:02.197382    5713 generic.go:356] "Generic (PLEG): container finished" podID="a8371114-ea5d-4cbc-b761-5aa5cdfef8fd" containerID="34fe9f1df14cd9447cc433149fe247156cad5de5a7172beb8ec319f00165fc8b" exitCode=60
Oct 30 21:00:02.197574 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:02.197471    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d061d0-29889900-sp86r" event={"ID":"a8371114-ea5d-4cbc-b761-5aa5cdfef8fd","Type":"ContainerDied","Data":"34fe9f1df14cd9447cc433149fe247156cad5de5a7172beb8ec319f00165fc8b"}
Oct 30 21:00:02.197722 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:02.197705    5713 scope.go:122] "RemoveContainer" containerID="34fe9f1df14cd9447cc433149fe247156cad5de5a7172beb8ec319f00165fc8b"
Oct 30 21:00:02.200194 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:02.199902    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj" event={"ID":"abfc50b7-cb57-4beb-bed1-70204a890a8a","Type":"ContainerStarted","Data":"41fbd6aa0515284ade2bdb53b910c4d2662ab55942efce7633fb565c93cfb232"}
Oct 30 21:00:02.200194 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:02.199958    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj" event={"ID":"abfc50b7-cb57-4beb-bed1-70204a890a8a","Type":"ContainerStarted","Data":"4b95131a0a417dc9b14c2cd182e1b2d25a488090752affeced873d1ee5e2ff12"}
Oct 30 21:00:02.202628 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:02.202607    5713 generic.go:356] "Generic (PLEG): container finished" podID="76aed203-0d7c-410e-84bd-637f57a1620a" containerID="a7014bfd801c2503588516aa611e4f4fa71cc9aa09503bfeb574470151840263" exitCode=60
Oct 30 21:00:02.202897 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:02.202876    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b425d-29889900-sz9cc" event={"ID":"76aed203-0d7c-410e-84bd-637f57a1620a","Type":"ContainerDied","Data":"a7014bfd801c2503588516aa611e4f4fa71cc9aa09503bfeb574470151840263"}
Oct 30 21:00:02.203139 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:02.203123    5713 scope.go:122] "RemoveContainer" containerID="a7014bfd801c2503588516aa611e4f4fa71cc9aa09503bfeb574470151840263"
Oct 30 21:00:02.217837 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:02.206895    5713 generic.go:356] "Generic (PLEG): container finished" podID="10eec293-a840-4727-ade8-d2afa45e9f9a" containerID="5906e90aea0377d8fe94a7ead988d74e09cffffac8faf459dd8d3da84cbd76cb" exitCode=60
Oct 30 21:00:02.217837 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:02.207003    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-781058-29889900-tfdx9" event={"ID":"10eec293-a840-4727-ade8-d2afa45e9f9a","Type":"ContainerDied","Data":"5906e90aea0377d8fe94a7ead988d74e09cffffac8faf459dd8d3da84cbd76cb"}
Oct 30 21:00:02.217837 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:02.207291    5713 scope.go:122] "RemoveContainer" containerID="5906e90aea0377d8fe94a7ead988d74e09cffffac8faf459dd8d3da84cbd76cb"
Oct 30 21:00:02.217837 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:02.209894    5713 generic.go:356] "Generic (PLEG): container finished" podID="280ab52d-2d8d-4fe7-b293-73f470d3cd28" containerID="f545228ff37edbb59db90fb486ccb96ef6582579f5f7d4006b8d5ab9657199f5" exitCode=60
Oct 30 21:00:02.217837 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:02.210007    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889900-dtz4k" event={"ID":"280ab52d-2d8d-4fe7-b293-73f470d3cd28","Type":"ContainerDied","Data":"f545228ff37edbb59db90fb486ccb96ef6582579f5f7d4006b8d5ab9657199f5"}
Oct 30 21:00:02.217837 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:02.210325    5713 scope.go:122] "RemoveContainer" containerID="f545228ff37edbb59db90fb486ccb96ef6582579f5f7d4006b8d5ab9657199f5"
Oct 30 21:00:02.424983 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:02.424827    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889900-hm4sk" podStartSLOduration=2.424765274 podStartE2EDuration="2.424765274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:02.310052845 +0000 UTC m=+1285.374829242" watchObservedRunningTime="2026-10-30 21:00:02.424765274 +0000 UTC m=+1285.489541673"
Oct 30 21:00:02.424983 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:02.424953    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff"
Oct 30 21:00:02.472058 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:02.468751    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj"
Oct 30 21:00:02.488534 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:02.488470    5713 patch_prober.go:29] interesting pod/e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 30 21:00:02.488814 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:02.488740    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj" podUID="abfc50b7-cb57-4beb-bed1-70204a890a8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 30 21:00:02.696378 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:02.693063    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj" podStartSLOduration=1.693045155 podStartE2EDuration="1.693045155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:02.643382503 +0000 UTC m=+1285.708158976" watchObservedRunningTime="2026-10-30 21:00:02.693045155 +0000 UTC m=+1285.757821555"
Oct 30 21:00:02.907605 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:02.906476    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff"]
Oct 30 21:00:03.107856 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:03.098988    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj"]
Oct 30 21:00:03.142817 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:00:03.124578    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc58cfb06_4eaa_4e30_8382_ab66354eec5a.slice/crio-f478fd1aa02965e07f08eb0089840c6092b22524de6c07341a741072ccbf8ac2 WatchSource:0}: Error finding container f478fd1aa02965e07f08eb0089840c6092b22524de6c07341a741072ccbf8ac2: Status 404 returned error can't find the container with id f478fd1aa02965e07f08eb0089840c6092b22524de6c07341a741072ccbf8ac2
Oct 30 21:00:03.481217 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:03.481172    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj"
Oct 30 21:00:03.675098 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:03.556569    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b4b9def-33e5-4d81-a159-1389d71ced98-workload-socket\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj\" (UID: \"4b4b9def-33e5-4d81-a159-1389d71ced98\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj"
Oct 30 21:00:03.675098 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:03.664332    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b4b9def-33e5-4d81-a159-1389d71ced98-istio-token\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj\" (UID: \"4b4b9def-33e5-4d81-a159-1389d71ced98\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj"
Oct 30 21:00:03.675098 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:03.664491    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b4b9def-33e5-4d81-a159-1389d71ced98-istio-envoy\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj\" (UID: \"4b4b9def-33e5-4d81-a159-1389d71ced98\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj"
Oct 30 21:00:03.675098 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:03.664892    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp2jw\" (UniqueName: \"kubernetes.io/projected/4b4b9def-33e5-4d81-a159-1389d71ced98-kube-api-access-kp2jw\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj\" (UID: \"4b4b9def-33e5-4d81-a159-1389d71ced98\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj"
Oct 30 21:00:03.675098 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:03.664985    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b4b9def-33e5-4d81-a159-1389d71ced98-istio-data\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj\" (UID: \"4b4b9def-33e5-4d81-a159-1389d71ced98\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj"
Oct 30 21:00:03.675098 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:03.665067    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b4b9def-33e5-4d81-a159-1389d71ced98-istiod-ca-cert\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj\" (UID: \"4b4b9def-33e5-4d81-a159-1389d71ced98\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj"
Oct 30 21:00:03.675098 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:03.665167    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b4b9def-33e5-4d81-a159-1389d71ced98-workload-certs\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj\" (UID: \"4b4b9def-33e5-4d81-a159-1389d71ced98\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj"
Oct 30 21:00:03.675098 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:03.665224    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b4b9def-33e5-4d81-a159-1389d71ced98-istio-podinfo\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj\" (UID: \"4b4b9def-33e5-4d81-a159-1389d71ced98\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj"
Oct 30 21:00:03.675098 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:03.665546    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b4b9def-33e5-4d81-a159-1389d71ced98-credential-socket\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj\" (UID: \"4b4b9def-33e5-4d81-a159-1389d71ced98\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj"
Oct 30 21:00:03.675098 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:03.607535    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:00:03.675098 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:03.667002    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:00:03.675098 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:03.667242    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:00:03.804927 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:03.624601    5713 patch_prober.go:29] interesting pod/e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 30 21:00:03.811920 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:03.811877    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj" podUID="abfc50b7-cb57-4beb-bed1-70204a890a8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 30 21:00:03.841586 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:03.841545    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b4b9def-33e5-4d81-a159-1389d71ced98-istiod-ca-cert\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj\" (UID: \"4b4b9def-33e5-4d81-a159-1389d71ced98\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj"
Oct 30 21:00:03.846016 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:03.845966    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b4b9def-33e5-4d81-a159-1389d71ced98-workload-certs\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj\" (UID: \"4b4b9def-33e5-4d81-a159-1389d71ced98\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj"
Oct 30 21:00:03.907031 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:03.843935    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-781058/premium-w0-781058-kserve-7f5c8964c9-z5fmk" podUID="57626f4f-844e-4ec9-ab89-359d015923af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 30 21:00:03.907410 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:03.842652    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b4b9def-33e5-4d81-a159-1389d71ced98-istiod-ca-cert\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj\" (UID: \"4b4b9def-33e5-4d81-a159-1389d71ced98\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj"
Oct 30 21:00:03.907559 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:03.860430    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b4b9def-33e5-4d81-a159-1389d71ced98-workload-certs\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj\" (UID: \"4b4b9def-33e5-4d81-a159-1389d71ced98\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj"
Oct 30 21:00:04.006304 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:03.906976    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b4b9def-33e5-4d81-a159-1389d71ced98-istio-podinfo\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj\" (UID: \"4b4b9def-33e5-4d81-a159-1389d71ced98\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj"
Oct 30 21:00:04.006640 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:04.006599    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b4b9def-33e5-4d81-a159-1389d71ced98-credential-socket\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj\" (UID: \"4b4b9def-33e5-4d81-a159-1389d71ced98\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj"
Oct 30 21:00:04.061928 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:04.007089    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b4b9def-33e5-4d81-a159-1389d71ced98-credential-socket\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj\" (UID: \"4b4b9def-33e5-4d81-a159-1389d71ced98\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj"
Oct 30 21:00:04.061928 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:04.036655    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b425d/simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk" podUID="e8108824-03ff-4c1d-9f5b-07be3a3a92d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:04.063447 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:04.063376    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b4b9def-33e5-4d81-a159-1389d71ced98-workload-socket\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj\" (UID: \"4b4b9def-33e5-4d81-a159-1389d71ced98\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj"
Oct 30 21:00:04.063671 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:04.063603    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b4b9def-33e5-4d81-a159-1389d71ced98-istio-token\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj\" (UID: \"4b4b9def-33e5-4d81-a159-1389d71ced98\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj"
Oct 30 21:00:04.063962 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:04.063888    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b4b9def-33e5-4d81-a159-1389d71ced98-istio-envoy\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj\" (UID: \"4b4b9def-33e5-4d81-a159-1389d71ced98\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj"
Oct 30 21:00:04.064151 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:04.064090    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp2jw\" (UniqueName: \"kubernetes.io/projected/4b4b9def-33e5-4d81-a159-1389d71ced98-kube-api-access-kp2jw\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj\" (UID: \"4b4b9def-33e5-4d81-a159-1389d71ced98\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj"
Oct 30 21:00:04.064310 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:04.064288    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b4b9def-33e5-4d81-a159-1389d71ced98-workload-socket\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj\" (UID: \"4b4b9def-33e5-4d81-a159-1389d71ced98\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj"
Oct 30 21:00:04.091918 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:04.064300    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b4b9def-33e5-4d81-a159-1389d71ced98-istio-data\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj\" (UID: \"4b4b9def-33e5-4d81-a159-1389d71ced98\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj"
Oct 30 21:00:04.092091 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:04.064630    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b4b9def-33e5-4d81-a159-1389d71ced98-istio-data\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj\" (UID: \"4b4b9def-33e5-4d81-a159-1389d71ced98\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj"
Oct 30 21:00:04.092091 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:04.068404    5713 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2b425d/simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk" podUID="e8108824-03ff-4c1d-9f5b-07be3a3a92d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:04.092091 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:04.068436    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l" podUID="fc9510c5-2180-41d6-90bc-9acdff0f3616" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded"
Oct 30 21:00:04.136402 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:04.135926    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm" podUID="cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:04.168813 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:04.163875    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7" podUID="b2193213-ebae-498a-911a-6056a8e94312" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:04.208333 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:04.208292    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b4b9def-33e5-4d81-a159-1389d71ced98-istio-podinfo\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj\" (UID: \"4b4b9def-33e5-4d81-a159-1389d71ced98\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj"
Oct 30 21:00:04.263503 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:04.263471    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj"]
Oct 30 21:00:04.295403 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:04.295325    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62ce42bb-vw2k7"]
Oct 30 21:00:04.299813 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:04.296307    5713 scope.go:122] "RemoveContainer" containerID="b306a5f351c8f9b5f67203274e5d5ab0f9e9553afd82647b49f7f60d5dde3417"
Oct 30 21:00:04.300258 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:04.300216    5713 generic.go:356] "Generic (PLEG): container finished" podID="c0df799c-ee0f-45c0-bbe2-c7928deb0e3b" containerID="b306a5f351c8f9b5f67203274e5d5ab0f9e9553afd82647b49f7f60d5dde3417" exitCode=60
Oct 30 21:00:04.315088 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:04.300276    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889900-hm4sk" event={"ID":"c0df799c-ee0f-45c0-bbe2-c7928deb0e3b","Type":"ContainerDied","Data":"b306a5f351c8f9b5f67203274e5d5ab0f9e9553afd82647b49f7f60d5dde3417"}
Oct 30 21:00:04.315370 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:04.304103    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b4b9def-33e5-4d81-a159-1389d71ced98-istio-envoy\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj\" (UID: \"4b4b9def-33e5-4d81-a159-1389d71ced98\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj"
Oct 30 21:00:04.344817 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:04.318084    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62ce42bb-vw2k7"]
Oct 30 21:00:04.347200 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:04.347163    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d061d0-29889900-sp86r" event={"ID":"a8371114-ea5d-4cbc-b761-5aa5cdfef8fd","Type":"ContainerDied","Data":"30729bbc4c99bc320a3f5dc589f67af341ac8a7f4b685f5c8fe509115d6eb3d2"}
Oct 30 21:00:04.347427 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:04.335028    5713 generic.go:356] "Generic (PLEG): container finished" podID="a8371114-ea5d-4cbc-b761-5aa5cdfef8fd" containerID="30729bbc4c99bc320a3f5dc589f67af341ac8a7f4b685f5c8fe509115d6eb3d2" exitCode=60
Oct 30 21:00:04.379143 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:04.345905    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp2jw\" (UniqueName: \"kubernetes.io/projected/4b4b9def-33e5-4d81-a159-1389d71ced98-kube-api-access-kp2jw\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj\" (UID: \"4b4b9def-33e5-4d81-a159-1389d71ced98\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj"
Oct 30 21:00:04.379143 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:04.335516    5713 scope.go:122] "RemoveContainer" containerID="30729bbc4c99bc320a3f5dc589f67af341ac8a7f4b685f5c8fe509115d6eb3d2"
Oct 30 21:00:04.379143 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:00:04.354543    5713 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-d061d0-29889900-sp86r_odh-ai-gateway-infra(a8371114-ea5d-4cbc-b761-5aa5cdfef8fd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d061d0-29889900-sp86r" podUID="a8371114-ea5d-4cbc-b761-5aa5cdfef8fd"
Oct 30 21:00:04.379143 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:04.354611    5713 scope.go:122] "RemoveContainer" containerID="34fe9f1df14cd9447cc433149fe247156cad5de5a7172beb8ec319f00165fc8b"
Oct 30 21:00:04.383429 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:04.383392    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b4b9def-33e5-4d81-a159-1389d71ced98-istio-token\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj\" (UID: \"4b4b9def-33e5-4d81-a159-1389d71ced98\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj"
Oct 30 21:00:04.392199 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:04.386341    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-682999/premium-w3-682999-kserve-5f47cf9b8d-xqnnr" podUID="a54cd33d-feef-42bb-a78b-7480634cb58d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:04.398182 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:04.394914    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b425d/distinct-2-w5-2b425d-kserve-6694867c68-bzpvn" podUID="b9d391e6-de6f-476d-a926-d1b333474109" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded"
Oct 30 21:00:04.398182 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:04.395359    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-781058/simulator-w0-781058-kserve-55b954cb4b-4k7z6" podUID="92c43343-2820-4097-b829-9aae96b670de" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Oct 30 21:00:04.405332 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:04.405296    5713 generic.go:356] "Generic (PLEG): container finished" podID="76aed203-0d7c-410e-84bd-637f57a1620a" containerID="fdfbe2a1e6a4970d2b8720c7948a02ed5d9b6feb84847141ee30d76399ef5b2a" exitCode=60
Oct 30 21:00:04.414410 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:04.405603    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b425d-29889900-sz9cc" event={"ID":"76aed203-0d7c-410e-84bd-637f57a1620a","Type":"ContainerDied","Data":"fdfbe2a1e6a4970d2b8720c7948a02ed5d9b6feb84847141ee30d76399ef5b2a"}
Oct 30 21:00:04.414769 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:04.410195    5713 scope.go:122] "RemoveContainer" containerID="fdfbe2a1e6a4970d2b8720c7948a02ed5d9b6feb84847141ee30d76399ef5b2a"
Oct 30 21:00:04.428338 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:00:04.428300    5713 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-2b425d-29889900-sz9cc_odh-ai-gateway-infra(76aed203-0d7c-410e-84bd-637f57a1620a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b425d-29889900-sz9cc" podUID="76aed203-0d7c-410e-84bd-637f57a1620a"
Oct 30 21:00:04.428885 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:04.428860    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-781058-29889900-tfdx9" event={"ID":"10eec293-a840-4727-ade8-d2afa45e9f9a","Type":"ContainerStarted","Data":"3eb2e919ab636810c98a7f0a22f27b03bde36c9e067d74086847375411150f06"}
Oct 30 21:00:04.435838 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:04.432683    5713 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l" podUID="fc9510c5-2180-41d6-90bc-9acdff0f3616" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:04.436069 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:04.435479    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b425d/premium-w5-2b425d-kserve-78756b6c5b-9x5xb" podUID="abf7b512-50e5-4400-9ca3-9ffccaead3d7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 30 21:00:04.484016 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:04.483982    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj"
Oct 30 21:00:04.552256 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:04.484577    5713 patch_prober.go:29] interesting pod/e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 30 21:00:04.552256 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:04.550819    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj" podUID="abfc50b7-cb57-4beb-bed1-70204a890a8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 30 21:00:04.552256 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:04.534202    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889900-dtz4k" event={"ID":"280ab52d-2d8d-4fe7-b293-73f470d3cd28","Type":"ContainerStarted","Data":"65a86913089e9ad50dafd0fdee5e35881e1026ecfa89ed7677fda9d906ffbcf0"}
Oct 30 21:00:04.552256 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:04.532953    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-682999/distinct-w3-682999-kserve-74d49d4955-nndlc" podUID="2a35314b-31ea-42e6-b49a-15ab1305b338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:04.552256 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:04.551959    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff" event={"ID":"c58cfb06-4eaa-4e30-8382-ab66354eec5a","Type":"ContainerStarted","Data":"f478fd1aa02965e07f08eb0089840c6092b22524de6c07341a741072ccbf8ac2"}
Oct 30 21:00:04.564441 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:04.564400    5713 scope.go:122] "RemoveContainer" containerID="a7014bfd801c2503588516aa611e4f4fa71cc9aa09503bfeb574470151840263"
Oct 30 21:00:04.590510 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:04.590472    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-781058/distinct-2-w0-781058-kserve-7f6c654757-ppmhm" podUID="5e818303-0e9f-4ba0-a635-b80b92c9e36d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:04.662052 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:04.659742    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f9f277/premium-w1-f9f277-kserve-5dbfd4d496-72w4n" podUID="1cc1f811-c149-4457-80f1-1a55621d73b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:04.662052 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:04.660629    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-682999/distinct-2-w3-682999-kserve-7bc687885b-gq5x7" podUID="0dcf0422-081f-46d7-b246-7a49b074c88c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:04.692145 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:04.692069    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-781058-29889900-tfdx9" podStartSLOduration=4.692046381 podStartE2EDuration="4.692046381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:04.671055566 +0000 UTC m=+1287.735831977" watchObservedRunningTime="2026-10-30 21:00:04.692046381 +0000 UTC m=+1287.756822781"
Oct 30 21:00:04.841361 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:04.813220    5713 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-f9f277/premium-w1-f9f277-kserve-5dbfd4d496-72w4n" podUID="1cc1f811-c149-4457-80f1-1a55621d73b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:04.841361 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:04.840359    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-682999/simulator-w3-682999-kserve-57cfcb6f8f-6mgrg" podUID="53296e7f-c6bd-45f2-ac31-004feddc63a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:04.932893 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:04.915238    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-781058/premium-w0-781058-kserve-7f5c8964c9-z5fmk" podUID="57626f4f-844e-4ec9-ab89-359d015923af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:04.932893 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:04.915711    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-781058/premium-w0-781058-kserve-7f5c8964c9-z5fmk"
Oct 30 21:00:04.932893 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:04.924158    5713 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7" podUID="b2193213-ebae-498a-911a-6056a8e94312" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:04.958776 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:04.957996    5713 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-781058/premium-w0-781058-kserve-7f5c8964c9-z5fmk" podUID="57626f4f-844e-4ec9-ab89-359d015923af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": context deadline exceeded"
Oct 30 21:00:05.041961 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:05.041917    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b425d/simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk" podUID="e8108824-03ff-4c1d-9f5b-07be3a3a92d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:05.044466 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:05.044440    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2b425d/simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk"
Oct 30 21:00:05.113167 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:05.113123    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b97e94c-3c5d-4733-bde5-9edb818d2a6b" path="/var/lib/kubelet/pods/9b97e94c-3c5d-4733-bde5-9edb818d2a6b/volumes"
Oct 30 21:00:05.136227 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:05.135938    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l" podUID="fc9510c5-2180-41d6-90bc-9acdff0f3616" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:05.136820 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:05.136781    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l"
Oct 30 21:00:05.182963 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:05.181773    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7" podUID="b2193213-ebae-498a-911a-6056a8e94312" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:05.183600 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:05.183250    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm" podUID="cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:05.196578 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:05.196458    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7"
Oct 30 21:00:05.196978 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:05.196961    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm"
Oct 30 21:00:05.218591 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:05.211501    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889900-dtz4k" podStartSLOduration=5.211478848 podStartE2EDuration="5.211478848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:05.114709647 +0000 UTC m=+1288.179486045" watchObservedRunningTime="2026-10-30 21:00:05.211478848 +0000 UTC m=+1288.276255250"
Oct 30 21:00:05.403016 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:05.402750    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-682999/premium-w3-682999-kserve-5f47cf9b8d-xqnnr" podUID="a54cd33d-feef-42bb-a78b-7480634cb58d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:05.412442 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:05.412413    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-682999/premium-w3-682999-kserve-5f47cf9b8d-xqnnr"
Oct 30 21:00:05.460956 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:05.460910    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b425d/distinct-2-w5-2b425d-kserve-6694867c68-bzpvn" podUID="b9d391e6-de6f-476d-a926-d1b333474109" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:05.499826 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:05.499295    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b425d/premium-w5-2b425d-kserve-78756b6c5b-9x5xb" podUID="abf7b512-50e5-4400-9ca3-9ffccaead3d7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 30 21:00:05.608007 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:05.566397    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2b425d/distinct-2-w5-2b425d-kserve-6694867c68-bzpvn"
Oct 30 21:00:05.608357 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:05.608330    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2b425d/premium-w5-2b425d-kserve-78756b6c5b-9x5xb"
Oct 30 21:00:05.609425 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:05.598716    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-781058/distinct-2-w0-781058-kserve-7f6c654757-ppmhm" podUID="5e818303-0e9f-4ba0-a635-b80b92c9e36d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:05.704840 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:05.609598    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-781058/simulator-w0-781058-kserve-55b954cb4b-4k7z6" podUID="92c43343-2820-4097-b829-9aae96b670de" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:05.731805 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:05.598825    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-682999/distinct-w3-682999-kserve-74d49d4955-nndlc" podUID="2a35314b-31ea-42e6-b49a-15ab1305b338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Oct 30 21:00:05.731805 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:05.637366    5713 patch_prober.go:29] interesting pod/e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 30 21:00:05.731805 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:05.726353    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj" podUID="abfc50b7-cb57-4beb-bed1-70204a890a8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 30 21:00:05.731805 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:05.717401    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-781058/distinct-2-w0-781058-kserve-7f6c654757-ppmhm"
Oct 30 21:00:05.731805 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:05.726464    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2b425d/distinct-2-w5-2b425d-kserve-6694867c68-bzpvn"
Oct 30 21:00:05.731805 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:05.726505    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-781058/premium-w0-781058-kserve-7f5c8964c9-z5fmk"
Oct 30 21:00:05.731805 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:05.726546    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-781058/simulator-w0-781058-kserve-55b954cb4b-4k7z6"
Oct 30 21:00:05.731805 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:05.726579    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-682999/distinct-w3-682999-kserve-74d49d4955-nndlc"
Oct 30 21:00:05.731805 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:05.712469    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f9f277/premium-w1-f9f277-kserve-5dbfd4d496-72w4n" podUID="1cc1f811-c149-4457-80f1-1a55621d73b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:05.731805 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:05.727572    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f9f277/premium-w1-f9f277-kserve-5dbfd4d496-72w4n"
Oct 30 21:00:05.797314 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:05.797281    5713 generic.go:356] "Generic (PLEG): container finished" podID="10eec293-a840-4727-ade8-d2afa45e9f9a" containerID="3eb2e919ab636810c98a7f0a22f27b03bde36c9e067d74086847375411150f06" exitCode=60
Oct 30 21:00:05.797655 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:05.797629    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-781058-29889900-tfdx9" event={"ID":"10eec293-a840-4727-ade8-d2afa45e9f9a","Type":"ContainerDied","Data":"3eb2e919ab636810c98a7f0a22f27b03bde36c9e067d74086847375411150f06"}
Oct 30 21:00:05.797778 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:05.797752    5713 scope.go:122] "RemoveContainer" containerID="fdfbe2a1e6a4970d2b8720c7948a02ed5d9b6feb84847141ee30d76399ef5b2a"
Oct 30 21:00:05.797935 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:05.797917    5713 scope.go:122] "RemoveContainer" containerID="5906e90aea0377d8fe94a7ead988d74e09cffffac8faf459dd8d3da84cbd76cb"
Oct 30 21:00:05.798046 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:00:05.798021    5713 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-2b425d-29889900-sz9cc_odh-ai-gateway-infra(76aed203-0d7c-410e-84bd-637f57a1620a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b425d-29889900-sz9cc" podUID="76aed203-0d7c-410e-84bd-637f57a1620a"
Oct 30 21:00:05.798463 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:05.798381    5713 scope.go:122] "RemoveContainer" containerID="3eb2e919ab636810c98a7f0a22f27b03bde36c9e067d74086847375411150f06"
Oct 30 21:00:05.798825 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:00:05.798617    5713 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-781058-29889900-tfdx9_odh-ai-gateway-infra(10eec293-a840-4727-ade8-d2afa45e9f9a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-781058-29889900-tfdx9" podUID="10eec293-a840-4727-ade8-d2afa45e9f9a"
Oct 30 21:00:05.803129 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:05.803106    5713 generic.go:356] "Generic (PLEG): container finished" podID="280ab52d-2d8d-4fe7-b293-73f470d3cd28" containerID="65a86913089e9ad50dafd0fdee5e35881e1026ecfa89ed7677fda9d906ffbcf0" exitCode=60
Oct 30 21:00:05.804887 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:05.804865    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889900-dtz4k" event={"ID":"280ab52d-2d8d-4fe7-b293-73f470d3cd28","Type":"ContainerDied","Data":"65a86913089e9ad50dafd0fdee5e35881e1026ecfa89ed7677fda9d906ffbcf0"}
Oct 30 21:00:05.805616 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:05.805598    5713 scope.go:122] "RemoveContainer" containerID="65a86913089e9ad50dafd0fdee5e35881e1026ecfa89ed7677fda9d906ffbcf0"
Oct 30 21:00:05.806191 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:00:05.806167    5713 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889900-dtz4k_odh-ai-gateway-infra(280ab52d-2d8d-4fe7-b293-73f470d3cd28)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889900-dtz4k" podUID="280ab52d-2d8d-4fe7-b293-73f470d3cd28"
Oct 30 21:00:05.807842 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:05.807824    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-682999/distinct-w3-682999-kserve-74d49d4955-nndlc"
Oct 30 21:00:05.811043 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:05.811019    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff" event={"ID":"c58cfb06-4eaa-4e30-8382-ab66354eec5a","Type":"ContainerStarted","Data":"78cb10ce4647f90759b455fd84f99cb23c65d7290f042a1f206bca954246fb5a"}
Oct 30 21:00:05.850172 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:05.848081    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-682999/simulator-w3-682999-kserve-57cfcb6f8f-6mgrg" podUID="53296e7f-c6bd-45f2-ac31-004feddc63a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:05.850172 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:05.849478    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-781058/distinct-2-w0-781058-kserve-7f6c654757-ppmhm"
Oct 30 21:00:05.864449 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:05.864408    5713 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2b425d/distinct-2-w5-2b425d-kserve-6694867c68-bzpvn" podUID="b9d391e6-de6f-476d-a926-d1b333474109" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:05.873271 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:05.873236    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-682999/simulator-w3-682999-kserve-57cfcb6f8f-6mgrg"
Oct 30 21:00:05.874759 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:05.874733    5713 scope.go:122] "RemoveContainer" containerID="f545228ff37edbb59db90fb486ccb96ef6582579f5f7d4006b8d5ab9657199f5"
Oct 30 21:00:05.875501 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:05.875467    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889900-hm4sk" event={"ID":"c0df799c-ee0f-45c0-bbe2-c7928deb0e3b","Type":"ContainerStarted","Data":"38984b6e6bd5c7d93ba0fad6cd09169749ea72f4a6e93a069d5bdc90d52af096"}
Oct 30 21:00:05.876627 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:05.876607    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f9f277/premium-w1-f9f277-kserve-5dbfd4d496-72w4n"
Oct 30 21:00:05.882943 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:05.878551    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2b425d/premium-w5-2b425d-kserve-78756b6c5b-9x5xb"
Oct 30 21:00:05.892756 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:05.892730    5713 scope.go:122] "RemoveContainer" containerID="30729bbc4c99bc320a3f5dc589f67af341ac8a7f4b685f5c8fe509115d6eb3d2"
Oct 30 21:00:05.896384 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:00:05.896343    5713 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-d061d0-29889900-sp86r_odh-ai-gateway-infra(a8371114-ea5d-4cbc-b761-5aa5cdfef8fd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d061d0-29889900-sp86r" podUID="a8371114-ea5d-4cbc-b761-5aa5cdfef8fd"
Oct 30 21:00:05.920472 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:05.920440    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-682999/simulator-w3-682999-kserve-57cfcb6f8f-6mgrg"
Oct 30 21:00:05.930392 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:05.929841    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7"
Oct 30 21:00:05.996572 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:05.996386    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm"
Oct 30 21:00:05.996572 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:05.996508    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-781058/simulator-w0-781058-kserve-55b954cb4b-4k7z6"
Oct 30 21:00:06.005586 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:06.005561    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l"
Oct 30 21:00:06.006655 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:06.006635    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-682999/premium-w3-682999-kserve-5f47cf9b8d-xqnnr"
Oct 30 21:00:06.009579 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:06.009457    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2b425d/simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk"
Oct 30 21:00:06.082918 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:06.082888    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj"]
Oct 30 21:00:06.087676 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:00:06.087645    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b4b9def_33e5_4d81_a159_1389d71ced98.slice/crio-be792cf26fcfb01d8ba3a9b42146c24e105e6e9c07a4397e5c86840732a362fd WatchSource:0}: Error finding container be792cf26fcfb01d8ba3a9b42146c24e105e6e9c07a4397e5c86840732a362fd: Status 404 returned error can't find the container with id be792cf26fcfb01d8ba3a9b42146c24e105e6e9c07a4397e5c86840732a362fd
Oct 30 21:00:06.090986 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:06.090954    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:00:06.091081 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:06.091047    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:00:06.091138 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:06.091090    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:00:06.426485 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:06.426451    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff"
Oct 30 21:00:06.427222 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:06.427202    5713 patch_prober.go:29] interesting pod/e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 30 21:00:06.427379 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:06.427360    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff" podUID="c58cfb06-4eaa-4e30-8382-ab66354eec5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 30 21:00:06.471873 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:06.471180    5713 patch_prober.go:29] interesting pod/e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 30 21:00:06.471873 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:06.471259    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj" podUID="abfc50b7-cb57-4beb-bed1-70204a890a8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 30 21:00:06.908095 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:06.908027    5713 scope.go:122] "RemoveContainer" containerID="3eb2e919ab636810c98a7f0a22f27b03bde36c9e067d74086847375411150f06"
Oct 30 21:00:06.909631 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:00:06.909606    5713 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-781058-29889900-tfdx9_odh-ai-gateway-infra(10eec293-a840-4727-ade8-d2afa45e9f9a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-781058-29889900-tfdx9" podUID="10eec293-a840-4727-ade8-d2afa45e9f9a"
Oct 30 21:00:06.911538 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:06.911524    5713 scope.go:122] "RemoveContainer" containerID="65a86913089e9ad50dafd0fdee5e35881e1026ecfa89ed7677fda9d906ffbcf0"
Oct 30 21:00:06.912023 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:00:06.912003    5713 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889900-dtz4k_odh-ai-gateway-infra(280ab52d-2d8d-4fe7-b293-73f470d3cd28)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889900-dtz4k" podUID="280ab52d-2d8d-4fe7-b293-73f470d3cd28"
Oct 30 21:00:06.914604 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:06.914586    5713 generic.go:356] "Generic (PLEG): container finished" podID="c0df799c-ee0f-45c0-bbe2-c7928deb0e3b" containerID="38984b6e6bd5c7d93ba0fad6cd09169749ea72f4a6e93a069d5bdc90d52af096" exitCode=60
Oct 30 21:00:06.914745 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:06.914731    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889900-hm4sk" event={"ID":"c0df799c-ee0f-45c0-bbe2-c7928deb0e3b","Type":"ContainerDied","Data":"38984b6e6bd5c7d93ba0fad6cd09169749ea72f4a6e93a069d5bdc90d52af096"}
Oct 30 21:00:06.914869 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:06.914854    5713 scope.go:122] "RemoveContainer" containerID="b306a5f351c8f9b5f67203274e5d5ab0f9e9553afd82647b49f7f60d5dde3417"
Oct 30 21:00:06.915215 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:06.915192    5713 scope.go:122] "RemoveContainer" containerID="38984b6e6bd5c7d93ba0fad6cd09169749ea72f4a6e93a069d5bdc90d52af096"
Oct 30 21:00:06.915523 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:00:06.915500    5713 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889900-hm4sk_odh-ai-gateway-infra(c0df799c-ee0f-45c0-bbe2-c7928deb0e3b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889900-hm4sk" podUID="c0df799c-ee0f-45c0-bbe2-c7928deb0e3b"
Oct 30 21:00:06.917226 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:06.917208    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj" event={"ID":"4b4b9def-33e5-4d81-a159-1389d71ced98","Type":"ContainerStarted","Data":"f4a8e6c8b5d4bc22f229aea1dd9f16222b97e03f0028e172d56130e7d53ffe48"}
Oct 30 21:00:06.917357 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:06.917344    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj" event={"ID":"4b4b9def-33e5-4d81-a159-1389d71ced98","Type":"ContainerStarted","Data":"be792cf26fcfb01d8ba3a9b42146c24e105e6e9c07a4397e5c86840732a362fd"}
Oct 30 21:00:07.426153 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:07.426111    5713 patch_prober.go:29] interesting pod/e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 30 21:00:07.426346 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:07.426177    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff" podUID="c58cfb06-4eaa-4e30-8382-ab66354eec5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 30 21:00:07.467812 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:07.467046    5713 patch_prober.go:29] interesting pod/e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 30 21:00:07.467812 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:07.467103    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj" podUID="abfc50b7-cb57-4beb-bed1-70204a890a8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 30 21:00:07.503811 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:07.502761    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj"
Oct 30 21:00:07.503995 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:07.503862    5713 patch_prober.go:29] interesting pod/e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 30 21:00:07.503995 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:07.503938    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj" podUID="4b4b9def-33e5-4d81-a159-1389d71ced98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 30 21:00:07.570813 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:07.555127    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff" podStartSLOduration=6.555104554 podStartE2EDuration="6.555104554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:07.54787658 +0000 UTC m=+1290.612652978" watchObservedRunningTime="2026-10-30 21:00:07.555104554 +0000 UTC m=+1290.619880953"
Oct 30 21:00:08.332715 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:08.327253    5713 scope.go:122] "RemoveContainer" containerID="38984b6e6bd5c7d93ba0fad6cd09169749ea72f4a6e93a069d5bdc90d52af096"
Oct 30 21:00:08.332715 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:00:08.327578    5713 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889900-hm4sk_odh-ai-gateway-infra(c0df799c-ee0f-45c0-bbe2-c7928deb0e3b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889900-hm4sk" podUID="c0df799c-ee0f-45c0-bbe2-c7928deb0e3b"
Oct 30 21:00:08.425523 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:08.425492    5713 patch_prober.go:29] interesting pod/e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 30 21:00:08.425674 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:08.425568    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff" podUID="c58cfb06-4eaa-4e30-8382-ab66354eec5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 30 21:00:08.466392 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:08.466353    5713 patch_prober.go:29] interesting pod/e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 30 21:00:08.466752 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:08.466722    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj" podUID="abfc50b7-cb57-4beb-bed1-70204a890a8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 30 21:00:08.486382 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:08.486350    5713 patch_prober.go:29] interesting pod/e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 30 21:00:08.486661 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:08.486619    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj" podUID="4b4b9def-33e5-4d81-a159-1389d71ced98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 30 21:00:09.176384 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:09.176317    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj" podStartSLOduration=7.176297361 podStartE2EDuration="7.176297361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:09.174364921 +0000 UTC m=+1292.239141320" watchObservedRunningTime="2026-10-30 21:00:09.176297361 +0000 UTC m=+1292.241073759"
Oct 30 21:00:09.425900 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:09.425863    5713 patch_prober.go:29] interesting pod/e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 30 21:00:09.426288 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:09.425914    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff" podUID="c58cfb06-4eaa-4e30-8382-ab66354eec5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 30 21:00:09.472920 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:09.470895    5713 patch_prober.go:29] interesting pod/e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 30 21:00:09.472920 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:09.470954    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj" podUID="abfc50b7-cb57-4beb-bed1-70204a890a8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 30 21:00:09.495842 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:09.493203    5713 patch_prober.go:29] interesting pod/e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 30 21:00:09.495842 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:09.493257    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj" podUID="4b4b9def-33e5-4d81-a159-1389d71ced98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 30 21:00:10.425973 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:10.425925    5713 patch_prober.go:29] interesting pod/e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 30 21:00:10.426458 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:10.426007    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff" podUID="c58cfb06-4eaa-4e30-8382-ab66354eec5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 30 21:00:10.488818 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:10.488283    5713 patch_prober.go:29] interesting pod/e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 30 21:00:10.488818 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:10.488360    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj" podUID="4b4b9def-33e5-4d81-a159-1389d71ced98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 30 21:00:11.152813 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:11.150122    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj"
Oct 30 21:00:11.152813 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:11.151734    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj"
Oct 30 21:00:11.153252 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:11.153234    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj"
Oct 30 21:00:11.427352 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:11.427133    5713 patch_prober.go:29] interesting pod/e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 30 21:00:11.427352 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:11.427208    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff" podUID="c58cfb06-4eaa-4e30-8382-ab66354eec5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 30 21:00:11.491924 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:11.491872    5713 patch_prober.go:29] interesting pod/e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 30 21:00:11.492140 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:11.491942    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj" podUID="4b4b9def-33e5-4d81-a159-1389d71ced98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 30 21:00:12.426425 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:12.426389    5713 patch_prober.go:29] interesting pod/e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 30 21:00:12.426592 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:12.426459    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff" podUID="c58cfb06-4eaa-4e30-8382-ab66354eec5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 30 21:00:12.485315 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:12.485277    5713 patch_prober.go:29] interesting pod/e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 30 21:00:12.485691 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:12.485358    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj" podUID="4b4b9def-33e5-4d81-a159-1389d71ced98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 30 21:00:13.431663 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:13.431628    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff"
Oct 30 21:00:13.432519 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:13.432494    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff"
Oct 30 21:00:13.433216 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:13.433196    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff"
Oct 30 21:00:13.485038 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:13.485003    5713 patch_prober.go:29] interesting pod/e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 30 21:00:13.485244 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:13.485066    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj" podUID="4b4b9def-33e5-4d81-a159-1389d71ced98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 30 21:00:14.045462 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:14.045426    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-590deeb6-s2789"]
Oct 30 21:00:14.052571 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:14.052544    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-590deeb6-s2789"]
Oct 30 21:00:14.489955 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:14.489927    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj"
Oct 30 21:00:14.490423 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:14.490399    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj"
Oct 30 21:00:14.491047 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:14.491031    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj"
Oct 30 21:00:15.052803 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:15.052755    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bad2ec09-tjpll"]
Oct 30 21:00:15.063911 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:15.063883    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f51aea6-3cbd-4c94-b1d6-589a3ca05cc5" path="/var/lib/kubelet/pods/3f51aea6-3cbd-4c94-b1d6-589a3ca05cc5/volumes"
Oct 30 21:00:15.064309 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:15.064292    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bad2ec09-tjpll"]
Oct 30 21:00:15.932776 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:15.932747    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f22a82ec-fb4ps"]
Oct 30 21:00:15.938261 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:15.938242    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22a82ec-fb4ps"
Oct 30 21:00:15.994764 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:15.994731    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f22a82ec-fb4ps"]
Oct 30 21:00:16.009086 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:16.009043    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdd58401-d3df-4fa1-a8bd-68c8abccc563-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f22a82ec-fb4ps\" (UID: \"cdd58401-d3df-4fa1-a8bd-68c8abccc563\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22a82ec-fb4ps"
Oct 30 21:00:16.009275 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:16.009137    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwsxh\" (UniqueName: \"kubernetes.io/projected/cdd58401-d3df-4fa1-a8bd-68c8abccc563-kube-api-access-xwsxh\") pod \"maas-api-revoke-sub-f22a82ec-fb4ps\" (UID: \"cdd58401-d3df-4fa1-a8bd-68c8abccc563\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22a82ec-fb4ps"
Oct 30 21:00:16.009275 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:16.009174    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdd58401-d3df-4fa1-a8bd-68c8abccc563-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f22a82ec-fb4ps\" (UID: \"cdd58401-d3df-4fa1-a8bd-68c8abccc563\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22a82ec-fb4ps"
Oct 30 21:00:16.038856 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:16.038812    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:00:16.110305 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:16.110268    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdd58401-d3df-4fa1-a8bd-68c8abccc563-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f22a82ec-fb4ps\" (UID: \"cdd58401-d3df-4fa1-a8bd-68c8abccc563\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22a82ec-fb4ps"
Oct 30 21:00:16.110777 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:16.110340    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwsxh\" (UniqueName: \"kubernetes.io/projected/cdd58401-d3df-4fa1-a8bd-68c8abccc563-kube-api-access-xwsxh\") pod \"maas-api-revoke-sub-f22a82ec-fb4ps\" (UID: \"cdd58401-d3df-4fa1-a8bd-68c8abccc563\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22a82ec-fb4ps"
Oct 30 21:00:16.110777 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:16.110378    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdd58401-d3df-4fa1-a8bd-68c8abccc563-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f22a82ec-fb4ps\" (UID: \"cdd58401-d3df-4fa1-a8bd-68c8abccc563\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22a82ec-fb4ps"
Oct 30 21:00:16.111070 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:16.111049    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdd58401-d3df-4fa1-a8bd-68c8abccc563-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f22a82ec-fb4ps\" (UID: \"cdd58401-d3df-4fa1-a8bd-68c8abccc563\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22a82ec-fb4ps"
Oct 30 21:00:16.112924 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:16.112901    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdd58401-d3df-4fa1-a8bd-68c8abccc563-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f22a82ec-fb4ps\" (UID: \"cdd58401-d3df-4fa1-a8bd-68c8abccc563\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22a82ec-fb4ps"
Oct 30 21:00:16.167921 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:16.167893    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwsxh\" (UniqueName: \"kubernetes.io/projected/cdd58401-d3df-4fa1-a8bd-68c8abccc563-kube-api-access-xwsxh\") pod \"maas-api-revoke-sub-f22a82ec-fb4ps\" (UID: \"cdd58401-d3df-4fa1-a8bd-68c8abccc563\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22a82ec-fb4ps"
Oct 30 21:00:16.251882 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:16.251809    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22a82ec-fb4ps"
Oct 30 21:00:16.639179 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:16.636861    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f22a82ec-fb4ps"]
Oct 30 21:00:16.639179 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:00:16.637697    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdd58401_d3df_4fa1_a8bd_68c8abccc563.slice/crio-6c0519ec8d4a2aee4b50f11e7790ea75555d60d55b32eeb3d29e23d1c32f5d9f WatchSource:0}: Error finding container 6c0519ec8d4a2aee4b50f11e7790ea75555d60d55b32eeb3d29e23d1c32f5d9f: Status 404 returned error can't find the container with id 6c0519ec8d4a2aee4b50f11e7790ea75555d60d55b32eeb3d29e23d1c32f5d9f
Oct 30 21:00:17.064445 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:17.064407    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd57ccd2-d0d7-491a-8545-12c942aea379" path="/var/lib/kubelet/pods/dd57ccd2-d0d7-491a-8545-12c942aea379/volumes"
Oct 30 21:00:17.065168 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:17.065153    5713 scope.go:122] "RemoveContainer" containerID="30729bbc4c99bc320a3f5dc589f67af341ac8a7f4b685f5c8fe509115d6eb3d2"
Oct 30 21:00:17.438179 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:17.438147    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d061d0-29889900-sp86r" event={"ID":"a8371114-ea5d-4cbc-b761-5aa5cdfef8fd","Type":"ContainerStarted","Data":"751a7695d1006e927081b959055735a3a304700b092ab6a58d4a7c8ac7449634"}
Oct 30 21:00:17.440295 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:17.440265    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22a82ec-fb4ps" event={"ID":"cdd58401-d3df-4fa1-a8bd-68c8abccc563","Type":"ContainerStarted","Data":"cb3d45c5f447e45ad4b8beead70d6c86aa78662ee299d59fee99170a6eac06d0"}
Oct 30 21:00:17.440420 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:17.440323    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22a82ec-fb4ps" event={"ID":"cdd58401-d3df-4fa1-a8bd-68c8abccc563","Type":"ContainerStarted","Data":"6c0519ec8d4a2aee4b50f11e7790ea75555d60d55b32eeb3d29e23d1c32f5d9f"}
Oct 30 21:00:17.554209 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:17.554158    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d061d0-29889900-sp86r" podStartSLOduration=17.55414176 podStartE2EDuration="17.55414176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:17.551526992 +0000 UTC m=+1300.616303390" watchObservedRunningTime="2026-10-30 21:00:17.55414176 +0000 UTC m=+1300.618918158"
Oct 30 21:00:17.668953 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:17.668900    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22a82ec-fb4ps" podStartSLOduration=2.668887617 podStartE2EDuration="2.668887617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:17.665349536 +0000 UTC m=+1300.730125934" watchObservedRunningTime="2026-10-30 21:00:17.668887617 +0000 UTC m=+1300.733664094"
Oct 30 21:00:17.843204 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:17.843127    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm"]
Oct 30 21:00:17.847629 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:17.847608    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm"
Oct 30 21:00:17.907556 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:17.907523    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm"]
Oct 30 21:00:17.940957 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:17.940924    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9709b238-b5a6-4928-afe5-2337662a1e4e-istio-token\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm\" (UID: \"9709b238-b5a6-4928-afe5-2337662a1e4e\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm"
Oct 30 21:00:17.941110 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:17.940989    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9709b238-b5a6-4928-afe5-2337662a1e4e-workload-socket\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm\" (UID: \"9709b238-b5a6-4928-afe5-2337662a1e4e\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm"
Oct 30 21:00:17.941110 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:17.941085    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rljrv\" (UniqueName: \"kubernetes.io/projected/9709b238-b5a6-4928-afe5-2337662a1e4e-kube-api-access-rljrv\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm\" (UID: \"9709b238-b5a6-4928-afe5-2337662a1e4e\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm"
Oct 30 21:00:17.941223 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:17.941144    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9709b238-b5a6-4928-afe5-2337662a1e4e-workload-certs\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm\" (UID: \"9709b238-b5a6-4928-afe5-2337662a1e4e\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm"
Oct 30 21:00:17.941223 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:17.941173    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9709b238-b5a6-4928-afe5-2337662a1e4e-credential-socket\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm\" (UID: \"9709b238-b5a6-4928-afe5-2337662a1e4e\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm"
Oct 30 21:00:17.941223 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:17.941202    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9709b238-b5a6-4928-afe5-2337662a1e4e-istio-envoy\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm\" (UID: \"9709b238-b5a6-4928-afe5-2337662a1e4e\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm"
Oct 30 21:00:17.941378 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:17.941246    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9709b238-b5a6-4928-afe5-2337662a1e4e-istio-data\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm\" (UID: \"9709b238-b5a6-4928-afe5-2337662a1e4e\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm"
Oct 30 21:00:17.941378 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:17.941271    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9709b238-b5a6-4928-afe5-2337662a1e4e-istio-podinfo\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm\" (UID: \"9709b238-b5a6-4928-afe5-2337662a1e4e\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm"
Oct 30 21:00:17.941378 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:17.941349    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9709b238-b5a6-4928-afe5-2337662a1e4e-istiod-ca-cert\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm\" (UID: \"9709b238-b5a6-4928-afe5-2337662a1e4e\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm"
Oct 30 21:00:18.042900 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:18.042863    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9709b238-b5a6-4928-afe5-2337662a1e4e-istiod-ca-cert\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm\" (UID: \"9709b238-b5a6-4928-afe5-2337662a1e4e\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm"
Oct 30 21:00:18.043141 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:18.043125    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9709b238-b5a6-4928-afe5-2337662a1e4e-istio-token\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm\" (UID: \"9709b238-b5a6-4928-afe5-2337662a1e4e\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm"
Oct 30 21:00:18.043188 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:18.043172    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9709b238-b5a6-4928-afe5-2337662a1e4e-workload-socket\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm\" (UID: \"9709b238-b5a6-4928-afe5-2337662a1e4e\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm"
Oct 30 21:00:18.043332 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:18.043304    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rljrv\" (UniqueName: \"kubernetes.io/projected/9709b238-b5a6-4928-afe5-2337662a1e4e-kube-api-access-rljrv\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm\" (UID: \"9709b238-b5a6-4928-afe5-2337662a1e4e\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm"
Oct 30 21:00:18.043402 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:18.043385    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9709b238-b5a6-4928-afe5-2337662a1e4e-workload-certs\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm\" (UID: \"9709b238-b5a6-4928-afe5-2337662a1e4e\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm"
Oct 30 21:00:18.043458 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:18.043418    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9709b238-b5a6-4928-afe5-2337662a1e4e-credential-socket\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm\" (UID: \"9709b238-b5a6-4928-afe5-2337662a1e4e\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm"
Oct 30 21:00:18.043458 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:18.043451    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9709b238-b5a6-4928-afe5-2337662a1e4e-istio-envoy\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm\" (UID: \"9709b238-b5a6-4928-afe5-2337662a1e4e\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm"
Oct 30 21:00:18.043557 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:18.043502    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9709b238-b5a6-4928-afe5-2337662a1e4e-istio-data\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm\" (UID: \"9709b238-b5a6-4928-afe5-2337662a1e4e\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm"
Oct 30 21:00:18.043557 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:18.043530    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9709b238-b5a6-4928-afe5-2337662a1e4e-istio-podinfo\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm\" (UID: \"9709b238-b5a6-4928-afe5-2337662a1e4e\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm"
Oct 30 21:00:18.043557 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:18.043549    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9709b238-b5a6-4928-afe5-2337662a1e4e-workload-socket\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm\" (UID: \"9709b238-b5a6-4928-afe5-2337662a1e4e\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm"
Oct 30 21:00:18.043712 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:18.043517    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9709b238-b5a6-4928-afe5-2337662a1e4e-istiod-ca-cert\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm\" (UID: \"9709b238-b5a6-4928-afe5-2337662a1e4e\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm"
Oct 30 21:00:18.043762 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:18.043725    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9709b238-b5a6-4928-afe5-2337662a1e4e-workload-certs\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm\" (UID: \"9709b238-b5a6-4928-afe5-2337662a1e4e\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm"
Oct 30 21:00:18.043988 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:18.043966    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9709b238-b5a6-4928-afe5-2337662a1e4e-credential-socket\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm\" (UID: \"9709b238-b5a6-4928-afe5-2337662a1e4e\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm"
Oct 30 21:00:18.044044 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:18.043981    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9709b238-b5a6-4928-afe5-2337662a1e4e-istio-data\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm\" (UID: \"9709b238-b5a6-4928-afe5-2337662a1e4e\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm"
Oct 30 21:00:18.045761 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:18.045737    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9709b238-b5a6-4928-afe5-2337662a1e4e-istio-envoy\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm\" (UID: \"9709b238-b5a6-4928-afe5-2337662a1e4e\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm"
Oct 30 21:00:18.046046 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:18.046026    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9709b238-b5a6-4928-afe5-2337662a1e4e-istio-podinfo\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm\" (UID: \"9709b238-b5a6-4928-afe5-2337662a1e4e\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm"
Oct 30 21:00:18.056054 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:18.056035    5713 scope.go:122] "RemoveContainer" containerID="fdfbe2a1e6a4970d2b8720c7948a02ed5d9b6feb84847141ee30d76399ef5b2a"
Oct 30 21:00:18.056203 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:18.056188    5713 scope.go:122] "RemoveContainer" containerID="3eb2e919ab636810c98a7f0a22f27b03bde36c9e067d74086847375411150f06"
Oct 30 21:00:18.056329 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:18.056312    5713 scope.go:122] "RemoveContainer" containerID="65a86913089e9ad50dafd0fdee5e35881e1026ecfa89ed7677fda9d906ffbcf0"
Oct 30 21:00:18.100864 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:18.100824    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9709b238-b5a6-4928-afe5-2337662a1e4e-istio-token\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm\" (UID: \"9709b238-b5a6-4928-afe5-2337662a1e4e\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm"
Oct 30 21:00:18.101040 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:18.101024    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rljrv\" (UniqueName: \"kubernetes.io/projected/9709b238-b5a6-4928-afe5-2337662a1e4e-kube-api-access-rljrv\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm\" (UID: \"9709b238-b5a6-4928-afe5-2337662a1e4e\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm"
Oct 30 21:00:18.162440 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:18.162412    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm"
Oct 30 21:00:18.452534 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:18.452503    5713 generic.go:356] "Generic (PLEG): container finished" podID="a8371114-ea5d-4cbc-b761-5aa5cdfef8fd" containerID="751a7695d1006e927081b959055735a3a304700b092ab6a58d4a7c8ac7449634" exitCode=60
Oct 30 21:00:18.453084 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:18.452569    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d061d0-29889900-sp86r" event={"ID":"a8371114-ea5d-4cbc-b761-5aa5cdfef8fd","Type":"ContainerDied","Data":"751a7695d1006e927081b959055735a3a304700b092ab6a58d4a7c8ac7449634"}
Oct 30 21:00:18.453171 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:18.453137    5713 scope.go:122] "RemoveContainer" containerID="30729bbc4c99bc320a3f5dc589f67af341ac8a7f4b685f5c8fe509115d6eb3d2"
Oct 30 21:00:18.453516 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:18.453495    5713 scope.go:122] "RemoveContainer" containerID="751a7695d1006e927081b959055735a3a304700b092ab6a58d4a7c8ac7449634"
Oct 30 21:00:18.453880 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:00:18.453856    5713 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-d061d0-29889900-sp86r_odh-ai-gateway-infra(a8371114-ea5d-4cbc-b761-5aa5cdfef8fd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d061d0-29889900-sp86r" podUID="a8371114-ea5d-4cbc-b761-5aa5cdfef8fd"
Oct 30 21:00:18.454933 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:18.454919    5713 generic.go:356] "Generic (PLEG): container finished" podID="cdd58401-d3df-4fa1-a8bd-68c8abccc563" containerID="cb3d45c5f447e45ad4b8beead70d6c86aa78662ee299d59fee99170a6eac06d0" exitCode=0
Oct 30 21:00:18.455023 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:18.454966    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22a82ec-fb4ps" event={"ID":"cdd58401-d3df-4fa1-a8bd-68c8abccc563","Type":"ContainerDied","Data":"cb3d45c5f447e45ad4b8beead70d6c86aa78662ee299d59fee99170a6eac06d0"}
Oct 30 21:00:18.564056 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:18.564018    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d061d0-29889900-sp86r"]
Oct 30 21:00:18.614585 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:00:18.614549    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9709b238_b5a6_4928_afe5_2337662a1e4e.slice/crio-dd201631197fecda22f6b2d3986cecdc70a14395f17775f45af239655e393f06 WatchSource:0}: Error finding container dd201631197fecda22f6b2d3986cecdc70a14395f17775f45af239655e393f06: Status 404 returned error can't find the container with id dd201631197fecda22f6b2d3986cecdc70a14395f17775f45af239655e393f06
Oct 30 21:00:18.617475 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:18.617441    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:00:18.617575 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:18.617511    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:00:18.617575 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:18.617548    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:00:18.622075 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:18.622057    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm"]
Oct 30 21:00:19.067407 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:19.067332    5713 scope.go:122] "RemoveContainer" containerID="38984b6e6bd5c7d93ba0fad6cd09169749ea72f4a6e93a069d5bdc90d52af096"
Oct 30 21:00:19.471548 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:19.471511    5713 generic.go:356] "Generic (PLEG): container finished" podID="76aed203-0d7c-410e-84bd-637f57a1620a" containerID="8a551fbf78763378f6803bf1a93de2346c8a630cda4f3becb26136b3d703ae64" exitCode=60
Oct 30 21:00:19.472069 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:19.471577    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b425d-29889900-sz9cc" event={"ID":"76aed203-0d7c-410e-84bd-637f57a1620a","Type":"ContainerDied","Data":"8a551fbf78763378f6803bf1a93de2346c8a630cda4f3becb26136b3d703ae64"}
Oct 30 21:00:19.472069 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:19.471673    5713 scope.go:122] "RemoveContainer" containerID="fdfbe2a1e6a4970d2b8720c7948a02ed5d9b6feb84847141ee30d76399ef5b2a"
Oct 30 21:00:19.472069 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:19.472057    5713 scope.go:122] "RemoveContainer" containerID="8a551fbf78763378f6803bf1a93de2346c8a630cda4f3becb26136b3d703ae64"
Oct 30 21:00:19.472424 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:00:19.472398    5713 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-2b425d-29889900-sz9cc_odh-ai-gateway-infra(76aed203-0d7c-410e-84bd-637f57a1620a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b425d-29889900-sz9cc" podUID="76aed203-0d7c-410e-84bd-637f57a1620a"
Oct 30 21:00:19.477714 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:19.477584    5713 generic.go:356] "Generic (PLEG): container finished" podID="10eec293-a840-4727-ade8-d2afa45e9f9a" containerID="0b04c9d533a9d934fea81aafcc351f0439d985ad58eb9a2b76aeb243bff8d2f9" exitCode=60
Oct 30 21:00:19.477714 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:19.477635    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-781058-29889900-tfdx9" event={"ID":"10eec293-a840-4727-ade8-d2afa45e9f9a","Type":"ContainerDied","Data":"0b04c9d533a9d934fea81aafcc351f0439d985ad58eb9a2b76aeb243bff8d2f9"}
Oct 30 21:00:19.478108 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:19.478090    5713 scope.go:122] "RemoveContainer" containerID="0b04c9d533a9d934fea81aafcc351f0439d985ad58eb9a2b76aeb243bff8d2f9"
Oct 30 21:00:19.478436 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:00:19.478414    5713 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-781058-29889900-tfdx9_odh-ai-gateway-infra(10eec293-a840-4727-ade8-d2afa45e9f9a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-781058-29889900-tfdx9" podUID="10eec293-a840-4727-ade8-d2afa45e9f9a"
Oct 30 21:00:19.480967 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:19.480937    5713 generic.go:356] "Generic (PLEG): container finished" podID="280ab52d-2d8d-4fe7-b293-73f470d3cd28" containerID="f8637ab6c9fdaa7b26f396f44a1744303c43d07bdad4d7751902a9d9278bda88" exitCode=60
Oct 30 21:00:19.481125 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:19.481044    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889900-dtz4k" event={"ID":"280ab52d-2d8d-4fe7-b293-73f470d3cd28","Type":"ContainerDied","Data":"f8637ab6c9fdaa7b26f396f44a1744303c43d07bdad4d7751902a9d9278bda88"}
Oct 30 21:00:19.481496 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:19.481475    5713 scope.go:122] "RemoveContainer" containerID="f8637ab6c9fdaa7b26f396f44a1744303c43d07bdad4d7751902a9d9278bda88"
Oct 30 21:00:19.481969 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:00:19.481933    5713 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889900-dtz4k_odh-ai-gateway-infra(280ab52d-2d8d-4fe7-b293-73f470d3cd28)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889900-dtz4k" podUID="280ab52d-2d8d-4fe7-b293-73f470d3cd28"
Oct 30 21:00:19.483255 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:19.483229    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm" event={"ID":"9709b238-b5a6-4928-afe5-2337662a1e4e","Type":"ContainerStarted","Data":"e47639f67d5c1a6cf67c24a49803bf129d584d81998c2b3ae44e8db550423eec"}
Oct 30 21:00:19.483349 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:19.483281    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm" event={"ID":"9709b238-b5a6-4928-afe5-2337662a1e4e","Type":"ContainerStarted","Data":"dd201631197fecda22f6b2d3986cecdc70a14395f17775f45af239655e393f06"}
Oct 30 21:00:19.493781 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:19.493759    5713 scope.go:122] "RemoveContainer" containerID="3eb2e919ab636810c98a7f0a22f27b03bde36c9e067d74086847375411150f06"
Oct 30 21:00:19.529446 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:19.529274    5713 scope.go:122] "RemoveContainer" containerID="65a86913089e9ad50dafd0fdee5e35881e1026ecfa89ed7677fda9d906ffbcf0"
Oct 30 21:00:19.715148 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:19.715118    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d061d0-29889900-sp86r"
Oct 30 21:00:19.777636 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:19.777610    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22a82ec-fb4ps"
Oct 30 21:00:19.876467 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:19.876378    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8371114-ea5d-4cbc-b761-5aa5cdfef8fd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8371114-ea5d-4cbc-b761-5aa5cdfef8fd-maas-api-service-ca\") pod \"a8371114-ea5d-4cbc-b761-5aa5cdfef8fd\" (UID: \"a8371114-ea5d-4cbc-b761-5aa5cdfef8fd\") "
Oct 30 21:00:19.876467 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:19.876421    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdd58401-d3df-4fa1-a8bd-68c8abccc563-kube-api-access-xwsxh\" (UniqueName: \"kubernetes.io/projected/cdd58401-d3df-4fa1-a8bd-68c8abccc563-kube-api-access-xwsxh\") pod \"cdd58401-d3df-4fa1-a8bd-68c8abccc563\" (UID: \"cdd58401-d3df-4fa1-a8bd-68c8abccc563\") "
Oct 30 21:00:19.876467 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:19.876465    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdd58401-d3df-4fa1-a8bd-68c8abccc563-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdd58401-d3df-4fa1-a8bd-68c8abccc563-maas-api-xks-ca\") pod \"cdd58401-d3df-4fa1-a8bd-68c8abccc563\" (UID: \"cdd58401-d3df-4fa1-a8bd-68c8abccc563\") "
Oct 30 21:00:19.876776 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:19.876574    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8371114-ea5d-4cbc-b761-5aa5cdfef8fd-kube-api-access-rtssh\" (UniqueName: \"kubernetes.io/projected/a8371114-ea5d-4cbc-b761-5aa5cdfef8fd-kube-api-access-rtssh\") pod \"a8371114-ea5d-4cbc-b761-5aa5cdfef8fd\" (UID: \"a8371114-ea5d-4cbc-b761-5aa5cdfef8fd\") "
Oct 30 21:00:19.876881 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:19.876781    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8371114-ea5d-4cbc-b761-5aa5cdfef8fd-maas-api-service-ca" pod "a8371114-ea5d-4cbc-b761-5aa5cdfef8fd" (UID: "a8371114-ea5d-4cbc-b761-5aa5cdfef8fd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:19.876881 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:19.876838    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdd58401-d3df-4fa1-a8bd-68c8abccc563-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdd58401-d3df-4fa1-a8bd-68c8abccc563-maas-api-openshift-ca\") pod \"cdd58401-d3df-4fa1-a8bd-68c8abccc563\" (UID: \"cdd58401-d3df-4fa1-a8bd-68c8abccc563\") "
Oct 30 21:00:19.877278 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:19.877253    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdd58401-d3df-4fa1-a8bd-68c8abccc563-maas-api-openshift-ca" pod "cdd58401-d3df-4fa1-a8bd-68c8abccc563" (UID: "cdd58401-d3df-4fa1-a8bd-68c8abccc563"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:19.877697 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:19.877672    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdd58401-d3df-4fa1-a8bd-68c8abccc563-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:19.877824 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:19.877703    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8371114-ea5d-4cbc-b761-5aa5cdfef8fd-maas-api-service-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:19.879034 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:19.878992    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8371114-ea5d-4cbc-b761-5aa5cdfef8fd-kube-api-access-rtssh" pod "a8371114-ea5d-4cbc-b761-5aa5cdfef8fd" (UID: "a8371114-ea5d-4cbc-b761-5aa5cdfef8fd"). InnerVolumeSpecName "kube-api-access-rtssh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:19.879160 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:19.879046    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdd58401-d3df-4fa1-a8bd-68c8abccc563-kube-api-access-xwsxh" pod "cdd58401-d3df-4fa1-a8bd-68c8abccc563" (UID: "cdd58401-d3df-4fa1-a8bd-68c8abccc563"). InnerVolumeSpecName "kube-api-access-xwsxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:19.879160 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:19.879135    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdd58401-d3df-4fa1-a8bd-68c8abccc563-maas-api-xks-ca" pod "cdd58401-d3df-4fa1-a8bd-68c8abccc563" (UID: "cdd58401-d3df-4fa1-a8bd-68c8abccc563"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:00:19.979053 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:19.979021    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwsxh\" (UniqueName: \"kubernetes.io/projected/cdd58401-d3df-4fa1-a8bd-68c8abccc563-kube-api-access-xwsxh\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:19.979053 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:19.979050    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdd58401-d3df-4fa1-a8bd-68c8abccc563-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:19.979053 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:19.979060    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtssh\" (UniqueName: \"kubernetes.io/projected/a8371114-ea5d-4cbc-b761-5aa5cdfef8fd-kube-api-access-rtssh\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:20.091603 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:20.091546    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm" podStartSLOduration=3.091531833 podStartE2EDuration="3.091531833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:20.087901914 +0000 UTC m=+1303.152678330" watchObservedRunningTime="2026-10-30 21:00:20.091531833 +0000 UTC m=+1303.156308230"
Oct 30 21:00:20.163630 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:20.163532    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm"
Oct 30 21:00:20.165652 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:20.165622    5713 patch_prober.go:29] interesting pod/e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Oct 30 21:00:20.165817 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:20.165673    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm" podUID="9709b238-b5a6-4928-afe5-2337662a1e4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Oct 30 21:00:20.497103 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:20.497063    5713 generic.go:356] "Generic (PLEG): container finished" podID="c0df799c-ee0f-45c0-bbe2-c7928deb0e3b" containerID="e109482b659907bc3ca60fccb42afaf73c43b05668366a99f7d32f916ec93a69" exitCode=60
Oct 30 21:00:20.497569 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:20.497125    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889900-hm4sk" event={"ID":"c0df799c-ee0f-45c0-bbe2-c7928deb0e3b","Type":"ContainerDied","Data":"e109482b659907bc3ca60fccb42afaf73c43b05668366a99f7d32f916ec93a69"}
Oct 30 21:00:20.497569 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:20.497195    5713 scope.go:122] "RemoveContainer" containerID="38984b6e6bd5c7d93ba0fad6cd09169749ea72f4a6e93a069d5bdc90d52af096"
Oct 30 21:00:20.497724 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:20.497706    5713 scope.go:122] "RemoveContainer" containerID="e109482b659907bc3ca60fccb42afaf73c43b05668366a99f7d32f916ec93a69"
Oct 30 21:00:20.498104 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:00:20.498079    5713 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889900-hm4sk_odh-ai-gateway-infra(c0df799c-ee0f-45c0-bbe2-c7928deb0e3b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889900-hm4sk" podUID="c0df799c-ee0f-45c0-bbe2-c7928deb0e3b"
Oct 30 21:00:20.498855 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:20.498829    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d061d0-29889900-sp86r" event={"ID":"a8371114-ea5d-4cbc-b761-5aa5cdfef8fd","Type":"ContainerDied","Data":"a77542ff4bb79aae3c9b74693038ce0be22eaa4cfb5ac08d946ea9e0cb6b8a9b"}
Oct 30 21:00:20.498855 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:20.498843    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d061d0-29889900-sp86r"
Oct 30 21:00:20.500392 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:20.500374    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22a82ec-fb4ps"
Oct 30 21:00:20.500489 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:20.500376    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22a82ec-fb4ps" event={"ID":"cdd58401-d3df-4fa1-a8bd-68c8abccc563","Type":"ContainerDied","Data":"6c0519ec8d4a2aee4b50f11e7790ea75555d60d55b32eeb3d29e23d1c32f5d9f"}
Oct 30 21:00:20.500556 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:20.500496    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c0519ec8d4a2aee4b50f11e7790ea75555d60d55b32eeb3d29e23d1c32f5d9f"
Oct 30 21:00:20.512329 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:20.512313    5713 scope.go:122] "RemoveContainer" containerID="751a7695d1006e927081b959055735a3a304700b092ab6a58d4a7c8ac7449634"
Oct 30 21:00:20.597669 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:20.597636    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b425d-29889900-sz9cc"]
Oct 30 21:00:20.769997 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:20.769918    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d061d0-29889900-sp86r"]
Oct 30 21:00:20.774089 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:20.774060    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d061d0-29889900-sp86r"]
Oct 30 21:00:20.803677 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:20.803651    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b425d-29889900-sz9cc"
Oct 30 21:00:20.862235 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:20.862201    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-781058-29889900-tfdx9"]
Oct 30 21:00:20.891771 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:20.891739    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76aed203-0d7c-410e-84bd-637f57a1620a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76aed203-0d7c-410e-84bd-637f57a1620a-maas-api-service-ca\") pod \"76aed203-0d7c-410e-84bd-637f57a1620a\" (UID: \"76aed203-0d7c-410e-84bd-637f57a1620a\") "
Oct 30 21:00:20.892367 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:20.892341    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76aed203-0d7c-410e-84bd-637f57a1620a-kube-api-access-gv6wb\" (UniqueName: \"kubernetes.io/projected/76aed203-0d7c-410e-84bd-637f57a1620a-kube-api-access-gv6wb\") pod \"76aed203-0d7c-410e-84bd-637f57a1620a\" (UID: \"76aed203-0d7c-410e-84bd-637f57a1620a\") "
Oct 30 21:00:20.892722 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:20.892370    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76aed203-0d7c-410e-84bd-637f57a1620a-maas-api-service-ca" pod "76aed203-0d7c-410e-84bd-637f57a1620a" (UID: "76aed203-0d7c-410e-84bd-637f57a1620a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:20.893304 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:20.893285    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76aed203-0d7c-410e-84bd-637f57a1620a-maas-api-service-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:20.896241 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:20.895172    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76aed203-0d7c-410e-84bd-637f57a1620a-kube-api-access-gv6wb" pod "76aed203-0d7c-410e-84bd-637f57a1620a" (UID: "76aed203-0d7c-410e-84bd-637f57a1620a"). InnerVolumeSpecName "kube-api-access-gv6wb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:20.986184 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:20.986131    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889900-dtz4k"]
Oct 30 21:00:20.997620 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:20.995008    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gv6wb\" (UniqueName: \"kubernetes.io/projected/76aed203-0d7c-410e-84bd-637f57a1620a-kube-api-access-gv6wb\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:21.077840 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.073276    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-781058-29889900-tfdx9"
Oct 30 21:00:21.077840 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.075832    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8371114-ea5d-4cbc-b761-5aa5cdfef8fd" path="/var/lib/kubelet/pods/a8371114-ea5d-4cbc-b761-5aa5cdfef8fd/volumes"
Oct 30 21:00:21.165815 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.164404    5713 patch_prober.go:29] interesting pod/e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Oct 30 21:00:21.165815 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.164465    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm" podUID="9709b238-b5a6-4928-afe5-2337662a1e4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Oct 30 21:00:21.185883 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.185856    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889900-dtz4k"
Oct 30 21:00:21.198099 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.198078    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10eec293-a840-4727-ade8-d2afa45e9f9a-kube-api-access-xq7hr\" (UniqueName: \"kubernetes.io/projected/10eec293-a840-4727-ade8-d2afa45e9f9a-kube-api-access-xq7hr\") pod \"10eec293-a840-4727-ade8-d2afa45e9f9a\" (UID: \"10eec293-a840-4727-ade8-d2afa45e9f9a\") "
Oct 30 21:00:21.198193 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.198137    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10eec293-a840-4727-ade8-d2afa45e9f9a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10eec293-a840-4727-ade8-d2afa45e9f9a-maas-api-service-ca\") pod \"10eec293-a840-4727-ade8-d2afa45e9f9a\" (UID: \"10eec293-a840-4727-ade8-d2afa45e9f9a\") "
Oct 30 21:00:21.198933 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.198904    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10eec293-a840-4727-ade8-d2afa45e9f9a-maas-api-service-ca" pod "10eec293-a840-4727-ade8-d2afa45e9f9a" (UID: "10eec293-a840-4727-ade8-d2afa45e9f9a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:21.200950 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.200901    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10eec293-a840-4727-ade8-d2afa45e9f9a-kube-api-access-xq7hr" pod "10eec293-a840-4727-ade8-d2afa45e9f9a" (UID: "10eec293-a840-4727-ade8-d2afa45e9f9a"). InnerVolumeSpecName "kube-api-access-xq7hr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:21.265125 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.265079    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba72103f-l4l8h"]
Oct 30 21:00:21.265710 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.265696    5713 cpu_manager.go:414] "containerMap: removing container" podUID="76aed203-0d7c-410e-84bd-637f57a1620a" containerName="cleanup"
Oct 30 21:00:21.265750 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.265711    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76aed203-0d7c-410e-84bd-637f57a1620a" containerName="cleanup"
Oct 30 21:00:21.265750 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.265720    5713 cpu_manager.go:414] "containerMap: removing container" podUID="76aed203-0d7c-410e-84bd-637f57a1620a" containerName="cleanup"
Oct 30 21:00:21.265750 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.265725    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76aed203-0d7c-410e-84bd-637f57a1620a" containerName="cleanup"
Oct 30 21:00:21.265750 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.265733    5713 cpu_manager.go:414] "containerMap: removing container" podUID="10eec293-a840-4727-ade8-d2afa45e9f9a" containerName="cleanup"
Oct 30 21:00:21.265750 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.265738    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10eec293-a840-4727-ade8-d2afa45e9f9a" containerName="cleanup"
Oct 30 21:00:21.265955 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.265754    5713 cpu_manager.go:414] "containerMap: removing container" podUID="a8371114-ea5d-4cbc-b761-5aa5cdfef8fd" containerName="cleanup"
Oct 30 21:00:21.265955 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.265758    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8371114-ea5d-4cbc-b761-5aa5cdfef8fd" containerName="cleanup"
Oct 30 21:00:21.265955 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.265767    5713 cpu_manager.go:414] "containerMap: removing container" podUID="280ab52d-2d8d-4fe7-b293-73f470d3cd28" containerName="cleanup"
Oct 30 21:00:21.265955 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.265771    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="280ab52d-2d8d-4fe7-b293-73f470d3cd28" containerName="cleanup"
Oct 30 21:00:21.265955 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.265800    5713 cpu_manager.go:414] "containerMap: removing container" podUID="280ab52d-2d8d-4fe7-b293-73f470d3cd28" containerName="cleanup"
Oct 30 21:00:21.265955 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.265807    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="280ab52d-2d8d-4fe7-b293-73f470d3cd28" containerName="cleanup"
Oct 30 21:00:21.265955 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.265815    5713 cpu_manager.go:414] "containerMap: removing container" podUID="10eec293-a840-4727-ade8-d2afa45e9f9a" containerName="cleanup"
Oct 30 21:00:21.265955 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.265819    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10eec293-a840-4727-ade8-d2afa45e9f9a" containerName="cleanup"
Oct 30 21:00:21.265955 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.265845    5713 cpu_manager.go:414] "containerMap: removing container" podUID="10eec293-a840-4727-ade8-d2afa45e9f9a" containerName="cleanup"
Oct 30 21:00:21.265955 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.265849    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10eec293-a840-4727-ade8-d2afa45e9f9a" containerName="cleanup"
Oct 30 21:00:21.265955 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.265859    5713 cpu_manager.go:414] "containerMap: removing container" podUID="a8371114-ea5d-4cbc-b761-5aa5cdfef8fd" containerName="cleanup"
Oct 30 21:00:21.265955 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.265863    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8371114-ea5d-4cbc-b761-5aa5cdfef8fd" containerName="cleanup"
Oct 30 21:00:21.265955 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.265871    5713 cpu_manager.go:414] "containerMap: removing container" podUID="280ab52d-2d8d-4fe7-b293-73f470d3cd28" containerName="cleanup"
Oct 30 21:00:21.265955 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.265875    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="280ab52d-2d8d-4fe7-b293-73f470d3cd28" containerName="cleanup"
Oct 30 21:00:21.265955 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.265884    5713 cpu_manager.go:414] "containerMap: removing container" podUID="cdd58401-d3df-4fa1-a8bd-68c8abccc563" containerName="revoke-keys"
Oct 30 21:00:21.265955 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.265888    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdd58401-d3df-4fa1-a8bd-68c8abccc563" containerName="revoke-keys"
Oct 30 21:00:21.265955 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.265895    5713 cpu_manager.go:414] "containerMap: removing container" podUID="a8371114-ea5d-4cbc-b761-5aa5cdfef8fd" containerName="cleanup"
Oct 30 21:00:21.265955 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.265899    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8371114-ea5d-4cbc-b761-5aa5cdfef8fd" containerName="cleanup"
Oct 30 21:00:21.266508 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.265997    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="10eec293-a840-4727-ade8-d2afa45e9f9a" containerName="cleanup"
Oct 30 21:00:21.266508 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.266007    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8371114-ea5d-4cbc-b761-5aa5cdfef8fd" containerName="cleanup"
Oct 30 21:00:21.266508 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.266014    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="76aed203-0d7c-410e-84bd-637f57a1620a" containerName="cleanup"
Oct 30 21:00:21.266508 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.266018    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="280ab52d-2d8d-4fe7-b293-73f470d3cd28" containerName="cleanup"
Oct 30 21:00:21.266508 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.266023    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdd58401-d3df-4fa1-a8bd-68c8abccc563" containerName="revoke-keys"
Oct 30 21:00:21.266508 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.266030    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8371114-ea5d-4cbc-b761-5aa5cdfef8fd" containerName="cleanup"
Oct 30 21:00:21.266508 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.266039    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="76aed203-0d7c-410e-84bd-637f57a1620a" containerName="cleanup"
Oct 30 21:00:21.266508 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.266045    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8371114-ea5d-4cbc-b761-5aa5cdfef8fd" containerName="cleanup"
Oct 30 21:00:21.266508 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.266051    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="280ab52d-2d8d-4fe7-b293-73f470d3cd28" containerName="cleanup"
Oct 30 21:00:21.266508 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.266058    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="10eec293-a840-4727-ade8-d2afa45e9f9a" containerName="cleanup"
Oct 30 21:00:21.269865 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.269846    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba72103f-l4l8h"
Oct 30 21:00:21.299201 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.299169    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/280ab52d-2d8d-4fe7-b293-73f470d3cd28-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/280ab52d-2d8d-4fe7-b293-73f470d3cd28-maas-api-service-ca\") pod \"280ab52d-2d8d-4fe7-b293-73f470d3cd28\" (UID: \"280ab52d-2d8d-4fe7-b293-73f470d3cd28\") "
Oct 30 21:00:21.299385 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.299367    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/280ab52d-2d8d-4fe7-b293-73f470d3cd28-kube-api-access-t9ljw\" (UniqueName: \"kubernetes.io/projected/280ab52d-2d8d-4fe7-b293-73f470d3cd28-kube-api-access-t9ljw\") pod \"280ab52d-2d8d-4fe7-b293-73f470d3cd28\" (UID: \"280ab52d-2d8d-4fe7-b293-73f470d3cd28\") "
Oct 30 21:00:21.299584 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.299563    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/280ab52d-2d8d-4fe7-b293-73f470d3cd28-maas-api-service-ca" pod "280ab52d-2d8d-4fe7-b293-73f470d3cd28" (UID: "280ab52d-2d8d-4fe7-b293-73f470d3cd28"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:21.300016 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.299993    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xq7hr\" (UniqueName: \"kubernetes.io/projected/10eec293-a840-4727-ade8-d2afa45e9f9a-kube-api-access-xq7hr\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:21.300016 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.300018    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10eec293-a840-4727-ade8-d2afa45e9f9a-maas-api-service-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:21.300016 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.300030    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/280ab52d-2d8d-4fe7-b293-73f470d3cd28-maas-api-service-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:21.302002 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.301966    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/280ab52d-2d8d-4fe7-b293-73f470d3cd28-kube-api-access-t9ljw" pod "280ab52d-2d8d-4fe7-b293-73f470d3cd28" (UID: "280ab52d-2d8d-4fe7-b293-73f470d3cd28"). InnerVolumeSpecName "kube-api-access-t9ljw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:21.322013 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.321987    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:00:21.325061 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.325031    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba72103f-l4l8h"]
Oct 30 21:00:21.401653 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.401615    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0482372-b713-458e-aae5-412c81b49544-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba72103f-l4l8h\" (UID: \"c0482372-b713-458e-aae5-412c81b49544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba72103f-l4l8h"
Oct 30 21:00:21.401834 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.401806    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smmtl\" (UniqueName: \"kubernetes.io/projected/c0482372-b713-458e-aae5-412c81b49544-kube-api-access-smmtl\") pod \"maas-api-revoke-sub-ba72103f-l4l8h\" (UID: \"c0482372-b713-458e-aae5-412c81b49544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba72103f-l4l8h"
Oct 30 21:00:21.401953 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.401941    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0482372-b713-458e-aae5-412c81b49544-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba72103f-l4l8h\" (UID: \"c0482372-b713-458e-aae5-412c81b49544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba72103f-l4l8h"
Oct 30 21:00:21.402017 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.402006    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9ljw\" (UniqueName: \"kubernetes.io/projected/280ab52d-2d8d-4fe7-b293-73f470d3cd28-kube-api-access-t9ljw\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:21.503110 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.503073    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0482372-b713-458e-aae5-412c81b49544-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba72103f-l4l8h\" (UID: \"c0482372-b713-458e-aae5-412c81b49544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba72103f-l4l8h"
Oct 30 21:00:21.503578 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.503233    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smmtl\" (UniqueName: \"kubernetes.io/projected/c0482372-b713-458e-aae5-412c81b49544-kube-api-access-smmtl\") pod \"maas-api-revoke-sub-ba72103f-l4l8h\" (UID: \"c0482372-b713-458e-aae5-412c81b49544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba72103f-l4l8h"
Oct 30 21:00:21.503578 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.503384    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0482372-b713-458e-aae5-412c81b49544-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba72103f-l4l8h\" (UID: \"c0482372-b713-458e-aae5-412c81b49544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba72103f-l4l8h"
Oct 30 21:00:21.504229 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.504206    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0482372-b713-458e-aae5-412c81b49544-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba72103f-l4l8h\" (UID: \"c0482372-b713-458e-aae5-412c81b49544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba72103f-l4l8h"
Oct 30 21:00:21.505771 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.505746    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0482372-b713-458e-aae5-412c81b49544-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba72103f-l4l8h\" (UID: \"c0482372-b713-458e-aae5-412c81b49544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba72103f-l4l8h"
Oct 30 21:00:21.521243 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.521214    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b425d-29889900-sz9cc"
Oct 30 21:00:21.521243 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.521221    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b425d-29889900-sz9cc" event={"ID":"76aed203-0d7c-410e-84bd-637f57a1620a","Type":"ContainerDied","Data":"538733c7ac7b7c6b3919115f3ab35a0cea0464321e8588af565db8f87f5447f0"}
Oct 30 21:00:21.521455 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.521288    5713 scope.go:122] "RemoveContainer" containerID="8a551fbf78763378f6803bf1a93de2346c8a630cda4f3becb26136b3d703ae64"
Oct 30 21:00:21.522854 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.522832    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-781058-29889900-tfdx9" event={"ID":"10eec293-a840-4727-ade8-d2afa45e9f9a","Type":"ContainerDied","Data":"2d0bb4ea9bf183f21dba0c404d85b319f154d4bf6f2f5581bbeff6986018c4e7"}
Oct 30 21:00:21.522981 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.522867    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-781058-29889900-tfdx9"
Oct 30 21:00:21.525360 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.525334    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889900-dtz4k" event={"ID":"280ab52d-2d8d-4fe7-b293-73f470d3cd28","Type":"ContainerDied","Data":"d68fb16915d3aed91265833d642a779cc035314244038bbe979a05f370680d41"}
Oct 30 21:00:21.525690 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.525671    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889900-dtz4k"
Oct 30 21:00:21.563718 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.563692    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smmtl\" (UniqueName: \"kubernetes.io/projected/c0482372-b713-458e-aae5-412c81b49544-kube-api-access-smmtl\") pod \"maas-api-revoke-sub-ba72103f-l4l8h\" (UID: \"c0482372-b713-458e-aae5-412c81b49544\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba72103f-l4l8h"
Oct 30 21:00:21.565961 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.565942    5713 scope.go:122] "RemoveContainer" containerID="0b04c9d533a9d934fea81aafcc351f0439d985ad58eb9a2b76aeb243bff8d2f9"
Oct 30 21:00:21.582894 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.582872    5713 scope.go:122] "RemoveContainer" containerID="f8637ab6c9fdaa7b26f396f44a1744303c43d07bdad4d7751902a9d9278bda88"
Oct 30 21:00:21.583541 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.583520    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba72103f-l4l8h"
Oct 30 21:00:21.663811 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.661868    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889900-hm4sk"]
Oct 30 21:00:21.711240 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.709934    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b425d-29889900-sz9cc"]
Oct 30 21:00:21.715890 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.715361    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b425d-29889900-sz9cc"]
Oct 30 21:00:21.825313 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.825284    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-781058-29889900-tfdx9"]
Oct 30 21:00:21.831180 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.831147    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-781058-29889900-tfdx9"]
Oct 30 21:00:21.880130 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.880093    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889900-hm4sk"
Oct 30 21:00:21.964309 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.964273    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889900-dtz4k"]
Oct 30 21:00:21.968762 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.968738    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1fdaa7-29889900-dtz4k"]
Oct 30 21:00:21.996708 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:21.996679    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba72103f-l4l8h"]
Oct 30 21:00:21.997538 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:00:21.997511    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0482372_b713_458e_aae5_412c81b49544.slice/crio-bb80e89c75d70ca544be1af5d18dc34f4cf14da72f37f7388c3482b84c05e594 WatchSource:0}: Error finding container bb80e89c75d70ca544be1af5d18dc34f4cf14da72f37f7388c3482b84c05e594: Status 404 returned error can't find the container with id bb80e89c75d70ca544be1af5d18dc34f4cf14da72f37f7388c3482b84c05e594
Oct 30 21:00:22.017813 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:22.013867    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0df799c-ee0f-45c0-bbe2-c7928deb0e3b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0df799c-ee0f-45c0-bbe2-c7928deb0e3b-maas-api-service-ca\") pod \"c0df799c-ee0f-45c0-bbe2-c7928deb0e3b\" (UID: \"c0df799c-ee0f-45c0-bbe2-c7928deb0e3b\") "
Oct 30 21:00:22.017813 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:22.014003    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0df799c-ee0f-45c0-bbe2-c7928deb0e3b-kube-api-access-8hx59\" (UniqueName: \"kubernetes.io/projected/c0df799c-ee0f-45c0-bbe2-c7928deb0e3b-kube-api-access-8hx59\") pod \"c0df799c-ee0f-45c0-bbe2-c7928deb0e3b\" (UID: \"c0df799c-ee0f-45c0-bbe2-c7928deb0e3b\") "
Oct 30 21:00:22.017813 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:22.014401    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0df799c-ee0f-45c0-bbe2-c7928deb0e3b-maas-api-service-ca" pod "c0df799c-ee0f-45c0-bbe2-c7928deb0e3b" (UID: "c0df799c-ee0f-45c0-bbe2-c7928deb0e3b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:22.017813 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:22.014777    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0df799c-ee0f-45c0-bbe2-c7928deb0e3b-maas-api-service-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:22.017813 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:22.016646    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0df799c-ee0f-45c0-bbe2-c7928deb0e3b-kube-api-access-8hx59" pod "c0df799c-ee0f-45c0-bbe2-c7928deb0e3b" (UID: "c0df799c-ee0f-45c0-bbe2-c7928deb0e3b"). InnerVolumeSpecName "kube-api-access-8hx59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:22.064665 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:22.064625    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03415aa4-pckkr"]
Oct 30 21:00:22.068595 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:22.068561    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03415aa4-pckkr"]
Oct 30 21:00:22.116761 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:22.116618    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hx59\" (UniqueName: \"kubernetes.io/projected/c0df799c-ee0f-45c0-bbe2-c7928deb0e3b-kube-api-access-8hx59\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:22.163966 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:22.163883    5713 patch_prober.go:29] interesting pod/e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Oct 30 21:00:22.163966 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:22.163940    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm" podUID="9709b238-b5a6-4928-afe5-2337662a1e4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Oct 30 21:00:22.557815 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:22.557650    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889900-hm4sk" event={"ID":"c0df799c-ee0f-45c0-bbe2-c7928deb0e3b","Type":"ContainerDied","Data":"a87869e13eb741594ec32ff0637f1015bda349867ac14e1ee995263445a672b7"}
Oct 30 21:00:22.557815 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:22.557730    5713 scope.go:122] "RemoveContainer" containerID="e109482b659907bc3ca60fccb42afaf73c43b05668366a99f7d32f916ec93a69"
Oct 30 21:00:22.558373 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:22.557994    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889900-hm4sk"
Oct 30 21:00:22.584560 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:22.584521    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba72103f-l4l8h" event={"ID":"c0482372-b713-458e-aae5-412c81b49544","Type":"ContainerStarted","Data":"520343b3a94b83fa11e96f6cd390b95e23a29c03d51dc9f0e6d6bb0b2b2fdb4d"}
Oct 30 21:00:22.584738 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:22.584613    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba72103f-l4l8h" event={"ID":"c0482372-b713-458e-aae5-412c81b49544","Type":"ContainerStarted","Data":"bb80e89c75d70ca544be1af5d18dc34f4cf14da72f37f7388c3482b84c05e594"}
Oct 30 21:00:22.726144 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:22.726048    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889900-hm4sk"]
Oct 30 21:00:22.729892 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:22.729867    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889900-hm4sk"]
Oct 30 21:00:22.861082 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:22.861005    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba72103f-l4l8h" podStartSLOduration=1.860982114 podStartE2EDuration="1.860982114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:22.855105138 +0000 UTC m=+1305.919881539" watchObservedRunningTime="2026-10-30 21:00:22.860982114 +0000 UTC m=+1305.925758512"
Oct 30 21:00:23.070276 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:23.070199    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10eec293-a840-4727-ade8-d2afa45e9f9a" path="/var/lib/kubelet/pods/10eec293-a840-4727-ade8-d2afa45e9f9a/volumes"
Oct 30 21:00:23.070716 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:23.070701    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="280ab52d-2d8d-4fe7-b293-73f470d3cd28" path="/var/lib/kubelet/pods/280ab52d-2d8d-4fe7-b293-73f470d3cd28/volumes"
Oct 30 21:00:23.071133 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:23.071119    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76aed203-0d7c-410e-84bd-637f57a1620a" path="/var/lib/kubelet/pods/76aed203-0d7c-410e-84bd-637f57a1620a/volumes"
Oct 30 21:00:23.071843 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:23.071828    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="960607a7-c808-45b5-aa2f-00d454c2dac9" path="/var/lib/kubelet/pods/960607a7-c808-45b5-aa2f-00d454c2dac9/volumes"
Oct 30 21:00:23.072210 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:23.072197    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0df799c-ee0f-45c0-bbe2-c7928deb0e3b" path="/var/lib/kubelet/pods/c0df799c-ee0f-45c0-bbe2-c7928deb0e3b/volumes"
Oct 30 21:00:23.163180 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:23.163145    5713 patch_prober.go:29] interesting pod/e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Oct 30 21:00:23.163373 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:23.163202    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm" podUID="9709b238-b5a6-4928-afe5-2337662a1e4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Oct 30 21:00:23.595207 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:23.595177    5713 generic.go:356] "Generic (PLEG): container finished" podID="c0482372-b713-458e-aae5-412c81b49544" containerID="520343b3a94b83fa11e96f6cd390b95e23a29c03d51dc9f0e6d6bb0b2b2fdb4d" exitCode=0
Oct 30 21:00:23.595654 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:23.595255    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba72103f-l4l8h" event={"ID":"c0482372-b713-458e-aae5-412c81b49544","Type":"ContainerDied","Data":"520343b3a94b83fa11e96f6cd390b95e23a29c03d51dc9f0e6d6bb0b2b2fdb4d"}
Oct 30 21:00:24.163012 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:24.162981    5713 patch_prober.go:29] interesting pod/e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Oct 30 21:00:24.163206 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:24.163041    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm" podUID="9709b238-b5a6-4928-afe5-2337662a1e4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Oct 30 21:00:24.484348 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:00:24.484313    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:00:24.780466 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:24.780439    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba72103f-l4l8h"
Oct 30 21:00:24.875567 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:24.875533    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0482372-b713-458e-aae5-412c81b49544-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0482372-b713-458e-aae5-412c81b49544-maas-api-xks-ca\") pod \"c0482372-b713-458e-aae5-412c81b49544\" (UID: \"c0482372-b713-458e-aae5-412c81b49544\") "
Oct 30 21:00:24.875760 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:24.875680    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0482372-b713-458e-aae5-412c81b49544-kube-api-access-smmtl\" (UniqueName: \"kubernetes.io/projected/c0482372-b713-458e-aae5-412c81b49544-kube-api-access-smmtl\") pod \"c0482372-b713-458e-aae5-412c81b49544\" (UID: \"c0482372-b713-458e-aae5-412c81b49544\") "
Oct 30 21:00:24.875760 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:24.875742    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0482372-b713-458e-aae5-412c81b49544-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0482372-b713-458e-aae5-412c81b49544-maas-api-openshift-ca\") pod \"c0482372-b713-458e-aae5-412c81b49544\" (UID: \"c0482372-b713-458e-aae5-412c81b49544\") "
Oct 30 21:00:24.876189 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:24.876160    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0482372-b713-458e-aae5-412c81b49544-maas-api-openshift-ca" pod "c0482372-b713-458e-aae5-412c81b49544" (UID: "c0482372-b713-458e-aae5-412c81b49544"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:24.876504 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:24.876469    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0482372-b713-458e-aae5-412c81b49544-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:24.877987 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:24.877958    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0482372-b713-458e-aae5-412c81b49544-maas-api-xks-ca" pod "c0482372-b713-458e-aae5-412c81b49544" (UID: "c0482372-b713-458e-aae5-412c81b49544"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:00:24.878220 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:24.878190    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0482372-b713-458e-aae5-412c81b49544-kube-api-access-smmtl" pod "c0482372-b713-458e-aae5-412c81b49544" (UID: "c0482372-b713-458e-aae5-412c81b49544"). InnerVolumeSpecName "kube-api-access-smmtl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:24.978465 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:24.978427    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smmtl\" (UniqueName: \"kubernetes.io/projected/c0482372-b713-458e-aae5-412c81b49544-kube-api-access-smmtl\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:24.978465 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:24.978464    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0482372-b713-458e-aae5-412c81b49544-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:25.163937 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:25.163900    5713 patch_prober.go:29] interesting pod/e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Oct 30 21:00:25.164144 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:25.163964    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm" podUID="9709b238-b5a6-4928-afe5-2337662a1e4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Oct 30 21:00:25.613010 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:25.612977    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba72103f-l4l8h"
Oct 30 21:00:25.613010 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:25.612998    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba72103f-l4l8h" event={"ID":"c0482372-b713-458e-aae5-412c81b49544","Type":"ContainerDied","Data":"bb80e89c75d70ca544be1af5d18dc34f4cf14da72f37f7388c3482b84c05e594"}
Oct 30 21:00:25.613294 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:25.613055    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb80e89c75d70ca544be1af5d18dc34f4cf14da72f37f7388c3482b84c05e594"
Oct 30 21:00:26.167453 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:26.167424    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm"
Oct 30 21:00:26.167952 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:26.167820    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm"
Oct 30 21:00:26.168540 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:26.168518    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm"
Oct 30 21:00:28.119898 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:28.119861    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14e949b0-6rch5"]
Oct 30 21:00:28.120686 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:28.120663    5713 cpu_manager.go:414] "containerMap: removing container" podUID="c0df799c-ee0f-45c0-bbe2-c7928deb0e3b" containerName="cleanup"
Oct 30 21:00:28.120819 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:28.120688    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0df799c-ee0f-45c0-bbe2-c7928deb0e3b" containerName="cleanup"
Oct 30 21:00:28.120819 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:28.120734    5713 cpu_manager.go:414] "containerMap: removing container" podUID="c0482372-b713-458e-aae5-412c81b49544" containerName="revoke-keys"
Oct 30 21:00:28.120819 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:28.120742    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0482372-b713-458e-aae5-412c81b49544" containerName="revoke-keys"
Oct 30 21:00:28.120819 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:28.120805    5713 cpu_manager.go:414] "containerMap: removing container" podUID="76aed203-0d7c-410e-84bd-637f57a1620a" containerName="cleanup"
Oct 30 21:00:28.120819 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:28.120813    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76aed203-0d7c-410e-84bd-637f57a1620a" containerName="cleanup"
Oct 30 21:00:28.121084 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:28.120831    5713 cpu_manager.go:414] "containerMap: removing container" podUID="c0df799c-ee0f-45c0-bbe2-c7928deb0e3b" containerName="cleanup"
Oct 30 21:00:28.121084 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:28.120838    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0df799c-ee0f-45c0-bbe2-c7928deb0e3b" containerName="cleanup"
Oct 30 21:00:28.121084 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:28.121000    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0df799c-ee0f-45c0-bbe2-c7928deb0e3b" containerName="cleanup"
Oct 30 21:00:28.121084 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:28.121015    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0df799c-ee0f-45c0-bbe2-c7928deb0e3b" containerName="cleanup"
Oct 30 21:00:28.121084 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:28.121027    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="10eec293-a840-4727-ade8-d2afa45e9f9a" containerName="cleanup"
Oct 30 21:00:28.121084 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:28.121039    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="76aed203-0d7c-410e-84bd-637f57a1620a" containerName="cleanup"
Oct 30 21:00:28.121084 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:28.121049    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="280ab52d-2d8d-4fe7-b293-73f470d3cd28" containerName="cleanup"
Oct 30 21:00:28.121084 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:28.121060    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0482372-b713-458e-aae5-412c81b49544" containerName="revoke-keys"
Oct 30 21:00:28.124891 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:28.124870    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e949b0-6rch5"
Oct 30 21:00:28.194431 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:28.194389    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14e949b0-6rch5"]
Oct 30 21:00:28.227206 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:28.227156    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c5f608f-fd3f-4180-af2f-e9b162996c67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14e949b0-6rch5\" (UID: \"4c5f608f-fd3f-4180-af2f-e9b162996c67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e949b0-6rch5"
Oct 30 21:00:28.227426 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:28.227366    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c5f608f-fd3f-4180-af2f-e9b162996c67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14e949b0-6rch5\" (UID: \"4c5f608f-fd3f-4180-af2f-e9b162996c67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e949b0-6rch5"
Oct 30 21:00:28.227599 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:28.227569    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk4j9\" (UniqueName: \"kubernetes.io/projected/4c5f608f-fd3f-4180-af2f-e9b162996c67-kube-api-access-rk4j9\") pod \"maas-api-revoke-sub-14e949b0-6rch5\" (UID: \"4c5f608f-fd3f-4180-af2f-e9b162996c67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e949b0-6rch5"
Oct 30 21:00:28.228220 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:28.228195    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:00:28.228345 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:28.228326    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:00:28.228423 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:28.228344    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:00:28.228423 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:28.228409    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:00:28.329070 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:28.329034    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk4j9\" (UniqueName: \"kubernetes.io/projected/4c5f608f-fd3f-4180-af2f-e9b162996c67-kube-api-access-rk4j9\") pod \"maas-api-revoke-sub-14e949b0-6rch5\" (UID: \"4c5f608f-fd3f-4180-af2f-e9b162996c67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e949b0-6rch5"
Oct 30 21:00:28.329268 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:28.329135    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c5f608f-fd3f-4180-af2f-e9b162996c67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14e949b0-6rch5\" (UID: \"4c5f608f-fd3f-4180-af2f-e9b162996c67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e949b0-6rch5"
Oct 30 21:00:28.329268 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:28.329241    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c5f608f-fd3f-4180-af2f-e9b162996c67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14e949b0-6rch5\" (UID: \"4c5f608f-fd3f-4180-af2f-e9b162996c67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e949b0-6rch5"
Oct 30 21:00:28.329800 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:28.329766    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c5f608f-fd3f-4180-af2f-e9b162996c67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14e949b0-6rch5\" (UID: \"4c5f608f-fd3f-4180-af2f-e9b162996c67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e949b0-6rch5"
Oct 30 21:00:28.331722 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:28.331694    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c5f608f-fd3f-4180-af2f-e9b162996c67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14e949b0-6rch5\" (UID: \"4c5f608f-fd3f-4180-af2f-e9b162996c67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e949b0-6rch5"
Oct 30 21:00:28.386091 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:28.386019    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk4j9\" (UniqueName: \"kubernetes.io/projected/4c5f608f-fd3f-4180-af2f-e9b162996c67-kube-api-access-rk4j9\") pod \"maas-api-revoke-sub-14e949b0-6rch5\" (UID: \"4c5f608f-fd3f-4180-af2f-e9b162996c67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e949b0-6rch5"
Oct 30 21:00:28.438599 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:28.438568    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e949b0-6rch5"
Oct 30 21:00:28.817933 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:28.817903    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14e949b0-6rch5"]
Oct 30 21:00:28.818341 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:00:28.818318    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c5f608f_fd3f_4180_af2f_e9b162996c67.slice/crio-3c5540b61e7fb5affa7aa579ba2a7831fc57419c9767e34525c732bd459829c5 WatchSource:0}: Error finding container 3c5540b61e7fb5affa7aa579ba2a7831fc57419c9767e34525c732bd459829c5: Status 404 returned error can't find the container with id 3c5540b61e7fb5affa7aa579ba2a7831fc57419c9767e34525c732bd459829c5
Oct 30 21:00:29.648429 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:29.648391    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e949b0-6rch5" event={"ID":"4c5f608f-fd3f-4180-af2f-e9b162996c67","Type":"ContainerStarted","Data":"67d051a3eece47998d8af66bbbc9b3e01756838dd951cd314c2eea46932f6525"}
Oct 30 21:00:29.648955 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:29.648454    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e949b0-6rch5" event={"ID":"4c5f608f-fd3f-4180-af2f-e9b162996c67","Type":"ContainerStarted","Data":"3c5540b61e7fb5affa7aa579ba2a7831fc57419c9767e34525c732bd459829c5"}
Oct 30 21:00:30.658682 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:30.658594    5713 generic.go:356] "Generic (PLEG): container finished" podID="4c5f608f-fd3f-4180-af2f-e9b162996c67" containerID="67d051a3eece47998d8af66bbbc9b3e01756838dd951cd314c2eea46932f6525" exitCode=0
Oct 30 21:00:30.659141 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:30.658684    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e949b0-6rch5" event={"ID":"4c5f608f-fd3f-4180-af2f-e9b162996c67","Type":"ContainerDied","Data":"67d051a3eece47998d8af66bbbc9b3e01756838dd951cd314c2eea46932f6525"}
Oct 30 21:00:31.836445 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:31.836413    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e949b0-6rch5"
Oct 30 21:00:31.878712 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:31.878682    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c5f608f-fd3f-4180-af2f-e9b162996c67-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c5f608f-fd3f-4180-af2f-e9b162996c67-maas-api-xks-ca\") pod \"4c5f608f-fd3f-4180-af2f-e9b162996c67\" (UID: \"4c5f608f-fd3f-4180-af2f-e9b162996c67\") "
Oct 30 21:00:31.878712 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:31.878718    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c5f608f-fd3f-4180-af2f-e9b162996c67-kube-api-access-rk4j9\" (UniqueName: \"kubernetes.io/projected/4c5f608f-fd3f-4180-af2f-e9b162996c67-kube-api-access-rk4j9\") pod \"4c5f608f-fd3f-4180-af2f-e9b162996c67\" (UID: \"4c5f608f-fd3f-4180-af2f-e9b162996c67\") "
Oct 30 21:00:31.879039 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:31.878980    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c5f608f-fd3f-4180-af2f-e9b162996c67-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c5f608f-fd3f-4180-af2f-e9b162996c67-maas-api-openshift-ca\") pod \"4c5f608f-fd3f-4180-af2f-e9b162996c67\" (UID: \"4c5f608f-fd3f-4180-af2f-e9b162996c67\") "
Oct 30 21:00:31.879386 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:31.879361    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c5f608f-fd3f-4180-af2f-e9b162996c67-maas-api-openshift-ca" pod "4c5f608f-fd3f-4180-af2f-e9b162996c67" (UID: "4c5f608f-fd3f-4180-af2f-e9b162996c67"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:31.879607 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:31.879565    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c5f608f-fd3f-4180-af2f-e9b162996c67-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:31.881275 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:31.881247    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c5f608f-fd3f-4180-af2f-e9b162996c67-kube-api-access-rk4j9" pod "4c5f608f-fd3f-4180-af2f-e9b162996c67" (UID: "4c5f608f-fd3f-4180-af2f-e9b162996c67"). InnerVolumeSpecName "kube-api-access-rk4j9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:31.881404 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:31.881385    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c5f608f-fd3f-4180-af2f-e9b162996c67-maas-api-xks-ca" pod "4c5f608f-fd3f-4180-af2f-e9b162996c67" (UID: "4c5f608f-fd3f-4180-af2f-e9b162996c67"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:00:31.981030 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:31.980983    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c5f608f-fd3f-4180-af2f-e9b162996c67-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:31.981030 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:31.981029    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rk4j9\" (UniqueName: \"kubernetes.io/projected/4c5f608f-fd3f-4180-af2f-e9b162996c67-kube-api-access-rk4j9\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:32.051772 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:32.051735    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49ff7895-7z6ww"]
Oct 30 21:00:32.059487 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:32.059460    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49ff7895-7z6ww"]
Oct 30 21:00:32.495042 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:32.495011    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq"]
Oct 30 21:00:32.495728 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:32.495713    5713 cpu_manager.go:414] "containerMap: removing container" podUID="c0df799c-ee0f-45c0-bbe2-c7928deb0e3b" containerName="cleanup"
Oct 30 21:00:32.495780 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:32.495729    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0df799c-ee0f-45c0-bbe2-c7928deb0e3b" containerName="cleanup"
Oct 30 21:00:32.495780 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:32.495752    5713 cpu_manager.go:414] "containerMap: removing container" podUID="4c5f608f-fd3f-4180-af2f-e9b162996c67" containerName="revoke-keys"
Oct 30 21:00:32.495780 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:32.495758    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c5f608f-fd3f-4180-af2f-e9b162996c67" containerName="revoke-keys"
Oct 30 21:00:32.495889 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:32.495877    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0df799c-ee0f-45c0-bbe2-c7928deb0e3b" containerName="cleanup"
Oct 30 21:00:32.495889 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:32.495886    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c5f608f-fd3f-4180-af2f-e9b162996c67" containerName="revoke-keys"
Oct 30 21:00:32.499830 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:32.499811    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq"
Oct 30 21:00:32.590809 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:32.590749    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsr8f\" (UniqueName: \"kubernetes.io/projected/8b28543a-e85d-4593-93c9-44f0d5b1016a-kube-api-access-xsr8f\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq\" (UID: \"8b28543a-e85d-4593-93c9-44f0d5b1016a\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq"
Oct 30 21:00:32.591001 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:32.590827    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b28543a-e85d-4593-93c9-44f0d5b1016a-istio-envoy\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq\" (UID: \"8b28543a-e85d-4593-93c9-44f0d5b1016a\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq"
Oct 30 21:00:32.591001 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:32.590900    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b28543a-e85d-4593-93c9-44f0d5b1016a-istio-data\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq\" (UID: \"8b28543a-e85d-4593-93c9-44f0d5b1016a\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq"
Oct 30 21:00:32.591001 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:32.590947    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b28543a-e85d-4593-93c9-44f0d5b1016a-credential-socket\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq\" (UID: \"8b28543a-e85d-4593-93c9-44f0d5b1016a\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq"
Oct 30 21:00:32.591254 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:32.591022    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b28543a-e85d-4593-93c9-44f0d5b1016a-istio-token\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq\" (UID: \"8b28543a-e85d-4593-93c9-44f0d5b1016a\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq"
Oct 30 21:00:32.591254 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:32.591060    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b28543a-e85d-4593-93c9-44f0d5b1016a-workload-certs\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq\" (UID: \"8b28543a-e85d-4593-93c9-44f0d5b1016a\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq"
Oct 30 21:00:32.591254 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:32.591176    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b28543a-e85d-4593-93c9-44f0d5b1016a-istiod-ca-cert\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq\" (UID: \"8b28543a-e85d-4593-93c9-44f0d5b1016a\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq"
Oct 30 21:00:32.591254 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:32.591235    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b28543a-e85d-4593-93c9-44f0d5b1016a-workload-socket\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq\" (UID: \"8b28543a-e85d-4593-93c9-44f0d5b1016a\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq"
Oct 30 21:00:32.591564 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:32.591358    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b28543a-e85d-4593-93c9-44f0d5b1016a-istio-podinfo\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq\" (UID: \"8b28543a-e85d-4593-93c9-44f0d5b1016a\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq"
Oct 30 21:00:32.649839 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:32.649803    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-378d8807-h944m"]
Oct 30 21:00:32.654103 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:32.654083    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-378d8807-h944m"
Oct 30 21:00:32.677066 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:32.677035    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e949b0-6rch5"
Oct 30 21:00:32.677220 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:32.677073    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14e949b0-6rch5" event={"ID":"4c5f608f-fd3f-4180-af2f-e9b162996c67","Type":"ContainerDied","Data":"3c5540b61e7fb5affa7aa579ba2a7831fc57419c9767e34525c732bd459829c5"}
Oct 30 21:00:32.677220 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:32.677131    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c5540b61e7fb5affa7aa579ba2a7831fc57419c9767e34525c732bd459829c5"
Oct 30 21:00:32.692938 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:32.692911    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b28543a-e85d-4593-93c9-44f0d5b1016a-istiod-ca-cert\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq\" (UID: \"8b28543a-e85d-4593-93c9-44f0d5b1016a\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq"
Oct 30 21:00:32.693098 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:32.692986    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b28543a-e85d-4593-93c9-44f0d5b1016a-workload-socket\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq\" (UID: \"8b28543a-e85d-4593-93c9-44f0d5b1016a\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq"
Oct 30 21:00:32.693165 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:32.693104    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b28543a-e85d-4593-93c9-44f0d5b1016a-istio-podinfo\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq\" (UID: \"8b28543a-e85d-4593-93c9-44f0d5b1016a\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq"
Oct 30 21:00:32.693165 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:32.693147    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsr8f\" (UniqueName: \"kubernetes.io/projected/8b28543a-e85d-4593-93c9-44f0d5b1016a-kube-api-access-xsr8f\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq\" (UID: \"8b28543a-e85d-4593-93c9-44f0d5b1016a\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq"
Oct 30 21:00:32.693276 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:32.693177    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b28543a-e85d-4593-93c9-44f0d5b1016a-istio-envoy\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq\" (UID: \"8b28543a-e85d-4593-93c9-44f0d5b1016a\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq"
Oct 30 21:00:32.693276 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:32.693237    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b28543a-e85d-4593-93c9-44f0d5b1016a-istio-data\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq\" (UID: \"8b28543a-e85d-4593-93c9-44f0d5b1016a\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq"
Oct 30 21:00:32.693923 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:32.693409    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b28543a-e85d-4593-93c9-44f0d5b1016a-credential-socket\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq\" (UID: \"8b28543a-e85d-4593-93c9-44f0d5b1016a\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq"
Oct 30 21:00:32.693923 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:32.693500    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b28543a-e85d-4593-93c9-44f0d5b1016a-workload-socket\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq\" (UID: \"8b28543a-e85d-4593-93c9-44f0d5b1016a\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq"
Oct 30 21:00:32.693923 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:32.693508    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b28543a-e85d-4593-93c9-44f0d5b1016a-istio-token\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq\" (UID: \"8b28543a-e85d-4593-93c9-44f0d5b1016a\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq"
Oct 30 21:00:32.693923 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:32.693548    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b28543a-e85d-4593-93c9-44f0d5b1016a-istio-data\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq\" (UID: \"8b28543a-e85d-4593-93c9-44f0d5b1016a\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq"
Oct 30 21:00:32.693923 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:32.693594    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b28543a-e85d-4593-93c9-44f0d5b1016a-workload-certs\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq\" (UID: \"8b28543a-e85d-4593-93c9-44f0d5b1016a\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq"
Oct 30 21:00:32.693923 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:32.693765    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b28543a-e85d-4593-93c9-44f0d5b1016a-istiod-ca-cert\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq\" (UID: \"8b28543a-e85d-4593-93c9-44f0d5b1016a\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq"
Oct 30 21:00:32.694888 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:32.693993    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b28543a-e85d-4593-93c9-44f0d5b1016a-workload-certs\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq\" (UID: \"8b28543a-e85d-4593-93c9-44f0d5b1016a\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq"
Oct 30 21:00:32.694888 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:32.694291    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b28543a-e85d-4593-93c9-44f0d5b1016a-credential-socket\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq\" (UID: \"8b28543a-e85d-4593-93c9-44f0d5b1016a\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq"
Oct 30 21:00:32.696119 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:32.696090    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b28543a-e85d-4593-93c9-44f0d5b1016a-istio-envoy\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq\" (UID: \"8b28543a-e85d-4593-93c9-44f0d5b1016a\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq"
Oct 30 21:00:32.696221 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:32.696164    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b28543a-e85d-4593-93c9-44f0d5b1016a-istio-podinfo\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq\" (UID: \"8b28543a-e85d-4593-93c9-44f0d5b1016a\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq"
Oct 30 21:00:32.702183 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:32.702153    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq"]
Oct 30 21:00:32.723542 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:32.723510    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-378d8807-h944m"]
Oct 30 21:00:32.750379 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:32.750297    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b28543a-e85d-4593-93c9-44f0d5b1016a-istio-token\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq\" (UID: \"8b28543a-e85d-4593-93c9-44f0d5b1016a\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq"
Oct 30 21:00:32.751405 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:32.751374    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsr8f\" (UniqueName: \"kubernetes.io/projected/8b28543a-e85d-4593-93c9-44f0d5b1016a-kube-api-access-xsr8f\") pod \"e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq\" (UID: \"8b28543a-e85d-4593-93c9-44f0d5b1016a\") " pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq"
Oct 30 21:00:32.795381 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:32.795340    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1d682ca-81f6-4509-8440-243c337a4117-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-378d8807-h944m\" (UID: \"a1d682ca-81f6-4509-8440-243c337a4117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-378d8807-h944m"
Oct 30 21:00:32.795565 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:32.795499    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1d682ca-81f6-4509-8440-243c337a4117-maas-api-xks-ca\") pod \"maas-api-revoke-sub-378d8807-h944m\" (UID: \"a1d682ca-81f6-4509-8440-243c337a4117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-378d8807-h944m"
Oct 30 21:00:32.795609 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:32.795586    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq9hg\" (UniqueName: \"kubernetes.io/projected/a1d682ca-81f6-4509-8440-243c337a4117-kube-api-access-mq9hg\") pod \"maas-api-revoke-sub-378d8807-h944m\" (UID: \"a1d682ca-81f6-4509-8440-243c337a4117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-378d8807-h944m"
Oct 30 21:00:32.813996 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:32.813957    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq"
Oct 30 21:00:32.897105 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:32.896666    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq9hg\" (UniqueName: \"kubernetes.io/projected/a1d682ca-81f6-4509-8440-243c337a4117-kube-api-access-mq9hg\") pod \"maas-api-revoke-sub-378d8807-h944m\" (UID: \"a1d682ca-81f6-4509-8440-243c337a4117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-378d8807-h944m"
Oct 30 21:00:32.897105 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:32.896833    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1d682ca-81f6-4509-8440-243c337a4117-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-378d8807-h944m\" (UID: \"a1d682ca-81f6-4509-8440-243c337a4117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-378d8807-h944m"
Oct 30 21:00:32.897105 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:32.896901    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1d682ca-81f6-4509-8440-243c337a4117-maas-api-xks-ca\") pod \"maas-api-revoke-sub-378d8807-h944m\" (UID: \"a1d682ca-81f6-4509-8440-243c337a4117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-378d8807-h944m"
Oct 30 21:00:32.899694 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:32.898520    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1d682ca-81f6-4509-8440-243c337a4117-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-378d8807-h944m\" (UID: \"a1d682ca-81f6-4509-8440-243c337a4117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-378d8807-h944m"
Oct 30 21:00:32.900073 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:32.900028    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1d682ca-81f6-4509-8440-243c337a4117-maas-api-xks-ca\") pod \"maas-api-revoke-sub-378d8807-h944m\" (UID: \"a1d682ca-81f6-4509-8440-243c337a4117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-378d8807-h944m"
Oct 30 21:00:32.955674 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:32.955634    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq9hg\" (UniqueName: \"kubernetes.io/projected/a1d682ca-81f6-4509-8440-243c337a4117-kube-api-access-mq9hg\") pod \"maas-api-revoke-sub-378d8807-h944m\" (UID: \"a1d682ca-81f6-4509-8440-243c337a4117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-378d8807-h944m"
Oct 30 21:00:32.969644 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:32.969614    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-378d8807-h944m"
Oct 30 21:00:33.098003 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:33.097973    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c81ff9be-12b4-41f3-8ea2-f86a043f5e2a" path="/var/lib/kubelet/pods/c81ff9be-12b4-41f3-8ea2-f86a043f5e2a/volumes"
Oct 30 21:00:33.104413 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:33.098423    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c80d8c0a-97b7d"]
Oct 30 21:00:33.116994 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:33.112434    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c80d8c0a-97b7d"]
Oct 30 21:00:33.116994 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:33.113903    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c80d8c0a-97b7d"
Oct 30 21:00:33.217405 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:33.217365    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e92b3948-3eca-4062-ae3a-f2163d552b54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c80d8c0a-97b7d\" (UID: \"e92b3948-3eca-4062-ae3a-f2163d552b54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c80d8c0a-97b7d"
Oct 30 21:00:33.217565 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:33.217418    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn74j\" (UniqueName: \"kubernetes.io/projected/e92b3948-3eca-4062-ae3a-f2163d552b54-kube-api-access-xn74j\") pod \"maas-api-revoke-sub-c80d8c0a-97b7d\" (UID: \"e92b3948-3eca-4062-ae3a-f2163d552b54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c80d8c0a-97b7d"
Oct 30 21:00:33.217654 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:33.217634    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e92b3948-3eca-4062-ae3a-f2163d552b54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c80d8c0a-97b7d\" (UID: \"e92b3948-3eca-4062-ae3a-f2163d552b54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c80d8c0a-97b7d"
Oct 30 21:00:33.255121 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:00:33.255090    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b28543a_e85d_4593_93c9_44f0d5b1016a.slice/crio-a5e468285eca9b68954aca6322fd741ea78a70c3a2642b6658d2093bd060c90a WatchSource:0}: Error finding container a5e468285eca9b68954aca6322fd741ea78a70c3a2642b6658d2093bd060c90a: Status 404 returned error can't find the container with id a5e468285eca9b68954aca6322fd741ea78a70c3a2642b6658d2093bd060c90a
Oct 30 21:00:33.255281 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:33.255257    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq"]
Oct 30 21:00:33.257887 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:33.257857    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:00:33.257958 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:33.257926    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:00:33.257994 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:33.257958    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:00:33.319202 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:33.319173    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e92b3948-3eca-4062-ae3a-f2163d552b54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c80d8c0a-97b7d\" (UID: \"e92b3948-3eca-4062-ae3a-f2163d552b54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c80d8c0a-97b7d"
Oct 30 21:00:33.319384 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:33.319364    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e92b3948-3eca-4062-ae3a-f2163d552b54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c80d8c0a-97b7d\" (UID: \"e92b3948-3eca-4062-ae3a-f2163d552b54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c80d8c0a-97b7d"
Oct 30 21:00:33.319458 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:33.319404    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn74j\" (UniqueName: \"kubernetes.io/projected/e92b3948-3eca-4062-ae3a-f2163d552b54-kube-api-access-xn74j\") pod \"maas-api-revoke-sub-c80d8c0a-97b7d\" (UID: \"e92b3948-3eca-4062-ae3a-f2163d552b54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c80d8c0a-97b7d"
Oct 30 21:00:33.320048 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:33.320016    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e92b3948-3eca-4062-ae3a-f2163d552b54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c80d8c0a-97b7d\" (UID: \"e92b3948-3eca-4062-ae3a-f2163d552b54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c80d8c0a-97b7d"
Oct 30 21:00:33.322221 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:33.322197    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e92b3948-3eca-4062-ae3a-f2163d552b54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c80d8c0a-97b7d\" (UID: \"e92b3948-3eca-4062-ae3a-f2163d552b54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c80d8c0a-97b7d"
Oct 30 21:00:33.359146 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:00:33.359100    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1d682ca_81f6_4509_8440_243c337a4117.slice/crio-7a7c3b0449f61d38086e83dc71e19e0c3cffa599dd62a511e2cab3ee1c51cff3 WatchSource:0}: Error finding container 7a7c3b0449f61d38086e83dc71e19e0c3cffa599dd62a511e2cab3ee1c51cff3: Status 404 returned error can't find the container with id 7a7c3b0449f61d38086e83dc71e19e0c3cffa599dd62a511e2cab3ee1c51cff3
Oct 30 21:00:33.359273 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:33.359240    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-378d8807-h944m"]
Oct 30 21:00:33.376823 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:33.376772    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn74j\" (UniqueName: \"kubernetes.io/projected/e92b3948-3eca-4062-ae3a-f2163d552b54-kube-api-access-xn74j\") pod \"maas-api-revoke-sub-c80d8c0a-97b7d\" (UID: \"e92b3948-3eca-4062-ae3a-f2163d552b54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c80d8c0a-97b7d"
Oct 30 21:00:33.433321 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:33.433292    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c80d8c0a-97b7d"
Oct 30 21:00:33.688573 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:33.688532    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq" event={"ID":"8b28543a-e85d-4593-93c9-44f0d5b1016a","Type":"ContainerStarted","Data":"51fd22ecb1c16e833fc4935f631f0d9a9b91dff21878c140abbae1cb3dab8f64"}
Oct 30 21:00:33.688748 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:33.688602    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq" event={"ID":"8b28543a-e85d-4593-93c9-44f0d5b1016a","Type":"ContainerStarted","Data":"a5e468285eca9b68954aca6322fd741ea78a70c3a2642b6658d2093bd060c90a"}
Oct 30 21:00:33.690166 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:33.690122    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-378d8807-h944m" event={"ID":"a1d682ca-81f6-4509-8440-243c337a4117","Type":"ContainerStarted","Data":"7a7c3b0449f61d38086e83dc71e19e0c3cffa599dd62a511e2cab3ee1c51cff3"}
Oct 30 21:00:33.809149 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:33.809092    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq" podStartSLOduration=1.809077803 podStartE2EDuration="1.809077803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:33.804709578 +0000 UTC m=+1316.869485977" watchObservedRunningTime="2026-10-30 21:00:33.809077803 +0000 UTC m=+1316.873854201"
Oct 30 21:00:33.814813 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:33.814762    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq"
Oct 30 21:00:33.821609 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:00:33.821569    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode92b3948_3eca_4062_ae3a_f2163d552b54.slice/crio-d8e55c2fbdaf1ea38cc1852f2dda51449336c96c335e03a1e3373f95a38750fd WatchSource:0}: Error finding container d8e55c2fbdaf1ea38cc1852f2dda51449336c96c335e03a1e3373f95a38750fd: Status 404 returned error can't find the container with id d8e55c2fbdaf1ea38cc1852f2dda51449336c96c335e03a1e3373f95a38750fd
Oct 30 21:00:33.859638 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:33.859610    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c80d8c0a-97b7d"]
Oct 30 21:00:34.699980 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:34.699946    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c80d8c0a-97b7d" event={"ID":"e92b3948-3eca-4062-ae3a-f2163d552b54","Type":"ContainerStarted","Data":"41a24ca00d9e8ef9950f6133112d43da5d7127d33712d95444167d4e0c4e17de"}
Oct 30 21:00:34.700453 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:34.700001    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c80d8c0a-97b7d" event={"ID":"e92b3948-3eca-4062-ae3a-f2163d552b54","Type":"ContainerStarted","Data":"d8e55c2fbdaf1ea38cc1852f2dda51449336c96c335e03a1e3373f95a38750fd"}
Oct 30 21:00:34.701670 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:34.701644    5713 generic.go:356] "Generic (PLEG): container finished" podID="a1d682ca-81f6-4509-8440-243c337a4117" containerID="9fa5eda54fb8661dfa6eb5d769e21917a3f66d79d5560116e25258d74e6deef7" exitCode=0
Oct 30 21:00:34.701802 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:34.701717    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-378d8807-h944m" event={"ID":"a1d682ca-81f6-4509-8440-243c337a4117","Type":"ContainerDied","Data":"9fa5eda54fb8661dfa6eb5d769e21917a3f66d79d5560116e25258d74e6deef7"}
Oct 30 21:00:34.815721 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:34.815632    5713 patch_prober.go:29] interesting pod/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Oct 30 21:00:34.815721 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:34.815691    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq" podUID="8b28543a-e85d-4593-93c9-44f0d5b1016a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Oct 30 21:00:34.826703 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:34.826650    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c80d8c0a-97b7d" podStartSLOduration=1.8266356400000001 podStartE2EDuration="1.82663564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:34.822503649 +0000 UTC m=+1317.887280046" watchObservedRunningTime="2026-10-30 21:00:34.82663564 +0000 UTC m=+1317.891412039"
Oct 30 21:00:35.717632 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:35.717593    5713 generic.go:356] "Generic (PLEG): container finished" podID="e92b3948-3eca-4062-ae3a-f2163d552b54" containerID="41a24ca00d9e8ef9950f6133112d43da5d7127d33712d95444167d4e0c4e17de" exitCode=0
Oct 30 21:00:35.718126 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:35.717636    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c80d8c0a-97b7d" event={"ID":"e92b3948-3eca-4062-ae3a-f2163d552b54","Type":"ContainerDied","Data":"41a24ca00d9e8ef9950f6133112d43da5d7127d33712d95444167d4e0c4e17de"}
Oct 30 21:00:35.816823 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:35.814934    5713 patch_prober.go:29] interesting pod/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Oct 30 21:00:35.816823 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:35.814998    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq" podUID="8b28543a-e85d-4593-93c9-44f0d5b1016a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Oct 30 21:00:35.929667 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:35.929642    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-378d8807-h944m"
Oct 30 21:00:36.084166 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:36.084071    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1d682ca-81f6-4509-8440-243c337a4117-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1d682ca-81f6-4509-8440-243c337a4117-maas-api-openshift-ca\") pod \"a1d682ca-81f6-4509-8440-243c337a4117\" (UID: \"a1d682ca-81f6-4509-8440-243c337a4117\") "
Oct 30 21:00:36.084166 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:36.084146    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1d682ca-81f6-4509-8440-243c337a4117-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1d682ca-81f6-4509-8440-243c337a4117-maas-api-xks-ca\") pod \"a1d682ca-81f6-4509-8440-243c337a4117\" (UID: \"a1d682ca-81f6-4509-8440-243c337a4117\") "
Oct 30 21:00:36.084447 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:36.084414    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1d682ca-81f6-4509-8440-243c337a4117-kube-api-access-mq9hg\" (UniqueName: \"kubernetes.io/projected/a1d682ca-81f6-4509-8440-243c337a4117-kube-api-access-mq9hg\") pod \"a1d682ca-81f6-4509-8440-243c337a4117\" (UID: \"a1d682ca-81f6-4509-8440-243c337a4117\") "
Oct 30 21:00:36.084572 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:36.084530    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1d682ca-81f6-4509-8440-243c337a4117-maas-api-openshift-ca" pod "a1d682ca-81f6-4509-8440-243c337a4117" (UID: "a1d682ca-81f6-4509-8440-243c337a4117"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:36.084937 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:36.084917    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1d682ca-81f6-4509-8440-243c337a4117-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:36.087010 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:36.086984    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1d682ca-81f6-4509-8440-243c337a4117-kube-api-access-mq9hg" pod "a1d682ca-81f6-4509-8440-243c337a4117" (UID: "a1d682ca-81f6-4509-8440-243c337a4117"). InnerVolumeSpecName "kube-api-access-mq9hg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:36.091179 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:36.091145    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1d682ca-81f6-4509-8440-243c337a4117-maas-api-xks-ca" pod "a1d682ca-81f6-4509-8440-243c337a4117" (UID: "a1d682ca-81f6-4509-8440-243c337a4117"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:00:36.186851 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:36.186780    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq9hg\" (UniqueName: \"kubernetes.io/projected/a1d682ca-81f6-4509-8440-243c337a4117-kube-api-access-mq9hg\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:36.186851 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:36.186844    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1d682ca-81f6-4509-8440-243c337a4117-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:36.729333 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:36.729306    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-378d8807-h944m"
Oct 30 21:00:36.729777 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:36.729333    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-378d8807-h944m" event={"ID":"a1d682ca-81f6-4509-8440-243c337a4117","Type":"ContainerDied","Data":"7a7c3b0449f61d38086e83dc71e19e0c3cffa599dd62a511e2cab3ee1c51cff3"}
Oct 30 21:00:36.729777 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:36.729383    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a7c3b0449f61d38086e83dc71e19e0c3cffa599dd62a511e2cab3ee1c51cff3"
Oct 30 21:00:36.815158 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:36.815131    5713 patch_prober.go:29] interesting pod/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Oct 30 21:00:36.815273 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:36.815181    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq" podUID="8b28543a-e85d-4593-93c9-44f0d5b1016a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Oct 30 21:00:36.969851 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:36.969824    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c80d8c0a-97b7d"
Oct 30 21:00:37.098330 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:37.098249    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e92b3948-3eca-4062-ae3a-f2163d552b54-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e92b3948-3eca-4062-ae3a-f2163d552b54-maas-api-openshift-ca\") pod \"e92b3948-3eca-4062-ae3a-f2163d552b54\" (UID: \"e92b3948-3eca-4062-ae3a-f2163d552b54\") "
Oct 30 21:00:37.098475 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:37.098364    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e92b3948-3eca-4062-ae3a-f2163d552b54-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e92b3948-3eca-4062-ae3a-f2163d552b54-maas-api-xks-ca\") pod \"e92b3948-3eca-4062-ae3a-f2163d552b54\" (UID: \"e92b3948-3eca-4062-ae3a-f2163d552b54\") "
Oct 30 21:00:37.098566 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:37.098534    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e92b3948-3eca-4062-ae3a-f2163d552b54-kube-api-access-xn74j\" (UniqueName: \"kubernetes.io/projected/e92b3948-3eca-4062-ae3a-f2163d552b54-kube-api-access-xn74j\") pod \"e92b3948-3eca-4062-ae3a-f2163d552b54\" (UID: \"e92b3948-3eca-4062-ae3a-f2163d552b54\") "
Oct 30 21:00:37.098566 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:37.098543    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e92b3948-3eca-4062-ae3a-f2163d552b54-maas-api-openshift-ca" pod "e92b3948-3eca-4062-ae3a-f2163d552b54" (UID: "e92b3948-3eca-4062-ae3a-f2163d552b54"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:37.099113 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:37.099088    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e92b3948-3eca-4062-ae3a-f2163d552b54-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:37.100576 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:37.100551    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e92b3948-3eca-4062-ae3a-f2163d552b54-maas-api-xks-ca" pod "e92b3948-3eca-4062-ae3a-f2163d552b54" (UID: "e92b3948-3eca-4062-ae3a-f2163d552b54"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:00:37.100677 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:37.100662    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e92b3948-3eca-4062-ae3a-f2163d552b54-kube-api-access-xn74j" pod "e92b3948-3eca-4062-ae3a-f2163d552b54" (UID: "e92b3948-3eca-4062-ae3a-f2163d552b54"). InnerVolumeSpecName "kube-api-access-xn74j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:37.200380 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:37.200348    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xn74j\" (UniqueName: \"kubernetes.io/projected/e92b3948-3eca-4062-ae3a-f2163d552b54-kube-api-access-xn74j\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:37.200380 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:37.200377    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e92b3948-3eca-4062-ae3a-f2163d552b54-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:37.281517 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:37.281488    5713 scope.go:122] "RemoveContainer" containerID="987fc6c83b8aea11ef470282c8678aad0b35c2dbc15507e408b1f56779ae3c1d"
Oct 30 21:00:37.292621 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:37.292600    5713 scope.go:122] "RemoveContainer" containerID="901ecb0ffb090ba3698ca6dd19bc5a7e9eff3c81db9284f372af08872771ff86"
Oct 30 21:00:37.302699 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:37.302673    5713 scope.go:122] "RemoveContainer" containerID="1327fe84166cc430aac285f9c6d38e2d759c69de43450f717c2902db94d6149b"
Oct 30 21:00:37.312679 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:37.312661    5713 scope.go:122] "RemoveContainer" containerID="10c0d056d691a6ba1653a30b9eadd54b64e72d77b050b7ff1e4d8751b285225a"
Oct 30 21:00:37.322182 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:37.322161    5713 scope.go:122] "RemoveContainer" containerID="ba0a8578e50de52468a77ebfe40f9596feda3ada68001bdd9f377bc98411336e"
Oct 30 21:00:37.765172 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:37.765144    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c80d8c0a-97b7d"
Oct 30 21:00:37.765575 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:37.765187    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c80d8c0a-97b7d" event={"ID":"e92b3948-3eca-4062-ae3a-f2163d552b54","Type":"ContainerDied","Data":"d8e55c2fbdaf1ea38cc1852f2dda51449336c96c335e03a1e3373f95a38750fd"}
Oct 30 21:00:37.765575 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:37.765240    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8e55c2fbdaf1ea38cc1852f2dda51449336c96c335e03a1e3373f95a38750fd"
Oct 30 21:00:37.814366 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:37.814326    5713 patch_prober.go:29] interesting pod/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Oct 30 21:00:37.814541 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:37.814384    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq" podUID="8b28543a-e85d-4593-93c9-44f0d5b1016a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Oct 30 21:00:37.833064 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:00:37.833030    5713 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode92b3948_3eca_4062_ae3a_f2163d552b54.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode92b3948_3eca_4062_ae3a_f2163d552b54.slice/crio-d8e55c2fbdaf1ea38cc1852f2dda51449336c96c335e03a1e3373f95a38750fd\": RecentStats: unable to find data in memory cache]"
Oct 30 21:00:38.814643 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:38.814609    5713 patch_prober.go:29] interesting pod/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Oct 30 21:00:38.815064 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:38.814662    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq" podUID="8b28543a-e85d-4593-93c9-44f0d5b1016a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Oct 30 21:00:39.070085 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:39.070010    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4614c1a-q6hld"]
Oct 30 21:00:39.070564 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:39.070550    5713 cpu_manager.go:414] "containerMap: removing container" podUID="e92b3948-3eca-4062-ae3a-f2163d552b54" containerName="revoke-keys"
Oct 30 21:00:39.070611 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:39.070565    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e92b3948-3eca-4062-ae3a-f2163d552b54" containerName="revoke-keys"
Oct 30 21:00:39.070643 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:39.070612    5713 cpu_manager.go:414] "containerMap: removing container" podUID="a1d682ca-81f6-4509-8440-243c337a4117" containerName="revoke-keys"
Oct 30 21:00:39.070643 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:39.070618    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1d682ca-81f6-4509-8440-243c337a4117" containerName="revoke-keys"
Oct 30 21:00:39.070720 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:39.070711    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="e92b3948-3eca-4062-ae3a-f2163d552b54" containerName="revoke-keys"
Oct 30 21:00:39.070761 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:39.070729    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1d682ca-81f6-4509-8440-243c337a4117" containerName="revoke-keys"
Oct 30 21:00:39.076151 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:39.076131    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4614c1a-q6hld"
Oct 30 21:00:39.109337 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:39.109295    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4614c1a-q6hld"]
Oct 30 21:00:39.159158 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:39.159032    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:00:39.159158 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:39.159055    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:00:39.159158 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:39.159086    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:00:39.159158 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:39.159155    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:00:39.231014 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:39.230977    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfkj6\" (UniqueName: \"kubernetes.io/projected/7f2bc097-ac83-4106-91aa-a50abf5fc9c6-kube-api-access-vfkj6\") pod \"maas-api-revoke-sub-c4614c1a-q6hld\" (UID: \"7f2bc097-ac83-4106-91aa-a50abf5fc9c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4614c1a-q6hld"
Oct 30 21:00:39.231238 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:39.231194    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f2bc097-ac83-4106-91aa-a50abf5fc9c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4614c1a-q6hld\" (UID: \"7f2bc097-ac83-4106-91aa-a50abf5fc9c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4614c1a-q6hld"
Oct 30 21:00:39.231372 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:39.231349    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f2bc097-ac83-4106-91aa-a50abf5fc9c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4614c1a-q6hld\" (UID: \"7f2bc097-ac83-4106-91aa-a50abf5fc9c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4614c1a-q6hld"
Oct 30 21:00:39.332495 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:39.332410    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfkj6\" (UniqueName: \"kubernetes.io/projected/7f2bc097-ac83-4106-91aa-a50abf5fc9c6-kube-api-access-vfkj6\") pod \"maas-api-revoke-sub-c4614c1a-q6hld\" (UID: \"7f2bc097-ac83-4106-91aa-a50abf5fc9c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4614c1a-q6hld"
Oct 30 21:00:39.332631 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:39.332522    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f2bc097-ac83-4106-91aa-a50abf5fc9c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4614c1a-q6hld\" (UID: \"7f2bc097-ac83-4106-91aa-a50abf5fc9c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4614c1a-q6hld"
Oct 30 21:00:39.332631 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:39.332579    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f2bc097-ac83-4106-91aa-a50abf5fc9c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4614c1a-q6hld\" (UID: \"7f2bc097-ac83-4106-91aa-a50abf5fc9c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4614c1a-q6hld"
Oct 30 21:00:39.333158 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:39.333138    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f2bc097-ac83-4106-91aa-a50abf5fc9c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4614c1a-q6hld\" (UID: \"7f2bc097-ac83-4106-91aa-a50abf5fc9c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4614c1a-q6hld"
Oct 30 21:00:39.357812 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:39.349217    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f2bc097-ac83-4106-91aa-a50abf5fc9c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4614c1a-q6hld\" (UID: \"7f2bc097-ac83-4106-91aa-a50abf5fc9c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4614c1a-q6hld"
Oct 30 21:00:39.407990 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:39.407903    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfkj6\" (UniqueName: \"kubernetes.io/projected/7f2bc097-ac83-4106-91aa-a50abf5fc9c6-kube-api-access-vfkj6\") pod \"maas-api-revoke-sub-c4614c1a-q6hld\" (UID: \"7f2bc097-ac83-4106-91aa-a50abf5fc9c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4614c1a-q6hld"
Oct 30 21:00:39.690305 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:39.690268    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4614c1a-q6hld"
Oct 30 21:00:39.820999 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:39.814380    5713 patch_prober.go:29] interesting pod/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Oct 30 21:00:39.820999 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:39.814439    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq" podUID="8b28543a-e85d-4593-93c9-44f0d5b1016a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Oct 30 21:00:40.094192 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:40.094142    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4614c1a-q6hld"]
Oct 30 21:00:40.818814 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:40.818281    5713 patch_prober.go:29] interesting pod/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Oct 30 21:00:40.818814 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:40.818341    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq" podUID="8b28543a-e85d-4593-93c9-44f0d5b1016a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Oct 30 21:00:40.984868 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:40.984521    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4614c1a-q6hld" event={"ID":"7f2bc097-ac83-4106-91aa-a50abf5fc9c6","Type":"ContainerStarted","Data":"3b4855f2379d671680e3bfab4ea34141f77ef81df034b2e58678405d8923aa2d"}
Oct 30 21:00:40.984868 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:40.984591    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4614c1a-q6hld" event={"ID":"7f2bc097-ac83-4106-91aa-a50abf5fc9c6","Type":"ContainerStarted","Data":"f877f49c8467847137a41c122d5548e2ce9dc344728a1c3209196568f5c9f2d5"}
Oct 30 21:00:41.132816 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:41.126117    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4614c1a-q6hld" podStartSLOduration=3.126096158 podStartE2EDuration="3.126096158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:41.107387573 +0000 UTC m=+1324.172163976" watchObservedRunningTime="2026-10-30 21:00:41.126096158 +0000 UTC m=+1324.190872562"
Oct 30 21:00:41.811814 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:41.810899    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-682999/distinct-w3-682999-kserve-74d49d4955-nndlc" podUID="2a35314b-31ea-42e6-b49a-15ab1305b338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:41.854817 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:41.852910    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-781058/distinct-2-w0-781058-kserve-7f6c654757-ppmhm" podUID="5e818303-0e9f-4ba0-a635-b80b92c9e36d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:41.885813 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:41.885138    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f9f277/premium-w1-f9f277-kserve-5dbfd4d496-72w4n" podUID="1cc1f811-c149-4457-80f1-1a55621d73b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:41.928811 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:41.924952    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-682999/simulator-w3-682999-kserve-57cfcb6f8f-6mgrg" podUID="53296e7f-c6bd-45f2-ac31-004feddc63a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:41.928811 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:41.927513    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b425d/premium-w5-2b425d-kserve-78756b6c5b-9x5xb" podUID="abf7b512-50e5-4400-9ca3-9ffccaead3d7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:41.932814 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:41.929657    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7" podUID="b2193213-ebae-498a-911a-6056a8e94312" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:42.009944 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:42.009888    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-781058/simulator-w0-781058-kserve-55b954cb4b-4k7z6" podUID="92c43343-2820-4097-b829-9aae96b670de" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:42.010663 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:42.010632    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b425d/simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk" podUID="e8108824-03ff-4c1d-9f5b-07be3a3a92d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Oct 30 21:00:42.014814 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:42.011119    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm" podUID="cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Oct 30 21:00:42.027816 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:42.016331    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-682999/premium-w3-682999-kserve-5f47cf9b8d-xqnnr" podUID="a54cd33d-feef-42bb-a78b-7480634cb58d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 30 21:00:42.050814 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:42.028962    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l" podUID="fc9510c5-2180-41d6-90bc-9acdff0f3616" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:42.685866 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:42.668897    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-682999/distinct-2-w3-682999-kserve-7bc687885b-gq5x7" podUID="0dcf0422-081f-46d7-b246-7a49b074c88c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:42.734695 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:42.734646    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-781058/premium-w0-781058-kserve-7f5c8964c9-z5fmk" podUID="57626f4f-844e-4ec9-ab89-359d015923af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:42.746817 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:42.735294    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b425d/distinct-2-w5-2b425d-kserve-6694867c68-bzpvn" podUID="b9d391e6-de6f-476d-a926-d1b333474109" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded"
Oct 30 21:00:42.772264 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:42.772226    5713 generic.go:356] "Generic (PLEG): container finished" podID="7f2bc097-ac83-4106-91aa-a50abf5fc9c6" containerID="3b4855f2379d671680e3bfab4ea34141f77ef81df034b2e58678405d8923aa2d" exitCode=0
Oct 30 21:00:42.773419 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:42.773032    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4614c1a-q6hld" event={"ID":"7f2bc097-ac83-4106-91aa-a50abf5fc9c6","Type":"ContainerDied","Data":"3b4855f2379d671680e3bfab4ea34141f77ef81df034b2e58678405d8923aa2d"}
Oct 30 21:00:42.801725 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:42.801672    5713 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-682999/premium-w3-682999-kserve-5f47cf9b8d-xqnnr" podUID="a54cd33d-feef-42bb-a78b-7480634cb58d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:42.830818 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:42.829855    5713 patch_prober.go:29] interesting pod/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:00:42.830818 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:42.829934    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq" podUID="8b28543a-e85d-4593-93c9-44f0d5b1016a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:42.835963 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:42.835917    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-682999/distinct-w3-682999-kserve-74d49d4955-nndlc" podUID="2a35314b-31ea-42e6-b49a-15ab1305b338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:42.872246 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:42.862566    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-781058/distinct-2-w0-781058-kserve-7f6c654757-ppmhm" podUID="5e818303-0e9f-4ba0-a635-b80b92c9e36d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 30 21:00:42.872820 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:42.872778    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-682999/distinct-w3-682999-kserve-74d49d4955-nndlc"
Oct 30 21:00:42.873009 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:42.872996    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-781058/distinct-2-w0-781058-kserve-7f6c654757-ppmhm"
Oct 30 21:00:42.876087 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:42.876064    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq"
Oct 30 21:00:42.886952 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:42.886910    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f9f277/premium-w1-f9f277-kserve-5dbfd4d496-72w4n" podUID="1cc1f811-c149-4457-80f1-1a55621d73b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 30 21:00:42.887524 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:42.887498    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq"
Oct 30 21:00:42.887765 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:42.887746    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f9f277/premium-w1-f9f277-kserve-5dbfd4d496-72w4n"
Oct 30 21:00:42.900187 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:42.900151    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq"
Oct 30 21:00:42.929821 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:42.925970    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-682999/simulator-w3-682999-kserve-57cfcb6f8f-6mgrg" podUID="53296e7f-c6bd-45f2-ac31-004feddc63a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:42.929821 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:42.926384    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-682999/simulator-w3-682999-kserve-57cfcb6f8f-6mgrg"
Oct 30 21:00:42.930638 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:42.930601    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b425d/premium-w5-2b425d-kserve-78756b6c5b-9x5xb" podUID="abf7b512-50e5-4400-9ca3-9ffccaead3d7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:42.961875 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:42.938891    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7" podUID="b2193213-ebae-498a-911a-6056a8e94312" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded"
Oct 30 21:00:42.984426 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:42.972026    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2b425d/premium-w5-2b425d-kserve-78756b6c5b-9x5xb"
Oct 30 21:00:42.984717 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:42.984697    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7"
Oct 30 21:00:43.019818 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:43.012923    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b425d/simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk" podUID="e8108824-03ff-4c1d-9f5b-07be3a3a92d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:43.019818 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:43.013377    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-781058/simulator-w0-781058-kserve-55b954cb4b-4k7z6" podUID="92c43343-2820-4097-b829-9aae96b670de" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:43.066819 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:43.041927    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2b425d/simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk"
Oct 30 21:00:43.066819 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:43.042017    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-781058/simulator-w0-781058-kserve-55b954cb4b-4k7z6"
Oct 30 21:00:43.118684 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:43.097060    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm" podUID="cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Oct 30 21:00:43.121204 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:43.121148    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l" podUID="fc9510c5-2180-41d6-90bc-9acdff0f3616" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded"
Oct 30 21:00:43.142019 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:43.141972    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-682999/premium-w3-682999-kserve-5f47cf9b8d-xqnnr" podUID="a54cd33d-feef-42bb-a78b-7480634cb58d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 30 21:00:43.619096 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:43.619057    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm"
Oct 30 21:00:43.622813 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:43.619386    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l"
Oct 30 21:00:43.718815 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:43.631937    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-682999/premium-w3-682999-kserve-5f47cf9b8d-xqnnr"
Oct 30 21:00:43.737812 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:43.730512    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-682999/distinct-2-w3-682999-kserve-7bc687885b-gq5x7" podUID="0dcf0422-081f-46d7-b246-7a49b074c88c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:43.737812 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:43.730568    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-682999/premium-w3-682999-kserve-5f47cf9b8d-xqnnr"
Oct 30 21:00:43.738681 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:43.738653    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-682999/distinct-2-w3-682999-kserve-7bc687885b-gq5x7"
Oct 30 21:00:43.739806 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:43.739756    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-781058/premium-w0-781058-kserve-7f5c8964c9-z5fmk" podUID="57626f4f-844e-4ec9-ab89-359d015923af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:43.740279 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:43.740261    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-781058/premium-w0-781058-kserve-7f5c8964c9-z5fmk"
Oct 30 21:00:43.752293 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:43.752257    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b425d/distinct-2-w5-2b425d-kserve-6694867c68-bzpvn" podUID="b9d391e6-de6f-476d-a926-d1b333474109" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded"
Oct 30 21:00:43.763816 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:43.753930    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2b425d/distinct-2-w5-2b425d-kserve-6694867c68-bzpvn"
Oct 30 21:00:43.788615 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:43.788585    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-781058/premium-w0-781058-kserve-7f5c8964c9-z5fmk"
Oct 30 21:00:43.843210 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:43.843185    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-682999/distinct-2-w3-682999-kserve-7bc687885b-gq5x7"
Oct 30 21:00:43.864695 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:43.864663    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-781058/distinct-2-w0-781058-kserve-7f6c654757-ppmhm"
Oct 30 21:00:43.898070 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:43.888605    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f9f277/premium-w1-f9f277-kserve-5dbfd4d496-72w4n" podUID="1cc1f811-c149-4457-80f1-1a55621d73b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:43.900663 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:43.900621    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-682999/distinct-w3-682999-kserve-74d49d4955-nndlc" podUID="2a35314b-31ea-42e6-b49a-15ab1305b338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:43.928820 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:43.927306    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-682999/simulator-w3-682999-kserve-57cfcb6f8f-6mgrg" podUID="53296e7f-c6bd-45f2-ac31-004feddc63a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 30 21:00:43.958913 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:43.958858    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b425d/premium-w5-2b425d-kserve-78756b6c5b-9x5xb" podUID="abf7b512-50e5-4400-9ca3-9ffccaead3d7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:43.985826 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:43.984106    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7" podUID="b2193213-ebae-498a-911a-6056a8e94312" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:44.009539 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:44.000708    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2b425d/simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk"
Oct 30 21:00:44.013594 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:44.013560    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-781058/simulator-w0-781058-kserve-55b954cb4b-4k7z6" podUID="92c43343-2820-4097-b829-9aae96b670de" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:44.032840 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:44.024088    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-682999/distinct-w3-682999-kserve-74d49d4955-nndlc"
Oct 30 21:00:44.032840 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:44.026002    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm"
Oct 30 21:00:44.037169 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:44.033769    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2b425d/premium-w5-2b425d-kserve-78756b6c5b-9x5xb"
Oct 30 21:00:44.050401 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:44.049204    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-682999/simulator-w3-682999-kserve-57cfcb6f8f-6mgrg"
Oct 30 21:00:44.098827 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:44.073030    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7"
Oct 30 21:00:44.136051 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:44.128734    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-781058/simulator-w0-781058-kserve-55b954cb4b-4k7z6"
Oct 30 21:00:44.136051 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:44.130193    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l" podUID="fc9510c5-2180-41d6-90bc-9acdff0f3616" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:00:44.148770 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:44.148693    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2b425d/distinct-2-w5-2b425d-kserve-6694867c68-bzpvn"
Oct 30 21:00:44.167355 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:44.167328    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l"
Oct 30 21:00:44.168825 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:44.168804    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f9f277/premium-w1-f9f277-kserve-5dbfd4d496-72w4n"
Oct 30 21:00:44.315023 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:44.313951    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4614c1a-q6hld"
Oct 30 21:00:44.340357 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:44.340313    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf186c09-hcr7j"]
Oct 30 21:00:44.341379 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:44.341346    5713 cpu_manager.go:414] "containerMap: removing container" podUID="7f2bc097-ac83-4106-91aa-a50abf5fc9c6" containerName="revoke-keys"
Oct 30 21:00:44.341379 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:44.341374    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f2bc097-ac83-4106-91aa-a50abf5fc9c6" containerName="revoke-keys"
Oct 30 21:00:44.341596 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:44.341560    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f2bc097-ac83-4106-91aa-a50abf5fc9c6" containerName="revoke-keys"
Oct 30 21:00:44.346754 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:44.346720    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf186c09-hcr7j"
Oct 30 21:00:44.402883 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:44.402603    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf186c09-hcr7j"]
Oct 30 21:00:44.468919 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:44.468879    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f2bc097-ac83-4106-91aa-a50abf5fc9c6-kube-api-access-vfkj6\" (UniqueName: \"kubernetes.io/projected/7f2bc097-ac83-4106-91aa-a50abf5fc9c6-kube-api-access-vfkj6\") pod \"7f2bc097-ac83-4106-91aa-a50abf5fc9c6\" (UID: \"7f2bc097-ac83-4106-91aa-a50abf5fc9c6\") "
Oct 30 21:00:44.469131 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:44.468943    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f2bc097-ac83-4106-91aa-a50abf5fc9c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f2bc097-ac83-4106-91aa-a50abf5fc9c6-maas-api-xks-ca\") pod \"7f2bc097-ac83-4106-91aa-a50abf5fc9c6\" (UID: \"7f2bc097-ac83-4106-91aa-a50abf5fc9c6\") "
Oct 30 21:00:44.469131 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:44.469067    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f2bc097-ac83-4106-91aa-a50abf5fc9c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f2bc097-ac83-4106-91aa-a50abf5fc9c6-maas-api-openshift-ca\") pod \"7f2bc097-ac83-4106-91aa-a50abf5fc9c6\" (UID: \"7f2bc097-ac83-4106-91aa-a50abf5fc9c6\") "
Oct 30 21:00:44.469522 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:44.469488    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f2bc097-ac83-4106-91aa-a50abf5fc9c6-maas-api-openshift-ca" pod "7f2bc097-ac83-4106-91aa-a50abf5fc9c6" (UID: "7f2bc097-ac83-4106-91aa-a50abf5fc9c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:44.469697 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:44.469658    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96e75be8-c20d-4f62-9956-5ee10ea210ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf186c09-hcr7j\" (UID: \"96e75be8-c20d-4f62-9956-5ee10ea210ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf186c09-hcr7j"
Oct 30 21:00:44.470002 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:44.469975    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2sqn\" (UniqueName: \"kubernetes.io/projected/96e75be8-c20d-4f62-9956-5ee10ea210ac-kube-api-access-j2sqn\") pod \"maas-api-revoke-sub-bf186c09-hcr7j\" (UID: \"96e75be8-c20d-4f62-9956-5ee10ea210ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf186c09-hcr7j"
Oct 30 21:00:44.470223 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:44.470195    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96e75be8-c20d-4f62-9956-5ee10ea210ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf186c09-hcr7j\" (UID: \"96e75be8-c20d-4f62-9956-5ee10ea210ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf186c09-hcr7j"
Oct 30 21:00:44.470448 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:44.470429    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f2bc097-ac83-4106-91aa-a50abf5fc9c6-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:44.471946 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:44.471921    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f2bc097-ac83-4106-91aa-a50abf5fc9c6-kube-api-access-vfkj6" pod "7f2bc097-ac83-4106-91aa-a50abf5fc9c6" (UID: "7f2bc097-ac83-4106-91aa-a50abf5fc9c6"). InnerVolumeSpecName "kube-api-access-vfkj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:44.472054 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:44.471955    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f2bc097-ac83-4106-91aa-a50abf5fc9c6-maas-api-xks-ca" pod "7f2bc097-ac83-4106-91aa-a50abf5fc9c6" (UID: "7f2bc097-ac83-4106-91aa-a50abf5fc9c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:00:44.572355 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:44.572302    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96e75be8-c20d-4f62-9956-5ee10ea210ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf186c09-hcr7j\" (UID: \"96e75be8-c20d-4f62-9956-5ee10ea210ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf186c09-hcr7j"
Oct 30 21:00:44.572566 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:44.572419    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2sqn\" (UniqueName: \"kubernetes.io/projected/96e75be8-c20d-4f62-9956-5ee10ea210ac-kube-api-access-j2sqn\") pod \"maas-api-revoke-sub-bf186c09-hcr7j\" (UID: \"96e75be8-c20d-4f62-9956-5ee10ea210ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf186c09-hcr7j"
Oct 30 21:00:44.572647 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:44.572561    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96e75be8-c20d-4f62-9956-5ee10ea210ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf186c09-hcr7j\" (UID: \"96e75be8-c20d-4f62-9956-5ee10ea210ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf186c09-hcr7j"
Oct 30 21:00:44.572710 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:44.572691    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfkj6\" (UniqueName: \"kubernetes.io/projected/7f2bc097-ac83-4106-91aa-a50abf5fc9c6-kube-api-access-vfkj6\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:44.572781 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:44.572710    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f2bc097-ac83-4106-91aa-a50abf5fc9c6-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:44.573295 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:44.573270    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96e75be8-c20d-4f62-9956-5ee10ea210ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf186c09-hcr7j\" (UID: \"96e75be8-c20d-4f62-9956-5ee10ea210ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf186c09-hcr7j"
Oct 30 21:00:44.575427 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:44.575406    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96e75be8-c20d-4f62-9956-5ee10ea210ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf186c09-hcr7j\" (UID: \"96e75be8-c20d-4f62-9956-5ee10ea210ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf186c09-hcr7j"
Oct 30 21:00:44.633477 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:44.633448    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2sqn\" (UniqueName: \"kubernetes.io/projected/96e75be8-c20d-4f62-9956-5ee10ea210ac-kube-api-access-j2sqn\") pod \"maas-api-revoke-sub-bf186c09-hcr7j\" (UID: \"96e75be8-c20d-4f62-9956-5ee10ea210ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf186c09-hcr7j"
Oct 30 21:00:44.692329 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:44.692276    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf186c09-hcr7j"
Oct 30 21:00:44.839175 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:44.839147    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4614c1a-q6hld"
Oct 30 21:00:44.839175 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:44.839165    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4614c1a-q6hld" event={"ID":"7f2bc097-ac83-4106-91aa-a50abf5fc9c6","Type":"ContainerDied","Data":"f877f49c8467847137a41c122d5548e2ce9dc344728a1c3209196568f5c9f2d5"}
Oct 30 21:00:44.839380 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:44.839217    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f877f49c8467847137a41c122d5548e2ce9dc344728a1c3209196568f5c9f2d5"
Oct 30 21:00:45.085247 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:45.085220    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf186c09-hcr7j"]
Oct 30 21:00:45.086139 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:00:45.086109    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96e75be8_c20d_4f62_9956_5ee10ea210ac.slice/crio-4c3bde4af0f9c04fe558fc4314a2cde62bfe8b0bad6429be83be5a29a4967a20 WatchSource:0}: Error finding container 4c3bde4af0f9c04fe558fc4314a2cde62bfe8b0bad6429be83be5a29a4967a20: Status 404 returned error can't find the container with id 4c3bde4af0f9c04fe558fc4314a2cde62bfe8b0bad6429be83be5a29a4967a20
Oct 30 21:00:45.849911 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:45.849780    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf186c09-hcr7j" event={"ID":"96e75be8-c20d-4f62-9956-5ee10ea210ac","Type":"ContainerStarted","Data":"dfc7d722ce40f8f2dc54f4c0035bffb3427dad285e0af969ae5e38086eb47105"}
Oct 30 21:00:45.850078 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:45.849923    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf186c09-hcr7j" event={"ID":"96e75be8-c20d-4f62-9956-5ee10ea210ac","Type":"ContainerStarted","Data":"4c3bde4af0f9c04fe558fc4314a2cde62bfe8b0bad6429be83be5a29a4967a20"}
Oct 30 21:00:46.689925 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:46.689864    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf186c09-hcr7j" podStartSLOduration=2.689845288 podStartE2EDuration="2.689845288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:46.687463715 +0000 UTC m=+1329.752240115" watchObservedRunningTime="2026-10-30 21:00:46.689845288 +0000 UTC m=+1329.754621690"
Oct 30 21:00:46.859850 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:46.859777    5713 generic.go:356] "Generic (PLEG): container finished" podID="96e75be8-c20d-4f62-9956-5ee10ea210ac" containerID="dfc7d722ce40f8f2dc54f4c0035bffb3427dad285e0af969ae5e38086eb47105" exitCode=0
Oct 30 21:00:46.859850 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:46.859840    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf186c09-hcr7j" event={"ID":"96e75be8-c20d-4f62-9956-5ee10ea210ac","Type":"ContainerDied","Data":"dfc7d722ce40f8f2dc54f4c0035bffb3427dad285e0af969ae5e38086eb47105"}
Oct 30 21:00:47.684281 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:47.684241    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6d5ec58-5k6ft"]
Oct 30 21:00:47.691201 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:47.691175    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d5ec58-5k6ft"
Oct 30 21:00:47.741255 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:47.741223    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6d5ec58-5k6ft"]
Oct 30 21:00:47.824357 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:47.824326    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98feba90-5e8f-443e-8c3e-0cfb79875caa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6d5ec58-5k6ft\" (UID: \"98feba90-5e8f-443e-8c3e-0cfb79875caa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d5ec58-5k6ft"
Oct 30 21:00:47.824511 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:47.824383    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98feba90-5e8f-443e-8c3e-0cfb79875caa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6d5ec58-5k6ft\" (UID: \"98feba90-5e8f-443e-8c3e-0cfb79875caa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d5ec58-5k6ft"
Oct 30 21:00:47.824579 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:47.824527    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl9wj\" (UniqueName: \"kubernetes.io/projected/98feba90-5e8f-443e-8c3e-0cfb79875caa-kube-api-access-vl9wj\") pod \"maas-api-revoke-sub-c6d5ec58-5k6ft\" (UID: \"98feba90-5e8f-443e-8c3e-0cfb79875caa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d5ec58-5k6ft"
Oct 30 21:00:47.927542 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:47.927497    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98feba90-5e8f-443e-8c3e-0cfb79875caa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6d5ec58-5k6ft\" (UID: \"98feba90-5e8f-443e-8c3e-0cfb79875caa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d5ec58-5k6ft"
Oct 30 21:00:47.927671 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:47.927603    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98feba90-5e8f-443e-8c3e-0cfb79875caa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6d5ec58-5k6ft\" (UID: \"98feba90-5e8f-443e-8c3e-0cfb79875caa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d5ec58-5k6ft"
Oct 30 21:00:47.927741 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:47.927678    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98feba90-5e8f-443e-8c3e-0cfb79875caa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6d5ec58-5k6ft\" (UID: \"98feba90-5e8f-443e-8c3e-0cfb79875caa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d5ec58-5k6ft"
Oct 30 21:00:47.927872 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:47.927853    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl9wj\" (UniqueName: \"kubernetes.io/projected/98feba90-5e8f-443e-8c3e-0cfb79875caa-kube-api-access-vl9wj\") pod \"maas-api-revoke-sub-c6d5ec58-5k6ft\" (UID: \"98feba90-5e8f-443e-8c3e-0cfb79875caa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d5ec58-5k6ft"
Oct 30 21:00:47.945214 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:47.944311    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98feba90-5e8f-443e-8c3e-0cfb79875caa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6d5ec58-5k6ft\" (UID: \"98feba90-5e8f-443e-8c3e-0cfb79875caa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d5ec58-5k6ft"
Oct 30 21:00:47.985309 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:47.985275    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl9wj\" (UniqueName: \"kubernetes.io/projected/98feba90-5e8f-443e-8c3e-0cfb79875caa-kube-api-access-vl9wj\") pod \"maas-api-revoke-sub-c6d5ec58-5k6ft\" (UID: \"98feba90-5e8f-443e-8c3e-0cfb79875caa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d5ec58-5k6ft"
Oct 30 21:00:48.005737 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:48.005705    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d5ec58-5k6ft"
Oct 30 21:00:48.063665 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:48.063639    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf186c09-hcr7j"
Oct 30 21:00:48.231888 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:48.231779    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96e75be8-c20d-4f62-9956-5ee10ea210ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96e75be8-c20d-4f62-9956-5ee10ea210ac-maas-api-xks-ca\") pod \"96e75be8-c20d-4f62-9956-5ee10ea210ac\" (UID: \"96e75be8-c20d-4f62-9956-5ee10ea210ac\") "
Oct 30 21:00:48.232043 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:48.231892    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96e75be8-c20d-4f62-9956-5ee10ea210ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96e75be8-c20d-4f62-9956-5ee10ea210ac-maas-api-openshift-ca\") pod \"96e75be8-c20d-4f62-9956-5ee10ea210ac\" (UID: \"96e75be8-c20d-4f62-9956-5ee10ea210ac\") "
Oct 30 21:00:48.232043 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:48.231946    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96e75be8-c20d-4f62-9956-5ee10ea210ac-kube-api-access-j2sqn\" (UniqueName: \"kubernetes.io/projected/96e75be8-c20d-4f62-9956-5ee10ea210ac-kube-api-access-j2sqn\") pod \"96e75be8-c20d-4f62-9956-5ee10ea210ac\" (UID: \"96e75be8-c20d-4f62-9956-5ee10ea210ac\") "
Oct 30 21:00:48.232292 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:48.232265    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96e75be8-c20d-4f62-9956-5ee10ea210ac-maas-api-openshift-ca" pod "96e75be8-c20d-4f62-9956-5ee10ea210ac" (UID: "96e75be8-c20d-4f62-9956-5ee10ea210ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:48.232683 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:48.232662    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96e75be8-c20d-4f62-9956-5ee10ea210ac-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:48.233972 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:48.233946    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96e75be8-c20d-4f62-9956-5ee10ea210ac-maas-api-xks-ca" pod "96e75be8-c20d-4f62-9956-5ee10ea210ac" (UID: "96e75be8-c20d-4f62-9956-5ee10ea210ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:00:48.234194 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:48.234181    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96e75be8-c20d-4f62-9956-5ee10ea210ac-kube-api-access-j2sqn" pod "96e75be8-c20d-4f62-9956-5ee10ea210ac" (UID: "96e75be8-c20d-4f62-9956-5ee10ea210ac"). InnerVolumeSpecName "kube-api-access-j2sqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:48.334487 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:48.334449    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96e75be8-c20d-4f62-9956-5ee10ea210ac-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:48.334487 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:48.334483    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2sqn\" (UniqueName: \"kubernetes.io/projected/96e75be8-c20d-4f62-9956-5ee10ea210ac-kube-api-access-j2sqn\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:48.401045 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:48.401013    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6d5ec58-5k6ft"]
Oct 30 21:00:48.401170 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:00:48.401039    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98feba90_5e8f_443e_8c3e_0cfb79875caa.slice/crio-2dfe5f0c6ab2c13629bd11482120a7045ce1d838da70c5bdbaac67b2b1715ffb WatchSource:0}: Error finding container 2dfe5f0c6ab2c13629bd11482120a7045ce1d838da70c5bdbaac67b2b1715ffb: Status 404 returned error can't find the container with id 2dfe5f0c6ab2c13629bd11482120a7045ce1d838da70c5bdbaac67b2b1715ffb
Oct 30 21:00:48.878444 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:48.878406    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d5ec58-5k6ft" event={"ID":"98feba90-5e8f-443e-8c3e-0cfb79875caa","Type":"ContainerStarted","Data":"6c4f2e2984a531950649833eb2594d0d1b93e76a9d791882f3f0d2e35e43c07f"}
Oct 30 21:00:48.878872 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:48.878469    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d5ec58-5k6ft" event={"ID":"98feba90-5e8f-443e-8c3e-0cfb79875caa","Type":"ContainerStarted","Data":"2dfe5f0c6ab2c13629bd11482120a7045ce1d838da70c5bdbaac67b2b1715ffb"}
Oct 30 21:00:48.879951 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:48.879923    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf186c09-hcr7j"
Oct 30 21:00:48.880058 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:48.879946    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf186c09-hcr7j" event={"ID":"96e75be8-c20d-4f62-9956-5ee10ea210ac","Type":"ContainerDied","Data":"4c3bde4af0f9c04fe558fc4314a2cde62bfe8b0bad6429be83be5a29a4967a20"}
Oct 30 21:00:48.880058 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:48.879991    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c3bde4af0f9c04fe558fc4314a2cde62bfe8b0bad6429be83be5a29a4967a20"
Oct 30 21:00:48.994181 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:48.994123    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d5ec58-5k6ft" podStartSLOduration=1.9941047589999998 podStartE2EDuration="1.994104759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:48.991878348 +0000 UTC m=+1332.056654747" watchObservedRunningTime="2026-10-30 21:00:48.994104759 +0000 UTC m=+1332.058881156"
Oct 30 21:00:49.890707 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:49.890670    5713 generic.go:356] "Generic (PLEG): container finished" podID="98feba90-5e8f-443e-8c3e-0cfb79875caa" containerID="6c4f2e2984a531950649833eb2594d0d1b93e76a9d791882f3f0d2e35e43c07f" exitCode=0
Oct 30 21:00:49.891180 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:49.890762    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d5ec58-5k6ft" event={"ID":"98feba90-5e8f-443e-8c3e-0cfb79875caa","Type":"ContainerDied","Data":"6c4f2e2984a531950649833eb2594d0d1b93e76a9d791882f3f0d2e35e43c07f"}
Oct 30 21:00:50.735607 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:50.735569    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb69c39a-6ctbl"]
Oct 30 21:00:50.736525 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:50.736507    5713 cpu_manager.go:414] "containerMap: removing container" podUID="96e75be8-c20d-4f62-9956-5ee10ea210ac" containerName="revoke-keys"
Oct 30 21:00:50.736574 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:50.736528    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96e75be8-c20d-4f62-9956-5ee10ea210ac" containerName="revoke-keys"
Oct 30 21:00:50.736711 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:50.736699    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="96e75be8-c20d-4f62-9956-5ee10ea210ac" containerName="revoke-keys"
Oct 30 21:00:50.740337 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:50.740321    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb69c39a-6ctbl"
Oct 30 21:00:50.796002 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:50.795967    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb69c39a-6ctbl"]
Oct 30 21:00:50.871889 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:50.871854    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ddf3248-69bc-458b-b12e-344bc0cb7ef8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb69c39a-6ctbl\" (UID: \"0ddf3248-69bc-458b-b12e-344bc0cb7ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb69c39a-6ctbl"
Oct 30 21:00:50.871889 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:50.871895    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnsc8\" (UniqueName: \"kubernetes.io/projected/0ddf3248-69bc-458b-b12e-344bc0cb7ef8-kube-api-access-mnsc8\") pod \"maas-api-revoke-sub-fb69c39a-6ctbl\" (UID: \"0ddf3248-69bc-458b-b12e-344bc0cb7ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb69c39a-6ctbl"
Oct 30 21:00:50.872120 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:50.872014    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ddf3248-69bc-458b-b12e-344bc0cb7ef8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb69c39a-6ctbl\" (UID: \"0ddf3248-69bc-458b-b12e-344bc0cb7ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb69c39a-6ctbl"
Oct 30 21:00:50.973732 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:50.973692    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ddf3248-69bc-458b-b12e-344bc0cb7ef8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb69c39a-6ctbl\" (UID: \"0ddf3248-69bc-458b-b12e-344bc0cb7ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb69c39a-6ctbl"
Oct 30 21:00:50.974261 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:50.973747    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnsc8\" (UniqueName: \"kubernetes.io/projected/0ddf3248-69bc-458b-b12e-344bc0cb7ef8-kube-api-access-mnsc8\") pod \"maas-api-revoke-sub-fb69c39a-6ctbl\" (UID: \"0ddf3248-69bc-458b-b12e-344bc0cb7ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb69c39a-6ctbl"
Oct 30 21:00:50.974261 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:50.973845    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ddf3248-69bc-458b-b12e-344bc0cb7ef8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb69c39a-6ctbl\" (UID: \"0ddf3248-69bc-458b-b12e-344bc0cb7ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb69c39a-6ctbl"
Oct 30 21:00:50.974861 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:50.974833    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ddf3248-69bc-458b-b12e-344bc0cb7ef8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb69c39a-6ctbl\" (UID: \"0ddf3248-69bc-458b-b12e-344bc0cb7ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb69c39a-6ctbl"
Oct 30 21:00:50.982582 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:50.982554    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ddf3248-69bc-458b-b12e-344bc0cb7ef8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb69c39a-6ctbl\" (UID: \"0ddf3248-69bc-458b-b12e-344bc0cb7ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb69c39a-6ctbl"
Oct 30 21:00:51.042066 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:51.040575    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnsc8\" (UniqueName: \"kubernetes.io/projected/0ddf3248-69bc-458b-b12e-344bc0cb7ef8-kube-api-access-mnsc8\") pod \"maas-api-revoke-sub-fb69c39a-6ctbl\" (UID: \"0ddf3248-69bc-458b-b12e-344bc0cb7ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb69c39a-6ctbl"
Oct 30 21:00:51.061861 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:51.058903    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb69c39a-6ctbl"
Oct 30 21:00:51.114417 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:51.114382    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d5ec58-5k6ft"
Oct 30 21:00:51.278058 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:51.278008    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98feba90-5e8f-443e-8c3e-0cfb79875caa-kube-api-access-vl9wj\" (UniqueName: \"kubernetes.io/projected/98feba90-5e8f-443e-8c3e-0cfb79875caa-kube-api-access-vl9wj\") pod \"98feba90-5e8f-443e-8c3e-0cfb79875caa\" (UID: \"98feba90-5e8f-443e-8c3e-0cfb79875caa\") "
Oct 30 21:00:51.278238 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:51.278213    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98feba90-5e8f-443e-8c3e-0cfb79875caa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98feba90-5e8f-443e-8c3e-0cfb79875caa-maas-api-xks-ca\") pod \"98feba90-5e8f-443e-8c3e-0cfb79875caa\" (UID: \"98feba90-5e8f-443e-8c3e-0cfb79875caa\") "
Oct 30 21:00:51.278238 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:51.278233    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98feba90-5e8f-443e-8c3e-0cfb79875caa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98feba90-5e8f-443e-8c3e-0cfb79875caa-maas-api-openshift-ca\") pod \"98feba90-5e8f-443e-8c3e-0cfb79875caa\" (UID: \"98feba90-5e8f-443e-8c3e-0cfb79875caa\") "
Oct 30 21:00:51.278647 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:51.278620    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98feba90-5e8f-443e-8c3e-0cfb79875caa-maas-api-openshift-ca" pod "98feba90-5e8f-443e-8c3e-0cfb79875caa" (UID: "98feba90-5e8f-443e-8c3e-0cfb79875caa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:51.278894 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:51.278850    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98feba90-5e8f-443e-8c3e-0cfb79875caa-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:51.280259 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:51.280236    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98feba90-5e8f-443e-8c3e-0cfb79875caa-kube-api-access-vl9wj" pod "98feba90-5e8f-443e-8c3e-0cfb79875caa" (UID: "98feba90-5e8f-443e-8c3e-0cfb79875caa"). InnerVolumeSpecName "kube-api-access-vl9wj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:51.280427 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:51.280398    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98feba90-5e8f-443e-8c3e-0cfb79875caa-maas-api-xks-ca" pod "98feba90-5e8f-443e-8c3e-0cfb79875caa" (UID: "98feba90-5e8f-443e-8c3e-0cfb79875caa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:00:51.380307 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:51.380204    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98feba90-5e8f-443e-8c3e-0cfb79875caa-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:51.380307 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:51.380249    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vl9wj\" (UniqueName: \"kubernetes.io/projected/98feba90-5e8f-443e-8c3e-0cfb79875caa-kube-api-access-vl9wj\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:51.450253 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:00:51.450222    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ddf3248_69bc_458b_b12e_344bc0cb7ef8.slice/crio-3650ebde545b6313b6980fccbd8bb5a03beb149ac103708b5fde1420d5dd9d57 WatchSource:0}: Error finding container 3650ebde545b6313b6980fccbd8bb5a03beb149ac103708b5fde1420d5dd9d57: Status 404 returned error can't find the container with id 3650ebde545b6313b6980fccbd8bb5a03beb149ac103708b5fde1420d5dd9d57
Oct 30 21:00:51.450624 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:51.450601    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb69c39a-6ctbl"]
Oct 30 21:00:51.909301 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:51.909217    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d5ec58-5k6ft"
Oct 30 21:00:51.909301 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:51.909241    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6d5ec58-5k6ft" event={"ID":"98feba90-5e8f-443e-8c3e-0cfb79875caa","Type":"ContainerDied","Data":"2dfe5f0c6ab2c13629bd11482120a7045ce1d838da70c5bdbaac67b2b1715ffb"}
Oct 30 21:00:51.909301 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:51.909292    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2dfe5f0c6ab2c13629bd11482120a7045ce1d838da70c5bdbaac67b2b1715ffb"
Oct 30 21:00:51.911275 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:51.911230    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb69c39a-6ctbl" event={"ID":"0ddf3248-69bc-458b-b12e-344bc0cb7ef8","Type":"ContainerStarted","Data":"6b5629c2a2cbde3e949ca0e55b04cfdf8785a5b9e5f750ac66d5df408de585db"}
Oct 30 21:00:51.911428 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:51.911294    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb69c39a-6ctbl" event={"ID":"0ddf3248-69bc-458b-b12e-344bc0cb7ef8","Type":"ContainerStarted","Data":"3650ebde545b6313b6980fccbd8bb5a03beb149ac103708b5fde1420d5dd9d57"}
Oct 30 21:00:52.026779 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:52.026721    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb69c39a-6ctbl" podStartSLOduration=2.026704416 podStartE2EDuration="2.026704416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:52.023567273 +0000 UTC m=+1335.088343674" watchObservedRunningTime="2026-10-30 21:00:52.026704416 +0000 UTC m=+1335.091480815"
Oct 30 21:00:52.814777 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:52.814743    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12aab4d8-ntbqw"]
Oct 30 21:00:52.816004 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:52.815985    5713 cpu_manager.go:414] "containerMap: removing container" podUID="98feba90-5e8f-443e-8c3e-0cfb79875caa" containerName="revoke-keys"
Oct 30 21:00:52.816065 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:52.816007    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98feba90-5e8f-443e-8c3e-0cfb79875caa" containerName="revoke-keys"
Oct 30 21:00:52.816197 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:52.816183    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="98feba90-5e8f-443e-8c3e-0cfb79875caa" containerName="revoke-keys"
Oct 30 21:00:52.820518 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:52.820499    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12aab4d8-ntbqw"
Oct 30 21:00:52.899412 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:52.899371    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12aab4d8-ntbqw"]
Oct 30 21:00:52.900034 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:52.900006    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlpmr\" (UniqueName: \"kubernetes.io/projected/5b827544-d79c-42dc-aeb5-12610c1c4674-kube-api-access-zlpmr\") pod \"maas-api-revoke-sub-12aab4d8-ntbqw\" (UID: \"5b827544-d79c-42dc-aeb5-12610c1c4674\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12aab4d8-ntbqw"
Oct 30 21:00:52.900175 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:52.900054    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b827544-d79c-42dc-aeb5-12610c1c4674-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12aab4d8-ntbqw\" (UID: \"5b827544-d79c-42dc-aeb5-12610c1c4674\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12aab4d8-ntbqw"
Oct 30 21:00:52.900242 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:52.900211    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b827544-d79c-42dc-aeb5-12610c1c4674-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12aab4d8-ntbqw\" (UID: \"5b827544-d79c-42dc-aeb5-12610c1c4674\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12aab4d8-ntbqw"
Oct 30 21:00:52.923869 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:52.923831    5713 generic.go:356] "Generic (PLEG): container finished" podID="0ddf3248-69bc-458b-b12e-344bc0cb7ef8" containerID="6b5629c2a2cbde3e949ca0e55b04cfdf8785a5b9e5f750ac66d5df408de585db" exitCode=0
Oct 30 21:00:52.924059 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:52.923927    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb69c39a-6ctbl" event={"ID":"0ddf3248-69bc-458b-b12e-344bc0cb7ef8","Type":"ContainerDied","Data":"6b5629c2a2cbde3e949ca0e55b04cfdf8785a5b9e5f750ac66d5df408de585db"}
Oct 30 21:00:53.002068 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:53.002029    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlpmr\" (UniqueName: \"kubernetes.io/projected/5b827544-d79c-42dc-aeb5-12610c1c4674-kube-api-access-zlpmr\") pod \"maas-api-revoke-sub-12aab4d8-ntbqw\" (UID: \"5b827544-d79c-42dc-aeb5-12610c1c4674\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12aab4d8-ntbqw"
Oct 30 21:00:53.002244 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:53.002100    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b827544-d79c-42dc-aeb5-12610c1c4674-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12aab4d8-ntbqw\" (UID: \"5b827544-d79c-42dc-aeb5-12610c1c4674\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12aab4d8-ntbqw"
Oct 30 21:00:53.002244 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:53.002146    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b827544-d79c-42dc-aeb5-12610c1c4674-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12aab4d8-ntbqw\" (UID: \"5b827544-d79c-42dc-aeb5-12610c1c4674\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12aab4d8-ntbqw"
Oct 30 21:00:53.002851 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:53.002826    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b827544-d79c-42dc-aeb5-12610c1c4674-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12aab4d8-ntbqw\" (UID: \"5b827544-d79c-42dc-aeb5-12610c1c4674\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12aab4d8-ntbqw"
Oct 30 21:00:53.005071 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:53.005043    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b827544-d79c-42dc-aeb5-12610c1c4674-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12aab4d8-ntbqw\" (UID: \"5b827544-d79c-42dc-aeb5-12610c1c4674\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12aab4d8-ntbqw"
Oct 30 21:00:53.059628 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:53.059596    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlpmr\" (UniqueName: \"kubernetes.io/projected/5b827544-d79c-42dc-aeb5-12610c1c4674-kube-api-access-zlpmr\") pod \"maas-api-revoke-sub-12aab4d8-ntbqw\" (UID: \"5b827544-d79c-42dc-aeb5-12610c1c4674\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12aab4d8-ntbqw"
Oct 30 21:00:53.148904 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:53.148823    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12aab4d8-ntbqw"
Oct 30 21:00:53.546523 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:53.546491    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12aab4d8-ntbqw"]
Oct 30 21:00:53.546698 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:00:53.546660    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b827544_d79c_42dc_aeb5_12610c1c4674.slice/crio-7402ee238d49576d9cbf0481a4e88feec5be2dedfc30d7d5320d97b8dd7fb686 WatchSource:0}: Error finding container 7402ee238d49576d9cbf0481a4e88feec5be2dedfc30d7d5320d97b8dd7fb686: Status 404 returned error can't find the container with id 7402ee238d49576d9cbf0481a4e88feec5be2dedfc30d7d5320d97b8dd7fb686
Oct 30 21:00:53.934299 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:53.934260    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12aab4d8-ntbqw" event={"ID":"5b827544-d79c-42dc-aeb5-12610c1c4674","Type":"ContainerStarted","Data":"7402ee238d49576d9cbf0481a4e88feec5be2dedfc30d7d5320d97b8dd7fb686"}
Oct 30 21:00:54.105017 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:54.104943    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb"]
Oct 30 21:00:54.109692 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:54.109670    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb"
Oct 30 21:00:54.144814 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:54.129394    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb69c39a-6ctbl"
Oct 30 21:00:54.178811 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:54.174036    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb"]
Oct 30 21:00:54.242758 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:54.239342    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-f55142ef-gw-openshift-default-dockercfg-kw7t4\""
Oct 30 21:00:54.277943 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:54.277906    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ddf3248-69bc-458b-b12e-344bc0cb7ef8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ddf3248-69bc-458b-b12e-344bc0cb7ef8-maas-api-openshift-ca\") pod \"0ddf3248-69bc-458b-b12e-344bc0cb7ef8\" (UID: \"0ddf3248-69bc-458b-b12e-344bc0cb7ef8\") "
Oct 30 21:00:54.278256 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:54.278233    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ddf3248-69bc-458b-b12e-344bc0cb7ef8-kube-api-access-mnsc8\" (UniqueName: \"kubernetes.io/projected/0ddf3248-69bc-458b-b12e-344bc0cb7ef8-kube-api-access-mnsc8\") pod \"0ddf3248-69bc-458b-b12e-344bc0cb7ef8\" (UID: \"0ddf3248-69bc-458b-b12e-344bc0cb7ef8\") "
Oct 30 21:00:54.278425 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:54.278404    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ddf3248-69bc-458b-b12e-344bc0cb7ef8-maas-api-openshift-ca" pod "0ddf3248-69bc-458b-b12e-344bc0cb7ef8" (UID: "0ddf3248-69bc-458b-b12e-344bc0cb7ef8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:54.278602 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:54.278572    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ddf3248-69bc-458b-b12e-344bc0cb7ef8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ddf3248-69bc-458b-b12e-344bc0cb7ef8-maas-api-xks-ca\") pod \"0ddf3248-69bc-458b-b12e-344bc0cb7ef8\" (UID: \"0ddf3248-69bc-458b-b12e-344bc0cb7ef8\") "
Oct 30 21:00:54.279132 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:54.279089    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d335bbaf-1e6b-49cb-8bb1-89d987d18a90-istiod-ca-cert\") pod \"e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb\" (UID: \"d335bbaf-1e6b-49cb-8bb1-89d987d18a90\") " pod="openshift-ingress/e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb"
Oct 30 21:00:54.279293 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:54.279279    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d335bbaf-1e6b-49cb-8bb1-89d987d18a90-credential-socket\") pod \"e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb\" (UID: \"d335bbaf-1e6b-49cb-8bb1-89d987d18a90\") " pod="openshift-ingress/e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb"
Oct 30 21:00:54.279452 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:54.279430    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fqqq\" (UniqueName: \"kubernetes.io/projected/d335bbaf-1e6b-49cb-8bb1-89d987d18a90-kube-api-access-4fqqq\") pod \"e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb\" (UID: \"d335bbaf-1e6b-49cb-8bb1-89d987d18a90\") " pod="openshift-ingress/e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb"
Oct 30 21:00:54.279631 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:54.279609    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d335bbaf-1e6b-49cb-8bb1-89d987d18a90-istio-data\") pod \"e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb\" (UID: \"d335bbaf-1e6b-49cb-8bb1-89d987d18a90\") " pod="openshift-ingress/e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb"
Oct 30 21:00:54.279748 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:54.279736    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d335bbaf-1e6b-49cb-8bb1-89d987d18a90-istio-envoy\") pod \"e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb\" (UID: \"d335bbaf-1e6b-49cb-8bb1-89d987d18a90\") " pod="openshift-ingress/e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb"
Oct 30 21:00:54.279896 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:54.279872    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d335bbaf-1e6b-49cb-8bb1-89d987d18a90-istio-token\") pod \"e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb\" (UID: \"d335bbaf-1e6b-49cb-8bb1-89d987d18a90\") " pod="openshift-ingress/e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb"
Oct 30 21:00:54.280020 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:54.280007    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d335bbaf-1e6b-49cb-8bb1-89d987d18a90-workload-socket\") pod \"e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb\" (UID: \"d335bbaf-1e6b-49cb-8bb1-89d987d18a90\") " pod="openshift-ingress/e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb"
Oct 30 21:00:54.280203 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:54.280191    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d335bbaf-1e6b-49cb-8bb1-89d987d18a90-istio-podinfo\") pod \"e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb\" (UID: \"d335bbaf-1e6b-49cb-8bb1-89d987d18a90\") " pod="openshift-ingress/e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb"
Oct 30 21:00:54.280320 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:54.280308    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d335bbaf-1e6b-49cb-8bb1-89d987d18a90-workload-certs\") pod \"e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb\" (UID: \"d335bbaf-1e6b-49cb-8bb1-89d987d18a90\") " pod="openshift-ingress/e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb"
Oct 30 21:00:54.280494 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:54.280474    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ddf3248-69bc-458b-b12e-344bc0cb7ef8-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:54.282833 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:54.282724    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ddf3248-69bc-458b-b12e-344bc0cb7ef8-kube-api-access-mnsc8" pod "0ddf3248-69bc-458b-b12e-344bc0cb7ef8" (UID: "0ddf3248-69bc-458b-b12e-344bc0cb7ef8"). InnerVolumeSpecName "kube-api-access-mnsc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:54.285258 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:54.285168    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ddf3248-69bc-458b-b12e-344bc0cb7ef8-maas-api-xks-ca" pod "0ddf3248-69bc-458b-b12e-344bc0cb7ef8" (UID: "0ddf3248-69bc-458b-b12e-344bc0cb7ef8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:00:54.382432 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:54.382342    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d335bbaf-1e6b-49cb-8bb1-89d987d18a90-credential-socket\") pod \"e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb\" (UID: \"d335bbaf-1e6b-49cb-8bb1-89d987d18a90\") " pod="openshift-ingress/e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb"
Oct 30 21:00:54.382589 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:54.382428    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fqqq\" (UniqueName: \"kubernetes.io/projected/d335bbaf-1e6b-49cb-8bb1-89d987d18a90-kube-api-access-4fqqq\") pod \"e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb\" (UID: \"d335bbaf-1e6b-49cb-8bb1-89d987d18a90\") " pod="openshift-ingress/e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb"
Oct 30 21:00:54.382589 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:54.382539    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d335bbaf-1e6b-49cb-8bb1-89d987d18a90-istio-data\") pod \"e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb\" (UID: \"d335bbaf-1e6b-49cb-8bb1-89d987d18a90\") " pod="openshift-ingress/e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb"
Oct 30 21:00:54.382589 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:54.382572    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d335bbaf-1e6b-49cb-8bb1-89d987d18a90-istio-envoy\") pod \"e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb\" (UID: \"d335bbaf-1e6b-49cb-8bb1-89d987d18a90\") " pod="openshift-ingress/e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb"
Oct 30 21:00:54.382857 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:54.382619    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d335bbaf-1e6b-49cb-8bb1-89d987d18a90-istio-token\") pod \"e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb\" (UID: \"d335bbaf-1e6b-49cb-8bb1-89d987d18a90\") " pod="openshift-ingress/e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb"
Oct 30 21:00:54.382857 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:54.382676    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d335bbaf-1e6b-49cb-8bb1-89d987d18a90-workload-socket\") pod \"e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb\" (UID: \"d335bbaf-1e6b-49cb-8bb1-89d987d18a90\") " pod="openshift-ingress/e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb"
Oct 30 21:00:54.382857 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:54.382776    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d335bbaf-1e6b-49cb-8bb1-89d987d18a90-istio-podinfo\") pod \"e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb\" (UID: \"d335bbaf-1e6b-49cb-8bb1-89d987d18a90\") " pod="openshift-ingress/e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb"
Oct 30 21:00:54.382857 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:54.382829    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d335bbaf-1e6b-49cb-8bb1-89d987d18a90-workload-certs\") pod \"e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb\" (UID: \"d335bbaf-1e6b-49cb-8bb1-89d987d18a90\") " pod="openshift-ingress/e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb"
Oct 30 21:00:54.383116 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:54.382865    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d335bbaf-1e6b-49cb-8bb1-89d987d18a90-credential-socket\") pod \"e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb\" (UID: \"d335bbaf-1e6b-49cb-8bb1-89d987d18a90\") " pod="openshift-ingress/e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb"
Oct 30 21:00:54.383116 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:54.382946    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d335bbaf-1e6b-49cb-8bb1-89d987d18a90-istio-data\") pod \"e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb\" (UID: \"d335bbaf-1e6b-49cb-8bb1-89d987d18a90\") " pod="openshift-ingress/e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb"
Oct 30 21:00:54.383116 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:54.382983    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d335bbaf-1e6b-49cb-8bb1-89d987d18a90-istiod-ca-cert\") pod \"e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb\" (UID: \"d335bbaf-1e6b-49cb-8bb1-89d987d18a90\") " pod="openshift-ingress/e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb"
Oct 30 21:00:54.383116 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:54.383099    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnsc8\" (UniqueName: \"kubernetes.io/projected/0ddf3248-69bc-458b-b12e-344bc0cb7ef8-kube-api-access-mnsc8\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:54.383116 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:54.383117    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ddf3248-69bc-458b-b12e-344bc0cb7ef8-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:54.383382 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:54.383187    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d335bbaf-1e6b-49cb-8bb1-89d987d18a90-workload-certs\") pod \"e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb\" (UID: \"d335bbaf-1e6b-49cb-8bb1-89d987d18a90\") " pod="openshift-ingress/e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb"
Oct 30 21:00:54.383382 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:54.383355    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d335bbaf-1e6b-49cb-8bb1-89d987d18a90-workload-socket\") pod \"e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb\" (UID: \"d335bbaf-1e6b-49cb-8bb1-89d987d18a90\") " pod="openshift-ingress/e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb"
Oct 30 21:00:54.383830 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:54.383767    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d335bbaf-1e6b-49cb-8bb1-89d987d18a90-istiod-ca-cert\") pod \"e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb\" (UID: \"d335bbaf-1e6b-49cb-8bb1-89d987d18a90\") " pod="openshift-ingress/e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb"
Oct 30 21:00:54.390934 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:54.386669    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d335bbaf-1e6b-49cb-8bb1-89d987d18a90-istio-envoy\") pod \"e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb\" (UID: \"d335bbaf-1e6b-49cb-8bb1-89d987d18a90\") " pod="openshift-ingress/e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb"
Oct 30 21:00:54.390934 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:54.388434    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d335bbaf-1e6b-49cb-8bb1-89d987d18a90-istio-podinfo\") pod \"e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb\" (UID: \"d335bbaf-1e6b-49cb-8bb1-89d987d18a90\") " pod="openshift-ingress/e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb"
Oct 30 21:00:54.445052 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:54.445018    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d335bbaf-1e6b-49cb-8bb1-89d987d18a90-istio-token\") pod \"e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb\" (UID: \"d335bbaf-1e6b-49cb-8bb1-89d987d18a90\") " pod="openshift-ingress/e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb"
Oct 30 21:00:54.445381 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:54.445360    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fqqq\" (UniqueName: \"kubernetes.io/projected/d335bbaf-1e6b-49cb-8bb1-89d987d18a90-kube-api-access-4fqqq\") pod \"e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb\" (UID: \"d335bbaf-1e6b-49cb-8bb1-89d987d18a90\") " pod="openshift-ingress/e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb"
Oct 30 21:00:54.467595 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:00:54.467557    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:00:54.545708 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:54.545667    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb"
Oct 30 21:00:54.943957 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:54.943929    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb69c39a-6ctbl"
Oct 30 21:00:54.944164 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:54.943952    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb69c39a-6ctbl" event={"ID":"0ddf3248-69bc-458b-b12e-344bc0cb7ef8","Type":"ContainerDied","Data":"3650ebde545b6313b6980fccbd8bb5a03beb149ac103708b5fde1420d5dd9d57"}
Oct 30 21:00:54.944164 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:54.944009    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3650ebde545b6313b6980fccbd8bb5a03beb149ac103708b5fde1420d5dd9d57"
Oct 30 21:00:54.956575 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:54.956547    5713 generic.go:356] "Generic (PLEG): container finished" podID="5b827544-d79c-42dc-aeb5-12610c1c4674" containerID="82dca2879f5ad45dc89f16a1132a80d1e06b803231c358bb7202def49f156f3b" exitCode=0
Oct 30 21:00:54.956708 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:54.956634    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12aab4d8-ntbqw" event={"ID":"5b827544-d79c-42dc-aeb5-12610c1c4674","Type":"ContainerDied","Data":"82dca2879f5ad45dc89f16a1132a80d1e06b803231c358bb7202def49f156f3b"}
Oct 30 21:00:55.007354 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:00:55.007321    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd335bbaf_1e6b_49cb_8bb1_89d987d18a90.slice/crio-4652051b1da9d2f8943e3908498219514c9c45a7e789359195c4eb9d5f66ad03 WatchSource:0}: Error finding container 4652051b1da9d2f8943e3908498219514c9c45a7e789359195c4eb9d5f66ad03: Status 404 returned error can't find the container with id 4652051b1da9d2f8943e3908498219514c9c45a7e789359195c4eb9d5f66ad03
Oct 30 21:00:55.009643 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:55.009617    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb"]
Oct 30 21:00:55.010130 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:55.010101    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:00:55.010194 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:55.010164    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:00:55.010247 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:55.010207    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:00:55.966898 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:55.966864    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb" event={"ID":"d335bbaf-1e6b-49cb-8bb1-89d987d18a90","Type":"ContainerStarted","Data":"a1f615299e1b3df5d35d745da95a15e5ee3e605e0fb9efb025bd729eb3163eab"}
Oct 30 21:00:55.967334 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:55.966920    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb" event={"ID":"d335bbaf-1e6b-49cb-8bb1-89d987d18a90","Type":"ContainerStarted","Data":"4652051b1da9d2f8943e3908498219514c9c45a7e789359195c4eb9d5f66ad03"}
Oct 30 21:00:56.157707 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:56.157678    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12aab4d8-ntbqw"
Oct 30 21:00:56.272689 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:56.272556    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb" podStartSLOduration=2.272535979 podStartE2EDuration="2.272535979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:56.144116803 +0000 UTC m=+1339.208893204" watchObservedRunningTime="2026-10-30 21:00:56.272535979 +0000 UTC m=+1339.337312378"
Oct 30 21:00:56.311741 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:56.311697    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b827544-d79c-42dc-aeb5-12610c1c4674-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b827544-d79c-42dc-aeb5-12610c1c4674-maas-api-openshift-ca\") pod \"5b827544-d79c-42dc-aeb5-12610c1c4674\" (UID: \"5b827544-d79c-42dc-aeb5-12610c1c4674\") "
Oct 30 21:00:56.312013 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:56.311995    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b827544-d79c-42dc-aeb5-12610c1c4674-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b827544-d79c-42dc-aeb5-12610c1c4674-maas-api-xks-ca\") pod \"5b827544-d79c-42dc-aeb5-12610c1c4674\" (UID: \"5b827544-d79c-42dc-aeb5-12610c1c4674\") "
Oct 30 21:00:56.312070 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:56.312041    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b827544-d79c-42dc-aeb5-12610c1c4674-kube-api-access-zlpmr\" (UniqueName: \"kubernetes.io/projected/5b827544-d79c-42dc-aeb5-12610c1c4674-kube-api-access-zlpmr\") pod \"5b827544-d79c-42dc-aeb5-12610c1c4674\" (UID: \"5b827544-d79c-42dc-aeb5-12610c1c4674\") "
Oct 30 21:00:56.312185 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:56.312157    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b827544-d79c-42dc-aeb5-12610c1c4674-maas-api-openshift-ca" pod "5b827544-d79c-42dc-aeb5-12610c1c4674" (UID: "5b827544-d79c-42dc-aeb5-12610c1c4674"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:00:56.312918 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:56.312723    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b827544-d79c-42dc-aeb5-12610c1c4674-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:56.315050 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:56.315020    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b827544-d79c-42dc-aeb5-12610c1c4674-maas-api-xks-ca" pod "5b827544-d79c-42dc-aeb5-12610c1c4674" (UID: "5b827544-d79c-42dc-aeb5-12610c1c4674"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:00:56.315134 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:56.315047    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b827544-d79c-42dc-aeb5-12610c1c4674-kube-api-access-zlpmr" pod "5b827544-d79c-42dc-aeb5-12610c1c4674" (UID: "5b827544-d79c-42dc-aeb5-12610c1c4674"). InnerVolumeSpecName "kube-api-access-zlpmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:00:56.415816 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:56.415169    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b827544-d79c-42dc-aeb5-12610c1c4674-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:56.415816 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:56.415209    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlpmr\" (UniqueName: \"kubernetes.io/projected/5b827544-d79c-42dc-aeb5-12610c1c4674-kube-api-access-zlpmr\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:00:56.546442 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:56.546332    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb"
Oct 30 21:00:56.552419 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:56.552391    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb"
Oct 30 21:00:56.980065 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:56.980025    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12aab4d8-ntbqw" event={"ID":"5b827544-d79c-42dc-aeb5-12610c1c4674","Type":"ContainerDied","Data":"7402ee238d49576d9cbf0481a4e88feec5be2dedfc30d7d5320d97b8dd7fb686"}
Oct 30 21:00:56.980670 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:56.980099    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7402ee238d49576d9cbf0481a4e88feec5be2dedfc30d7d5320d97b8dd7fb686"
Oct 30 21:00:56.980670 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:56.980041    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12aab4d8-ntbqw"
Oct 30 21:00:56.980670 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:56.980460    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb"
Oct 30 21:00:56.981698 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:56.981675    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb"
Oct 30 21:00:57.751613 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:57.751571    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6946eef6-f5z25"]
Oct 30 21:00:57.753138 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:57.753110    5713 cpu_manager.go:414] "containerMap: removing container" podUID="5b827544-d79c-42dc-aeb5-12610c1c4674" containerName="revoke-keys"
Oct 30 21:00:57.753138 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:57.753140    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b827544-d79c-42dc-aeb5-12610c1c4674" containerName="revoke-keys"
Oct 30 21:00:57.753319 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:57.753274    5713 cpu_manager.go:414] "containerMap: removing container" podUID="0ddf3248-69bc-458b-b12e-344bc0cb7ef8" containerName="revoke-keys"
Oct 30 21:00:57.753319 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:57.753283    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ddf3248-69bc-458b-b12e-344bc0cb7ef8" containerName="revoke-keys"
Oct 30 21:00:57.753615 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:57.753597    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ddf3248-69bc-458b-b12e-344bc0cb7ef8" containerName="revoke-keys"
Oct 30 21:00:57.753677 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:57.753633    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b827544-d79c-42dc-aeb5-12610c1c4674" containerName="revoke-keys"
Oct 30 21:00:57.759033 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:57.759005    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6946eef6-f5z25"
Oct 30 21:00:57.812176 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:57.812132    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:00:57.812379 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:57.812212    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:00:57.812379 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:57.812227    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:00:57.812379 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:57.812218    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:00:57.812379 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:57.812372    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6946eef6-f5z25"]
Oct 30 21:00:57.835336 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:57.835303    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/262866a6-b173-44f4-bebb-b129d4372f61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6946eef6-f5z25\" (UID: \"262866a6-b173-44f4-bebb-b129d4372f61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6946eef6-f5z25"
Oct 30 21:00:57.835476 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:57.835344    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/262866a6-b173-44f4-bebb-b129d4372f61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6946eef6-f5z25\" (UID: \"262866a6-b173-44f4-bebb-b129d4372f61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6946eef6-f5z25"
Oct 30 21:00:57.835476 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:57.835365    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzz2m\" (UniqueName: \"kubernetes.io/projected/262866a6-b173-44f4-bebb-b129d4372f61-kube-api-access-lzz2m\") pod \"maas-api-revoke-sub-6946eef6-f5z25\" (UID: \"262866a6-b173-44f4-bebb-b129d4372f61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6946eef6-f5z25"
Oct 30 21:00:57.936713 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:57.936667    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/262866a6-b173-44f4-bebb-b129d4372f61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6946eef6-f5z25\" (UID: \"262866a6-b173-44f4-bebb-b129d4372f61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6946eef6-f5z25"
Oct 30 21:00:57.936713 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:57.936718    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/262866a6-b173-44f4-bebb-b129d4372f61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6946eef6-f5z25\" (UID: \"262866a6-b173-44f4-bebb-b129d4372f61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6946eef6-f5z25"
Oct 30 21:00:57.937018 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:57.936738    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzz2m\" (UniqueName: \"kubernetes.io/projected/262866a6-b173-44f4-bebb-b129d4372f61-kube-api-access-lzz2m\") pod \"maas-api-revoke-sub-6946eef6-f5z25\" (UID: \"262866a6-b173-44f4-bebb-b129d4372f61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6946eef6-f5z25"
Oct 30 21:00:57.937514 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:57.937489    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/262866a6-b173-44f4-bebb-b129d4372f61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6946eef6-f5z25\" (UID: \"262866a6-b173-44f4-bebb-b129d4372f61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6946eef6-f5z25"
Oct 30 21:00:57.939728 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:57.939703    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/262866a6-b173-44f4-bebb-b129d4372f61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6946eef6-f5z25\" (UID: \"262866a6-b173-44f4-bebb-b129d4372f61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6946eef6-f5z25"
Oct 30 21:00:58.000498 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:58.000441    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzz2m\" (UniqueName: \"kubernetes.io/projected/262866a6-b173-44f4-bebb-b129d4372f61-kube-api-access-lzz2m\") pod \"maas-api-revoke-sub-6946eef6-f5z25\" (UID: \"262866a6-b173-44f4-bebb-b129d4372f61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6946eef6-f5z25"
Oct 30 21:00:58.073947 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:58.073861    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6946eef6-f5z25"
Oct 30 21:00:58.457498 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:00:58.457461    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod262866a6_b173_44f4_bebb_b129d4372f61.slice/crio-25b18b04e5a48d151f15418f49defa4ddfc89ccb14cef1106195d27393737610 WatchSource:0}: Error finding container 25b18b04e5a48d151f15418f49defa4ddfc89ccb14cef1106195d27393737610: Status 404 returned error can't find the container with id 25b18b04e5a48d151f15418f49defa4ddfc89ccb14cef1106195d27393737610
Oct 30 21:00:58.458230 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:58.458202    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6946eef6-f5z25"]
Oct 30 21:00:59.002696 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:59.002580    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6946eef6-f5z25" event={"ID":"262866a6-b173-44f4-bebb-b129d4372f61","Type":"ContainerStarted","Data":"f9650b438486baaebd42a342d20223691b469a09e17a89de329aba9c3a302650"}
Oct 30 21:00:59.002696 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:59.002662    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6946eef6-f5z25" event={"ID":"262866a6-b173-44f4-bebb-b129d4372f61","Type":"ContainerStarted","Data":"25b18b04e5a48d151f15418f49defa4ddfc89ccb14cef1106195d27393737610"}
Oct 30 21:00:59.150216 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:00:59.150160    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6946eef6-f5z25" podStartSLOduration=2.150145134 podStartE2EDuration="2.150145134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:59.143724284 +0000 UTC m=+1342.208500694" watchObservedRunningTime="2026-10-30 21:00:59.150145134 +0000 UTC m=+1342.214921532"
Oct 30 21:01:00.012285 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:00.012249    5713 generic.go:356] "Generic (PLEG): container finished" podID="262866a6-b173-44f4-bebb-b129d4372f61" containerID="f9650b438486baaebd42a342d20223691b469a09e17a89de329aba9c3a302650" exitCode=0
Oct 30 21:01:00.012700 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:00.012347    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6946eef6-f5z25" event={"ID":"262866a6-b173-44f4-bebb-b129d4372f61","Type":"ContainerDied","Data":"f9650b438486baaebd42a342d20223691b469a09e17a89de329aba9c3a302650"}
Oct 30 21:01:01.191324 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:01.191302    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6946eef6-f5z25"
Oct 30 21:01:01.289147 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:01.289058    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/262866a6-b173-44f4-bebb-b129d4372f61-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/262866a6-b173-44f4-bebb-b129d4372f61-maas-api-openshift-ca\") pod \"262866a6-b173-44f4-bebb-b129d4372f61\" (UID: \"262866a6-b173-44f4-bebb-b129d4372f61\") "
Oct 30 21:01:01.289390 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:01.289366    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/262866a6-b173-44f4-bebb-b129d4372f61-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/262866a6-b173-44f4-bebb-b129d4372f61-maas-api-xks-ca\") pod \"262866a6-b173-44f4-bebb-b129d4372f61\" (UID: \"262866a6-b173-44f4-bebb-b129d4372f61\") "
Oct 30 21:01:01.289452 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:01.289437    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/262866a6-b173-44f4-bebb-b129d4372f61-kube-api-access-lzz2m\" (UniqueName: \"kubernetes.io/projected/262866a6-b173-44f4-bebb-b129d4372f61-kube-api-access-lzz2m\") pod \"262866a6-b173-44f4-bebb-b129d4372f61\" (UID: \"262866a6-b173-44f4-bebb-b129d4372f61\") "
Oct 30 21:01:01.289496 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:01.289475    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/262866a6-b173-44f4-bebb-b129d4372f61-maas-api-openshift-ca" pod "262866a6-b173-44f4-bebb-b129d4372f61" (UID: "262866a6-b173-44f4-bebb-b129d4372f61"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:01:01.290060 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:01.290039    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/262866a6-b173-44f4-bebb-b129d4372f61-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:01.291560 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:01.291530    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/262866a6-b173-44f4-bebb-b129d4372f61-maas-api-xks-ca" pod "262866a6-b173-44f4-bebb-b129d4372f61" (UID: "262866a6-b173-44f4-bebb-b129d4372f61"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:01:01.291760 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:01.291733    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/262866a6-b173-44f4-bebb-b129d4372f61-kube-api-access-lzz2m" pod "262866a6-b173-44f4-bebb-b129d4372f61" (UID: "262866a6-b173-44f4-bebb-b129d4372f61"). InnerVolumeSpecName "kube-api-access-lzz2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:01:01.391303 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:01.391270    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/262866a6-b173-44f4-bebb-b129d4372f61-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:01.391467 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:01.391323    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzz2m\" (UniqueName: \"kubernetes.io/projected/262866a6-b173-44f4-bebb-b129d4372f61-kube-api-access-lzz2m\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:02.032019 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:02.031991    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6946eef6-f5z25"
Oct 30 21:01:02.032203 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:02.032027    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6946eef6-f5z25" event={"ID":"262866a6-b173-44f4-bebb-b129d4372f61","Type":"ContainerDied","Data":"25b18b04e5a48d151f15418f49defa4ddfc89ccb14cef1106195d27393737610"}
Oct 30 21:01:02.032203 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:02.032121    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25b18b04e5a48d151f15418f49defa4ddfc89ccb14cef1106195d27393737610"
Oct 30 21:01:02.854149 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:02.854118    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59"]
Oct 30 21:01:02.854728 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:02.854714    5713 cpu_manager.go:414] "containerMap: removing container" podUID="262866a6-b173-44f4-bebb-b129d4372f61" containerName="revoke-keys"
Oct 30 21:01:02.854818 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:02.854729    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="262866a6-b173-44f4-bebb-b129d4372f61" containerName="revoke-keys"
Oct 30 21:01:02.854889 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:02.854879    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="262866a6-b173-44f4-bebb-b129d4372f61" containerName="revoke-keys"
Oct 30 21:01:02.858526 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:02.858507    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59"
Oct 30 21:01:02.902711 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:02.902680    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz"]
Oct 30 21:01:02.906935 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:02.906918    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz"
Oct 30 21:01:02.922306 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:02.922280    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59"]
Oct 30 21:01:02.981284 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:02.981250    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz"]
Oct 30 21:01:03.014702 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:03.014670    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27rm7\" (UniqueName: \"kubernetes.io/projected/caf52803-d06e-4366-ab90-4e06f15a3ce3-kube-api-access-27rm7\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz\" (UID: \"caf52803-d06e-4366-ab90-4e06f15a3ce3\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz"
Oct 30 21:01:03.014898 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:03.014767    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caf52803-d06e-4366-ab90-4e06f15a3ce3-istiod-ca-cert\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz\" (UID: \"caf52803-d06e-4366-ab90-4e06f15a3ce3\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz"
Oct 30 21:01:03.014898 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:03.014812    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb252301-e703-4e46-b569-87b1039211d9-credential-socket\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59\" (UID: \"eb252301-e703-4e46-b569-87b1039211d9\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59"
Oct 30 21:01:03.014898 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:03.014846    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf52803-d06e-4366-ab90-4e06f15a3ce3-credential-socket\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz\" (UID: \"caf52803-d06e-4366-ab90-4e06f15a3ce3\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz"
Oct 30 21:01:03.014898 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:03.014876    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caf52803-d06e-4366-ab90-4e06f15a3ce3-istio-data\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz\" (UID: \"caf52803-d06e-4366-ab90-4e06f15a3ce3\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz"
Oct 30 21:01:03.015057 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:03.014906    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb252301-e703-4e46-b569-87b1039211d9-istio-envoy\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59\" (UID: \"eb252301-e703-4e46-b569-87b1039211d9\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59"
Oct 30 21:01:03.015057 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:03.014930    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caf52803-d06e-4366-ab90-4e06f15a3ce3-istio-envoy\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz\" (UID: \"caf52803-d06e-4366-ab90-4e06f15a3ce3\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz"
Oct 30 21:01:03.015057 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:03.014962    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb252301-e703-4e46-b569-87b1039211d9-workload-socket\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59\" (UID: \"eb252301-e703-4e46-b569-87b1039211d9\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59"
Oct 30 21:01:03.015249 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:03.015127    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caf52803-d06e-4366-ab90-4e06f15a3ce3-istio-token\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz\" (UID: \"caf52803-d06e-4366-ab90-4e06f15a3ce3\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz"
Oct 30 21:01:03.015249 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:03.015154    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb252301-e703-4e46-b569-87b1039211d9-workload-certs\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59\" (UID: \"eb252301-e703-4e46-b569-87b1039211d9\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59"
Oct 30 21:01:03.015249 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:03.015187    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb252301-e703-4e46-b569-87b1039211d9-istio-token\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59\" (UID: \"eb252301-e703-4e46-b569-87b1039211d9\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59"
Oct 30 21:01:03.015249 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:03.015221    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q65n\" (UniqueName: \"kubernetes.io/projected/eb252301-e703-4e46-b569-87b1039211d9-kube-api-access-4q65n\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59\" (UID: \"eb252301-e703-4e46-b569-87b1039211d9\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59"
Oct 30 21:01:03.015461 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:03.015263    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb252301-e703-4e46-b569-87b1039211d9-istio-data\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59\" (UID: \"eb252301-e703-4e46-b569-87b1039211d9\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59"
Oct 30 21:01:03.015461 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:03.015369    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb252301-e703-4e46-b569-87b1039211d9-istiod-ca-cert\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59\" (UID: \"eb252301-e703-4e46-b569-87b1039211d9\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59"
Oct 30 21:01:03.015461 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:03.015395    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caf52803-d06e-4366-ab90-4e06f15a3ce3-workload-certs\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz\" (UID: \"caf52803-d06e-4366-ab90-4e06f15a3ce3\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz"
Oct 30 21:01:03.015461 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:03.015438    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb252301-e703-4e46-b569-87b1039211d9-istio-podinfo\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59\" (UID: \"eb252301-e703-4e46-b569-87b1039211d9\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59"
Oct 30 21:01:03.015664 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:03.015546    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caf52803-d06e-4366-ab90-4e06f15a3ce3-istio-podinfo\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz\" (UID: \"caf52803-d06e-4366-ab90-4e06f15a3ce3\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz"
Oct 30 21:01:03.015664 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:03.015582    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf52803-d06e-4366-ab90-4e06f15a3ce3-workload-socket\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz\" (UID: \"caf52803-d06e-4366-ab90-4e06f15a3ce3\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz"
Oct 30 21:01:03.116743 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:03.116649    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caf52803-d06e-4366-ab90-4e06f15a3ce3-istio-podinfo\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz\" (UID: \"caf52803-d06e-4366-ab90-4e06f15a3ce3\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz"
Oct 30 21:01:03.116743 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:03.116704    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf52803-d06e-4366-ab90-4e06f15a3ce3-workload-socket\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz\" (UID: \"caf52803-d06e-4366-ab90-4e06f15a3ce3\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz"
Oct 30 21:01:03.117012 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:03.116760    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27rm7\" (UniqueName: \"kubernetes.io/projected/caf52803-d06e-4366-ab90-4e06f15a3ce3-kube-api-access-27rm7\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz\" (UID: \"caf52803-d06e-4366-ab90-4e06f15a3ce3\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz"
Oct 30 21:01:03.117012 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:03.116921    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caf52803-d06e-4366-ab90-4e06f15a3ce3-istiod-ca-cert\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz\" (UID: \"caf52803-d06e-4366-ab90-4e06f15a3ce3\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz"
Oct 30 21:01:03.117012 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:03.116955    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb252301-e703-4e46-b569-87b1039211d9-credential-socket\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59\" (UID: \"eb252301-e703-4e46-b569-87b1039211d9\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59"
Oct 30 21:01:03.117012 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:03.116988    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf52803-d06e-4366-ab90-4e06f15a3ce3-credential-socket\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz\" (UID: \"caf52803-d06e-4366-ab90-4e06f15a3ce3\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz"
Oct 30 21:01:03.117231 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:03.117021    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caf52803-d06e-4366-ab90-4e06f15a3ce3-istio-data\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz\" (UID: \"caf52803-d06e-4366-ab90-4e06f15a3ce3\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz"
Oct 30 21:01:03.117231 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:03.117064    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb252301-e703-4e46-b569-87b1039211d9-istio-envoy\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59\" (UID: \"eb252301-e703-4e46-b569-87b1039211d9\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59"
Oct 30 21:01:03.117231 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:03.117096    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caf52803-d06e-4366-ab90-4e06f15a3ce3-istio-envoy\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz\" (UID: \"caf52803-d06e-4366-ab90-4e06f15a3ce3\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz"
Oct 30 21:01:03.117231 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:03.117130    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb252301-e703-4e46-b569-87b1039211d9-workload-socket\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59\" (UID: \"eb252301-e703-4e46-b569-87b1039211d9\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59"
Oct 30 21:01:03.117231 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:03.117192    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caf52803-d06e-4366-ab90-4e06f15a3ce3-istio-token\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz\" (UID: \"caf52803-d06e-4366-ab90-4e06f15a3ce3\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz"
Oct 30 21:01:03.117231 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:03.117217    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb252301-e703-4e46-b569-87b1039211d9-workload-certs\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59\" (UID: \"eb252301-e703-4e46-b569-87b1039211d9\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59"
Oct 30 21:01:03.117534 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:03.117245    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf52803-d06e-4366-ab90-4e06f15a3ce3-workload-socket\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz\" (UID: \"caf52803-d06e-4366-ab90-4e06f15a3ce3\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz"
Oct 30 21:01:03.117534 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:03.117249    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb252301-e703-4e46-b569-87b1039211d9-istio-token\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59\" (UID: \"eb252301-e703-4e46-b569-87b1039211d9\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59"
Oct 30 21:01:03.117534 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:03.117326    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q65n\" (UniqueName: \"kubernetes.io/projected/eb252301-e703-4e46-b569-87b1039211d9-kube-api-access-4q65n\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59\" (UID: \"eb252301-e703-4e46-b569-87b1039211d9\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59"
Oct 30 21:01:03.117534 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:03.117379    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb252301-e703-4e46-b569-87b1039211d9-istio-data\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59\" (UID: \"eb252301-e703-4e46-b569-87b1039211d9\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59"
Oct 30 21:01:03.117534 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:03.117424    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf52803-d06e-4366-ab90-4e06f15a3ce3-credential-socket\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz\" (UID: \"caf52803-d06e-4366-ab90-4e06f15a3ce3\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz"
Oct 30 21:01:03.117534 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:03.117428    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb252301-e703-4e46-b569-87b1039211d9-istiod-ca-cert\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59\" (UID: \"eb252301-e703-4e46-b569-87b1039211d9\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59"
Oct 30 21:01:03.117534 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:03.117477    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb252301-e703-4e46-b569-87b1039211d9-credential-socket\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59\" (UID: \"eb252301-e703-4e46-b569-87b1039211d9\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59"
Oct 30 21:01:03.117921 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:03.117749    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caf52803-d06e-4366-ab90-4e06f15a3ce3-workload-certs\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz\" (UID: \"caf52803-d06e-4366-ab90-4e06f15a3ce3\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz"
Oct 30 21:01:03.117921 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:03.117484    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caf52803-d06e-4366-ab90-4e06f15a3ce3-workload-certs\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz\" (UID: \"caf52803-d06e-4366-ab90-4e06f15a3ce3\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz"
Oct 30 21:01:03.117921 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:03.117869    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb252301-e703-4e46-b569-87b1039211d9-istio-podinfo\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59\" (UID: \"eb252301-e703-4e46-b569-87b1039211d9\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59"
Oct 30 21:01:03.118084 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:03.117871    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caf52803-d06e-4366-ab90-4e06f15a3ce3-istiod-ca-cert\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz\" (UID: \"caf52803-d06e-4366-ab90-4e06f15a3ce3\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz"
Oct 30 21:01:03.118143 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:03.118081    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caf52803-d06e-4366-ab90-4e06f15a3ce3-istio-data\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz\" (UID: \"caf52803-d06e-4366-ab90-4e06f15a3ce3\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz"
Oct 30 21:01:03.118204 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:03.118166    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb252301-e703-4e46-b569-87b1039211d9-istiod-ca-cert\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59\" (UID: \"eb252301-e703-4e46-b569-87b1039211d9\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59"
Oct 30 21:01:03.118552 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:03.118365    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb252301-e703-4e46-b569-87b1039211d9-workload-socket\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59\" (UID: \"eb252301-e703-4e46-b569-87b1039211d9\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59"
Oct 30 21:01:03.118552 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:03.118436    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb252301-e703-4e46-b569-87b1039211d9-istio-data\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59\" (UID: \"eb252301-e703-4e46-b569-87b1039211d9\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59"
Oct 30 21:01:03.118728 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:03.118594    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb252301-e703-4e46-b569-87b1039211d9-workload-certs\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59\" (UID: \"eb252301-e703-4e46-b569-87b1039211d9\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59"
Oct 30 21:01:03.119634 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:03.119612    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caf52803-d06e-4366-ab90-4e06f15a3ce3-istio-podinfo\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz\" (UID: \"caf52803-d06e-4366-ab90-4e06f15a3ce3\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz"
Oct 30 21:01:03.120311 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:03.120290    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caf52803-d06e-4366-ab90-4e06f15a3ce3-istio-envoy\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz\" (UID: \"caf52803-d06e-4366-ab90-4e06f15a3ce3\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz"
Oct 30 21:01:03.120683 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:03.120659    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb252301-e703-4e46-b569-87b1039211d9-istio-envoy\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59\" (UID: \"eb252301-e703-4e46-b569-87b1039211d9\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59"
Oct 30 21:01:03.120683 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:03.120675    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb252301-e703-4e46-b569-87b1039211d9-istio-podinfo\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59\" (UID: \"eb252301-e703-4e46-b569-87b1039211d9\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59"
Oct 30 21:01:03.175196 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:03.175161    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caf52803-d06e-4366-ab90-4e06f15a3ce3-istio-token\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz\" (UID: \"caf52803-d06e-4366-ab90-4e06f15a3ce3\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz"
Oct 30 21:01:03.175371 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:03.175287    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27rm7\" (UniqueName: \"kubernetes.io/projected/caf52803-d06e-4366-ab90-4e06f15a3ce3-kube-api-access-27rm7\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz\" (UID: \"caf52803-d06e-4366-ab90-4e06f15a3ce3\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz"
Oct 30 21:01:03.175864 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:03.175842    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q65n\" (UniqueName: \"kubernetes.io/projected/eb252301-e703-4e46-b569-87b1039211d9-kube-api-access-4q65n\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59\" (UID: \"eb252301-e703-4e46-b569-87b1039211d9\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59"
Oct 30 21:01:03.176235 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:03.176219    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb252301-e703-4e46-b569-87b1039211d9-istio-token\") pod \"e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59\" (UID: \"eb252301-e703-4e46-b569-87b1039211d9\") " pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59"
Oct 30 21:01:03.221302 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:03.221272    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz"
Oct 30 21:01:03.473105 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:03.473069    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59"
Oct 30 21:01:03.666615 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:03.666583    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz"]
Oct 30 21:01:03.673869 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:01:03.673843    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaf52803_d06e_4366_ab90_4e06f15a3ce3.slice/crio-0c2796172db0dd424fec7af62d1c4f70e128a77754fefe2447dfd346737963cd WatchSource:0}: Error finding container 0c2796172db0dd424fec7af62d1c4f70e128a77754fefe2447dfd346737963cd: Status 404 returned error can't find the container with id 0c2796172db0dd424fec7af62d1c4f70e128a77754fefe2447dfd346737963cd
Oct 30 21:01:03.676337 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:03.676306    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:01:03.676419 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:03.676373    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:01:03.676419 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:03.676406    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:01:03.899277 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:01:03.899235    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb252301_e703_4e46_b569_87b1039211d9.slice/crio-6cdaaa54ae16b5a481a0e227569241a3cfb1e4c8d34f38257c28644a852ea900 WatchSource:0}: Error finding container 6cdaaa54ae16b5a481a0e227569241a3cfb1e4c8d34f38257c28644a852ea900: Status 404 returned error can't find the container with id 6cdaaa54ae16b5a481a0e227569241a3cfb1e4c8d34f38257c28644a852ea900
Oct 30 21:01:03.899690 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:03.899575    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59"]
Oct 30 21:01:03.902069 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:03.902038    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:01:03.902162 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:03.902116    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:01:03.902231 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:03.902162    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:01:04.086867 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:04.085269    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz" event={"ID":"caf52803-d06e-4366-ab90-4e06f15a3ce3","Type":"ContainerStarted","Data":"dbb59b8d8c5d66ed12e8a6cab98ad7f72e48010f04a62672d6d7c41f02d0b9bc"}
Oct 30 21:01:04.086867 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:04.085329    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz" event={"ID":"caf52803-d06e-4366-ab90-4e06f15a3ce3","Type":"ContainerStarted","Data":"0c2796172db0dd424fec7af62d1c4f70e128a77754fefe2447dfd346737963cd"}
Oct 30 21:01:04.110984 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:04.110943    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59" event={"ID":"eb252301-e703-4e46-b569-87b1039211d9","Type":"ContainerStarted","Data":"76ecd8330c15b09ee14ec2cc7140992dd8e7ec4896965380dc97f8943ea1a64a"}
Oct 30 21:01:04.111181 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:04.111168    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59" event={"ID":"eb252301-e703-4e46-b569-87b1039211d9","Type":"ContainerStarted","Data":"6cdaaa54ae16b5a481a0e227569241a3cfb1e4c8d34f38257c28644a852ea900"}
Oct 30 21:01:04.209551 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:04.209501    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz" podStartSLOduration=2.209486443 podStartE2EDuration="2.209486443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:01:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:04.206185499 +0000 UTC m=+1347.270961892" watchObservedRunningTime="2026-10-30 21:01:04.209486443 +0000 UTC m=+1347.274262841"
Oct 30 21:01:04.222088 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:04.222054    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz"
Oct 30 21:01:04.225877 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:04.225842    5713 patch_prober.go:29] interesting pod/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Oct 30 21:01:04.226064 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:04.225911    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz" podUID="caf52803-d06e-4366-ab90-4e06f15a3ce3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Oct 30 21:01:04.328224 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:04.326981    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59" podStartSLOduration=2.326964436 podStartE2EDuration="2.326964436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:01:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:04.325153944 +0000 UTC m=+1347.389930344" watchObservedRunningTime="2026-10-30 21:01:04.326964436 +0000 UTC m=+1347.391740834"
Oct 30 21:01:04.473694 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:04.473653    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59"
Oct 30 21:01:04.475148 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:04.475123    5713 patch_prober.go:29] interesting pod/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Oct 30 21:01:04.475270 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:04.475170    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59" podUID="eb252301-e703-4e46-b569-87b1039211d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Oct 30 21:01:05.222583 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:05.222550    5713 patch_prober.go:29] interesting pod/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Oct 30 21:01:05.223018 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:05.222602    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz" podUID="caf52803-d06e-4366-ab90-4e06f15a3ce3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Oct 30 21:01:05.487380 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:05.486253    5713 patch_prober.go:29] interesting pod/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Oct 30 21:01:05.487380 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:05.486328    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59" podUID="eb252301-e703-4e46-b569-87b1039211d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Oct 30 21:01:06.222580 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:06.222549    5713 patch_prober.go:29] interesting pod/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Oct 30 21:01:06.222740 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:06.222601    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz" podUID="caf52803-d06e-4366-ab90-4e06f15a3ce3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Oct 30 21:01:06.481092 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:06.480998    5713 patch_prober.go:29] interesting pod/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Oct 30 21:01:06.481360 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:06.481329    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59" podUID="eb252301-e703-4e46-b569-87b1039211d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Oct 30 21:01:07.222028 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:07.221988    5713 patch_prober.go:29] interesting pod/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Oct 30 21:01:07.222220 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:07.222045    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz" podUID="caf52803-d06e-4366-ab90-4e06f15a3ce3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Oct 30 21:01:07.474333 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:07.474235    5713 patch_prober.go:29] interesting pod/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Oct 30 21:01:07.475160 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:07.475127    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59" podUID="eb252301-e703-4e46-b569-87b1039211d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Oct 30 21:01:08.222005 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:08.221967    5713 patch_prober.go:29] interesting pod/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Oct 30 21:01:08.222193 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:08.222040    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz" podUID="caf52803-d06e-4366-ab90-4e06f15a3ce3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Oct 30 21:01:08.476288 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:08.476190    5713 patch_prober.go:29] interesting pod/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Oct 30 21:01:08.476903 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:08.476867    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59" podUID="eb252301-e703-4e46-b569-87b1039211d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Oct 30 21:01:08.589760 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:08.589714    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e134f233-v8lrp"]
Oct 30 21:01:08.596168 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:08.596140    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e134f233-v8lrp"
Oct 30 21:01:08.667816 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:08.659640    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e134f233-v8lrp"]
Oct 30 21:01:08.681032 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:08.681001    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:01:08.694088 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:08.681747    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:01:08.694328 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:08.681832    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:01:08.694431 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:08.681855    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:01:08.694516 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:08.682754    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be5f6063-9e6b-4805-a07f-ee96f0d437f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e134f233-v8lrp\" (UID: \"be5f6063-9e6b-4805-a07f-ee96f0d437f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e134f233-v8lrp"
Oct 30 21:01:08.694679 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:08.694658    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be5f6063-9e6b-4805-a07f-ee96f0d437f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e134f233-v8lrp\" (UID: \"be5f6063-9e6b-4805-a07f-ee96f0d437f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e134f233-v8lrp"
Oct 30 21:01:08.695199 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:08.695171    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdnf7\" (UniqueName: \"kubernetes.io/projected/be5f6063-9e6b-4805-a07f-ee96f0d437f8-kube-api-access-pdnf7\") pod \"maas-api-revoke-sub-e134f233-v8lrp\" (UID: \"be5f6063-9e6b-4805-a07f-ee96f0d437f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e134f233-v8lrp"
Oct 30 21:01:08.796529 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:08.796445    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdnf7\" (UniqueName: \"kubernetes.io/projected/be5f6063-9e6b-4805-a07f-ee96f0d437f8-kube-api-access-pdnf7\") pod \"maas-api-revoke-sub-e134f233-v8lrp\" (UID: \"be5f6063-9e6b-4805-a07f-ee96f0d437f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e134f233-v8lrp"
Oct 30 21:01:08.796778 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:08.796765    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be5f6063-9e6b-4805-a07f-ee96f0d437f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e134f233-v8lrp\" (UID: \"be5f6063-9e6b-4805-a07f-ee96f0d437f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e134f233-v8lrp"
Oct 30 21:01:08.796939 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:08.796920    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be5f6063-9e6b-4805-a07f-ee96f0d437f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e134f233-v8lrp\" (UID: \"be5f6063-9e6b-4805-a07f-ee96f0d437f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e134f233-v8lrp"
Oct 30 21:01:08.797512 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:08.797488    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be5f6063-9e6b-4805-a07f-ee96f0d437f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e134f233-v8lrp\" (UID: \"be5f6063-9e6b-4805-a07f-ee96f0d437f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e134f233-v8lrp"
Oct 30 21:01:08.799468 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:08.799436    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be5f6063-9e6b-4805-a07f-ee96f0d437f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e134f233-v8lrp\" (UID: \"be5f6063-9e6b-4805-a07f-ee96f0d437f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e134f233-v8lrp"
Oct 30 21:01:08.867559 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:08.867518    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdnf7\" (UniqueName: \"kubernetes.io/projected/be5f6063-9e6b-4805-a07f-ee96f0d437f8-kube-api-access-pdnf7\") pod \"maas-api-revoke-sub-e134f233-v8lrp\" (UID: \"be5f6063-9e6b-4805-a07f-ee96f0d437f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e134f233-v8lrp"
Oct 30 21:01:08.912857 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:08.912783    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e134f233-v8lrp"
Oct 30 21:01:09.221932 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:09.221901    5713 patch_prober.go:29] interesting pod/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Oct 30 21:01:09.222187 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:09.222161    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz" podUID="caf52803-d06e-4366-ab90-4e06f15a3ce3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Oct 30 21:01:09.479444 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:09.479405    5713 patch_prober.go:29] interesting pod/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Oct 30 21:01:09.484139 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:09.484088    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59" podUID="eb252301-e703-4e46-b569-87b1039211d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Oct 30 21:01:09.503010 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:09.502983    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e134f233-v8lrp"]
Oct 30 21:01:09.795595 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:09.795510    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-83851760-6b5fd6887d-hcdv2"]
Oct 30 21:01:09.801604 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:09.801583    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-83851760-6b5fd6887d-hcdv2"
Oct 30 21:01:09.831220 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:09.831181    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v4mh\" (UniqueName: \"kubernetes.io/projected/ed3bfc15-452e-45a3-b7bf-545c615fcb34-kube-api-access-7v4mh\") pod \"payload-processing-e2e-ait-83851760-6b5fd6887d-hcdv2\" (UID: \"ed3bfc15-452e-45a3-b7bf-545c615fcb34\") " pod="openshift-ingress/payload-processing-e2e-ait-83851760-6b5fd6887d-hcdv2"
Oct 30 21:01:09.831416 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:09.831396    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed3bfc15-452e-45a3-b7bf-545c615fcb34-plugins-config-volume\") pod \"payload-processing-e2e-ait-83851760-6b5fd6887d-hcdv2\" (UID: \"ed3bfc15-452e-45a3-b7bf-545c615fcb34\") " pod="openshift-ingress/payload-processing-e2e-ait-83851760-6b5fd6887d-hcdv2"
Oct 30 21:01:10.029774 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:10.029725    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v4mh\" (UniqueName: \"kubernetes.io/projected/ed3bfc15-452e-45a3-b7bf-545c615fcb34-kube-api-access-7v4mh\") pod \"payload-processing-e2e-ait-83851760-6b5fd6887d-hcdv2\" (UID: \"ed3bfc15-452e-45a3-b7bf-545c615fcb34\") " pod="openshift-ingress/payload-processing-e2e-ait-83851760-6b5fd6887d-hcdv2"
Oct 30 21:01:10.030197 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:10.030178    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed3bfc15-452e-45a3-b7bf-545c615fcb34-plugins-config-volume\") pod \"payload-processing-e2e-ait-83851760-6b5fd6887d-hcdv2\" (UID: \"ed3bfc15-452e-45a3-b7bf-545c615fcb34\") " pod="openshift-ingress/payload-processing-e2e-ait-83851760-6b5fd6887d-hcdv2"
Oct 30 21:01:10.070828 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:10.045234    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-83851760\""
Oct 30 21:01:10.070828 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:10.046100    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-83851760-dockercfg-fxm2f\""
Oct 30 21:01:10.070828 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:10.069049    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed3bfc15-452e-45a3-b7bf-545c615fcb34-plugins-config-volume\") pod \"payload-processing-e2e-ait-83851760-6b5fd6887d-hcdv2\" (UID: \"ed3bfc15-452e-45a3-b7bf-545c615fcb34\") " pod="openshift-ingress/payload-processing-e2e-ait-83851760-6b5fd6887d-hcdv2"
Oct 30 21:01:10.098997 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:10.097948    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v4mh\" (UniqueName: \"kubernetes.io/projected/ed3bfc15-452e-45a3-b7bf-545c615fcb34-kube-api-access-7v4mh\") pod \"payload-processing-e2e-ait-83851760-6b5fd6887d-hcdv2\" (UID: \"ed3bfc15-452e-45a3-b7bf-545c615fcb34\") " pod="openshift-ingress/payload-processing-e2e-ait-83851760-6b5fd6887d-hcdv2"
Oct 30 21:01:10.118762 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:10.118727    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-83851760-6b5fd6887d-hcdv2"]
Oct 30 21:01:10.152837 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:10.152003    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-83851760-6b5fd6887d-hcdv2"
Oct 30 21:01:10.204159 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:10.204100    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ce9871f-vgscg"]
Oct 30 21:01:10.218280 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:10.218251    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ce9871f-vgscg"
Oct 30 21:01:10.226701 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:10.226661    5713 patch_prober.go:29] interesting pod/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Oct 30 21:01:10.226876 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:10.226725    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz" podUID="caf52803-d06e-4366-ab90-4e06f15a3ce3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Oct 30 21:01:10.237764 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:10.234489    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkk2x\" (UniqueName: \"kubernetes.io/projected/827ff375-c35d-4a58-8d30-16ad01bb0e08-kube-api-access-zkk2x\") pod \"maas-api-revoke-sub-9ce9871f-vgscg\" (UID: \"827ff375-c35d-4a58-8d30-16ad01bb0e08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ce9871f-vgscg"
Oct 30 21:01:10.237764 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:10.234579    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/827ff375-c35d-4a58-8d30-16ad01bb0e08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ce9871f-vgscg\" (UID: \"827ff375-c35d-4a58-8d30-16ad01bb0e08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ce9871f-vgscg"
Oct 30 21:01:10.237764 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:10.234677    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/827ff375-c35d-4a58-8d30-16ad01bb0e08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ce9871f-vgscg\" (UID: \"827ff375-c35d-4a58-8d30-16ad01bb0e08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ce9871f-vgscg"
Oct 30 21:01:10.260848 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:10.260712    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e134f233-v8lrp" event={"ID":"be5f6063-9e6b-4805-a07f-ee96f0d437f8","Type":"ContainerStarted","Data":"23c6f02a7a32b678e468cb3e3aa53d4bee0898d7bb4ddec8aebae84862c9612a"}
Oct 30 21:01:10.261419 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:10.261384    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ce9871f-vgscg"]
Oct 30 21:01:10.360092 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:10.359990    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkk2x\" (UniqueName: \"kubernetes.io/projected/827ff375-c35d-4a58-8d30-16ad01bb0e08-kube-api-access-zkk2x\") pod \"maas-api-revoke-sub-9ce9871f-vgscg\" (UID: \"827ff375-c35d-4a58-8d30-16ad01bb0e08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ce9871f-vgscg"
Oct 30 21:01:10.360262 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:10.360096    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/827ff375-c35d-4a58-8d30-16ad01bb0e08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ce9871f-vgscg\" (UID: \"827ff375-c35d-4a58-8d30-16ad01bb0e08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ce9871f-vgscg"
Oct 30 21:01:10.360262 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:10.360202    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/827ff375-c35d-4a58-8d30-16ad01bb0e08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ce9871f-vgscg\" (UID: \"827ff375-c35d-4a58-8d30-16ad01bb0e08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ce9871f-vgscg"
Oct 30 21:01:10.361929 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:10.361894    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/827ff375-c35d-4a58-8d30-16ad01bb0e08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ce9871f-vgscg\" (UID: \"827ff375-c35d-4a58-8d30-16ad01bb0e08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ce9871f-vgscg"
Oct 30 21:01:10.365507 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:10.365475    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/827ff375-c35d-4a58-8d30-16ad01bb0e08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ce9871f-vgscg\" (UID: \"827ff375-c35d-4a58-8d30-16ad01bb0e08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ce9871f-vgscg"
Oct 30 21:01:10.419316 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:10.419282    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkk2x\" (UniqueName: \"kubernetes.io/projected/827ff375-c35d-4a58-8d30-16ad01bb0e08-kube-api-access-zkk2x\") pod \"maas-api-revoke-sub-9ce9871f-vgscg\" (UID: \"827ff375-c35d-4a58-8d30-16ad01bb0e08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ce9871f-vgscg"
Oct 30 21:01:10.473971 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:10.473933    5713 patch_prober.go:29] interesting pod/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Oct 30 21:01:10.474148 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:10.474001    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59" podUID="eb252301-e703-4e46-b569-87b1039211d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Oct 30 21:01:10.580400 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:10.580366    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ce9871f-vgscg"
Oct 30 21:01:10.630627 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:10.630587    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-83851760-6b5fd6887d-hcdv2"]
Oct 30 21:01:11.017574 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:11.017537    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ce9871f-vgscg"]
Oct 30 21:01:11.018647 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:01:11.018620    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod827ff375_c35d_4a58_8d30_16ad01bb0e08.slice/crio-28e848408cd09275963395714bffa3a9eb5e0c80e530de1a56b45fecdb544bb9 WatchSource:0}: Error finding container 28e848408cd09275963395714bffa3a9eb5e0c80e530de1a56b45fecdb544bb9: Status 404 returned error can't find the container with id 28e848408cd09275963395714bffa3a9eb5e0c80e530de1a56b45fecdb544bb9
Oct 30 21:01:11.284851 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:11.284134    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-83851760-6b5fd6887d-hcdv2" event={"ID":"ed3bfc15-452e-45a3-b7bf-545c615fcb34","Type":"ContainerStarted","Data":"690479b2aa2bc4d7b1d12c65902960bbb247d29d0a1fc78a12a1a908ab61cf55"}
Oct 30 21:01:11.284851 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:11.284205    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-83851760-6b5fd6887d-hcdv2" event={"ID":"ed3bfc15-452e-45a3-b7bf-545c615fcb34","Type":"ContainerStarted","Data":"374572b3b12eabcd82711a1f091d2fa5b386cd4433768fe96a10ac7800a0190f"}
Oct 30 21:01:11.285554 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:11.285496    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-83851760-6b5fd6887d-hcdv2"
Oct 30 21:01:11.288195 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:11.288168    5713 generic.go:356] "Generic (PLEG): container finished" podID="be5f6063-9e6b-4805-a07f-ee96f0d437f8" containerID="87c8b7a07675405eb16ccb472f8aa5a30c2cf1c1c0a90b9626ad1fda019d858f" exitCode=0
Oct 30 21:01:11.288577 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:11.288328    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e134f233-v8lrp" event={"ID":"be5f6063-9e6b-4805-a07f-ee96f0d437f8","Type":"ContainerDied","Data":"87c8b7a07675405eb16ccb472f8aa5a30c2cf1c1c0a90b9626ad1fda019d858f"}
Oct 30 21:01:11.290760 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:11.290734    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ce9871f-vgscg" event={"ID":"827ff375-c35d-4a58-8d30-16ad01bb0e08","Type":"ContainerStarted","Data":"28e848408cd09275963395714bffa3a9eb5e0c80e530de1a56b45fecdb544bb9"}
Oct 30 21:01:11.403228 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:11.403160    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-83851760-6b5fd6887d-hcdv2" podStartSLOduration=2.4031414939999998 podStartE2EDuration="2.403141494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:01:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:11.397838179 +0000 UTC m=+1354.462614578" watchObservedRunningTime="2026-10-30 21:01:11.403141494 +0000 UTC m=+1354.467917892"
Oct 30 21:01:11.554118 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:11.554023    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz"
Oct 30 21:01:11.554509 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:11.554492    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz"
Oct 30 21:01:11.556227 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:11.556199    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz"
Oct 30 21:01:11.743264 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:11.742698    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59"
Oct 30 21:01:11.743734 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:11.743534    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59"
Oct 30 21:01:11.744615 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:11.744588    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59"
Oct 30 21:01:12.306052 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:12.306018    5713 generic.go:356] "Generic (PLEG): container finished" podID="827ff375-c35d-4a58-8d30-16ad01bb0e08" containerID="938bc496bb9e8f8d48fc0367306b91902888465072a459f7bcc820b511ca5319" exitCode=0
Oct 30 21:01:12.306216 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:12.306066    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ce9871f-vgscg" event={"ID":"827ff375-c35d-4a58-8d30-16ad01bb0e08","Type":"ContainerDied","Data":"938bc496bb9e8f8d48fc0367306b91902888465072a459f7bcc820b511ca5319"}
Oct 30 21:01:12.504422 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:12.504389    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e134f233-v8lrp"
Oct 30 21:01:12.610610 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:12.610564    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be5f6063-9e6b-4805-a07f-ee96f0d437f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be5f6063-9e6b-4805-a07f-ee96f0d437f8-maas-api-xks-ca\") pod \"be5f6063-9e6b-4805-a07f-ee96f0d437f8\" (UID: \"be5f6063-9e6b-4805-a07f-ee96f0d437f8\") "
Oct 30 21:01:12.610781 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:12.610712    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be5f6063-9e6b-4805-a07f-ee96f0d437f8-kube-api-access-pdnf7\" (UniqueName: \"kubernetes.io/projected/be5f6063-9e6b-4805-a07f-ee96f0d437f8-kube-api-access-pdnf7\") pod \"be5f6063-9e6b-4805-a07f-ee96f0d437f8\" (UID: \"be5f6063-9e6b-4805-a07f-ee96f0d437f8\") "
Oct 30 21:01:12.610892 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:12.610854    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be5f6063-9e6b-4805-a07f-ee96f0d437f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be5f6063-9e6b-4805-a07f-ee96f0d437f8-maas-api-openshift-ca\") pod \"be5f6063-9e6b-4805-a07f-ee96f0d437f8\" (UID: \"be5f6063-9e6b-4805-a07f-ee96f0d437f8\") "
Oct 30 21:01:12.611227 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:12.611201    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be5f6063-9e6b-4805-a07f-ee96f0d437f8-maas-api-openshift-ca" pod "be5f6063-9e6b-4805-a07f-ee96f0d437f8" (UID: "be5f6063-9e6b-4805-a07f-ee96f0d437f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:01:12.611473 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:12.611410    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be5f6063-9e6b-4805-a07f-ee96f0d437f8-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:12.612966 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:12.612938    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be5f6063-9e6b-4805-a07f-ee96f0d437f8-maas-api-xks-ca" pod "be5f6063-9e6b-4805-a07f-ee96f0d437f8" (UID: "be5f6063-9e6b-4805-a07f-ee96f0d437f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:01:12.613138 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:12.613122    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be5f6063-9e6b-4805-a07f-ee96f0d437f8-kube-api-access-pdnf7" pod "be5f6063-9e6b-4805-a07f-ee96f0d437f8" (UID: "be5f6063-9e6b-4805-a07f-ee96f0d437f8"). InnerVolumeSpecName "kube-api-access-pdnf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:01:12.713188 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:12.713147    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdnf7\" (UniqueName: \"kubernetes.io/projected/be5f6063-9e6b-4805-a07f-ee96f0d437f8-kube-api-access-pdnf7\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:12.713188 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:12.713181    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be5f6063-9e6b-4805-a07f-ee96f0d437f8-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:13.316991 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:13.316928    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e134f233-v8lrp" event={"ID":"be5f6063-9e6b-4805-a07f-ee96f0d437f8","Type":"ContainerDied","Data":"23c6f02a7a32b678e468cb3e3aa53d4bee0898d7bb4ddec8aebae84862c9612a"}
Oct 30 21:01:13.317438 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:13.317013    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23c6f02a7a32b678e468cb3e3aa53d4bee0898d7bb4ddec8aebae84862c9612a"
Oct 30 21:01:13.317589 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:13.317467    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e134f233-v8lrp"
Oct 30 21:01:13.497115 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:13.497087    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ce9871f-vgscg"
Oct 30 21:01:13.628155 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:13.627997    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/827ff375-c35d-4a58-8d30-16ad01bb0e08-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/827ff375-c35d-4a58-8d30-16ad01bb0e08-maas-api-xks-ca\") pod \"827ff375-c35d-4a58-8d30-16ad01bb0e08\" (UID: \"827ff375-c35d-4a58-8d30-16ad01bb0e08\") "
Oct 30 21:01:13.628155 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:13.628082    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/827ff375-c35d-4a58-8d30-16ad01bb0e08-kube-api-access-zkk2x\" (UniqueName: \"kubernetes.io/projected/827ff375-c35d-4a58-8d30-16ad01bb0e08-kube-api-access-zkk2x\") pod \"827ff375-c35d-4a58-8d30-16ad01bb0e08\" (UID: \"827ff375-c35d-4a58-8d30-16ad01bb0e08\") "
Oct 30 21:01:13.654386 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:13.628408    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/827ff375-c35d-4a58-8d30-16ad01bb0e08-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/827ff375-c35d-4a58-8d30-16ad01bb0e08-maas-api-openshift-ca\") pod \"827ff375-c35d-4a58-8d30-16ad01bb0e08\" (UID: \"827ff375-c35d-4a58-8d30-16ad01bb0e08\") "
Oct 30 21:01:13.654386 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:13.628929    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/827ff375-c35d-4a58-8d30-16ad01bb0e08-maas-api-openshift-ca" pod "827ff375-c35d-4a58-8d30-16ad01bb0e08" (UID: "827ff375-c35d-4a58-8d30-16ad01bb0e08"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:01:13.654386 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:13.629074    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/827ff375-c35d-4a58-8d30-16ad01bb0e08-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:13.654386 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:13.630528    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/827ff375-c35d-4a58-8d30-16ad01bb0e08-kube-api-access-zkk2x" pod "827ff375-c35d-4a58-8d30-16ad01bb0e08" (UID: "827ff375-c35d-4a58-8d30-16ad01bb0e08"). InnerVolumeSpecName "kube-api-access-zkk2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:01:13.654386 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:13.630769    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/827ff375-c35d-4a58-8d30-16ad01bb0e08-maas-api-xks-ca" pod "827ff375-c35d-4a58-8d30-16ad01bb0e08" (UID: "827ff375-c35d-4a58-8d30-16ad01bb0e08"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:01:13.730245 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:13.730213    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkk2x\" (UniqueName: \"kubernetes.io/projected/827ff375-c35d-4a58-8d30-16ad01bb0e08-kube-api-access-zkk2x\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:13.730245 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:13.730244    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/827ff375-c35d-4a58-8d30-16ad01bb0e08-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:01:14.327920 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:14.327889    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ce9871f-vgscg"
Oct 30 21:01:14.328329 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:14.327921    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ce9871f-vgscg" event={"ID":"827ff375-c35d-4a58-8d30-16ad01bb0e08","Type":"ContainerDied","Data":"28e848408cd09275963395714bffa3a9eb5e0c80e530de1a56b45fecdb544bb9"}
Oct 30 21:01:14.328329 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:14.327976    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28e848408cd09275963395714bffa3a9eb5e0c80e530de1a56b45fecdb544bb9"
Oct 30 21:01:17.737458 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:17.737239    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-682999/premium-w3-682999-kserve-5f47cf9b8d-xqnnr" podUID="a54cd33d-feef-42bb-a78b-7480634cb58d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:17.821825 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:17.820469    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-781058/premium-w0-781058-kserve-7f5c8964c9-z5fmk" podUID="57626f4f-844e-4ec9-ab89-359d015923af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 30 21:01:17.851819 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:17.851134    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-682999/distinct-2-w3-682999-kserve-7bc687885b-gq5x7" podUID="0dcf0422-081f-46d7-b246-7a49b074c88c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:17.929734 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:17.928896    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-781058/distinct-2-w0-781058-kserve-7f6c654757-ppmhm" podUID="5e818303-0e9f-4ba0-a635-b80b92c9e36d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:17.930059 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:17.930027    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-682999/simulator-w3-682999-kserve-57cfcb6f8f-6mgrg" podUID="53296e7f-c6bd-45f2-ac31-004feddc63a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:18.009003 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:18.008896    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b425d/simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk" podUID="e8108824-03ff-4c1d-9f5b-07be3a3a92d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:18.045763 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:18.045388    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-682999/distinct-w3-682999-kserve-74d49d4955-nndlc" podUID="2a35314b-31ea-42e6-b49a-15ab1305b338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Oct 30 21:01:18.086211 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:18.083187    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b425d/premium-w5-2b425d-kserve-78756b6c5b-9x5xb" podUID="abf7b512-50e5-4400-9ca3-9ffccaead3d7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:18.086211 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:18.085034    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7" podUID="b2193213-ebae-498a-911a-6056a8e94312" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded"
Oct 30 21:01:18.086211 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:18.085309    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm" podUID="cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Oct 30 21:01:18.131734 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:18.130912    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l" podUID="fc9510c5-2180-41d6-90bc-9acdff0f3616" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded"
Oct 30 21:01:18.266877 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:18.239505    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f9f277/premium-w1-f9f277-kserve-5dbfd4d496-72w4n" podUID="1cc1f811-c149-4457-80f1-1a55621d73b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:18.351068 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:18.350952    5713 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-682999/simulator-w3-682999-kserve-57cfcb6f8f-6mgrg" podUID="53296e7f-c6bd-45f2-ac31-004feddc63a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:18.867628 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:18.867140    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-781058/premium-w0-781058-kserve-7f5c8964c9-z5fmk" podUID="57626f4f-844e-4ec9-ab89-359d015923af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:18.868228 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:18.868150    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-781058/premium-w0-781058-kserve-7f5c8964c9-z5fmk"
Oct 30 21:01:18.932807 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:18.929720    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-781058/distinct-2-w0-781058-kserve-7f6c654757-ppmhm" podUID="5e818303-0e9f-4ba0-a635-b80b92c9e36d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:01:18.932807 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:18.930765    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-781058/distinct-2-w0-781058-kserve-7f6c654757-ppmhm"
Oct 30 21:01:18.948713 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:18.948673    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-781058/premium-w0-781058-kserve-7f5c8964c9-z5fmk"
Oct 30 21:01:18.954943 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:18.954912    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-781058/distinct-2-w0-781058-kserve-7f6c654757-ppmhm"
Oct 30 21:01:22.896362 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:22.896320    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj"]
Oct 30 21:01:22.896998 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:22.896983    5713 cpu_manager.go:414] "containerMap: removing container" podUID="be5f6063-9e6b-4805-a07f-ee96f0d437f8" containerName="revoke-keys"
Oct 30 21:01:22.897052 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:22.896999    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be5f6063-9e6b-4805-a07f-ee96f0d437f8" containerName="revoke-keys"
Oct 30 21:01:22.897052 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:22.897025    5713 cpu_manager.go:414] "containerMap: removing container" podUID="827ff375-c35d-4a58-8d30-16ad01bb0e08" containerName="revoke-keys"
Oct 30 21:01:22.897052 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:22.897030    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="827ff375-c35d-4a58-8d30-16ad01bb0e08" containerName="revoke-keys"
Oct 30 21:01:22.897155 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:22.897146    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="827ff375-c35d-4a58-8d30-16ad01bb0e08" containerName="revoke-keys"
Oct 30 21:01:22.897191 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:22.897164    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="be5f6063-9e6b-4805-a07f-ee96f0d437f8" containerName="revoke-keys"
Oct 30 21:01:22.901077 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:22.901058    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj"
Oct 30 21:01:22.935467 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:22.935435    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59246922-40fa-40ad-a84b-7604b077ef92-istio-envoy\") pod \"e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj\" (UID: \"59246922-40fa-40ad-a84b-7604b077ef92\") " pod="openshift-ingress/e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj"
Oct 30 21:01:22.935842 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:22.935784    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59246922-40fa-40ad-a84b-7604b077ef92-credential-socket\") pod \"e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj\" (UID: \"59246922-40fa-40ad-a84b-7604b077ef92\") " pod="openshift-ingress/e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj"
Oct 30 21:01:22.936042 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:22.936028    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59246922-40fa-40ad-a84b-7604b077ef92-workload-socket\") pod \"e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj\" (UID: \"59246922-40fa-40ad-a84b-7604b077ef92\") " pod="openshift-ingress/e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj"
Oct 30 21:01:22.936265 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:22.936251    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59246922-40fa-40ad-a84b-7604b077ef92-istiod-ca-cert\") pod \"e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj\" (UID: \"59246922-40fa-40ad-a84b-7604b077ef92\") " pod="openshift-ingress/e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj"
Oct 30 21:01:22.936408 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:22.936394    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59246922-40fa-40ad-a84b-7604b077ef92-istio-podinfo\") pod \"e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj\" (UID: \"59246922-40fa-40ad-a84b-7604b077ef92\") " pod="openshift-ingress/e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj"
Oct 30 21:01:22.936671 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:22.936651    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8bgd\" (UniqueName: \"kubernetes.io/projected/59246922-40fa-40ad-a84b-7604b077ef92-kube-api-access-j8bgd\") pod \"e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj\" (UID: \"59246922-40fa-40ad-a84b-7604b077ef92\") " pod="openshift-ingress/e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj"
Oct 30 21:01:22.936856 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:22.936841    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59246922-40fa-40ad-a84b-7604b077ef92-istio-data\") pod \"e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj\" (UID: \"59246922-40fa-40ad-a84b-7604b077ef92\") " pod="openshift-ingress/e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj"
Oct 30 21:01:22.936993 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:22.936980    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59246922-40fa-40ad-a84b-7604b077ef92-istio-token\") pod \"e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj\" (UID: \"59246922-40fa-40ad-a84b-7604b077ef92\") " pod="openshift-ingress/e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj"
Oct 30 21:01:22.937443 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:22.937423    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59246922-40fa-40ad-a84b-7604b077ef92-workload-certs\") pod \"e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj\" (UID: \"59246922-40fa-40ad-a84b-7604b077ef92\") " pod="openshift-ingress/e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj"
Oct 30 21:01:23.083817 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:23.083220    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-bc88b34e-openshift-default-dockercfg-lnjtc\""
Oct 30 21:01:23.099836 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:23.099742    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59246922-40fa-40ad-a84b-7604b077ef92-workload-certs\") pod \"e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj\" (UID: \"59246922-40fa-40ad-a84b-7604b077ef92\") " pod="openshift-ingress/e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj"
Oct 30 21:01:23.100035 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:23.099870    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59246922-40fa-40ad-a84b-7604b077ef92-istio-envoy\") pod \"e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj\" (UID: \"59246922-40fa-40ad-a84b-7604b077ef92\") " pod="openshift-ingress/e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj"
Oct 30 21:01:23.100035 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:23.099910    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59246922-40fa-40ad-a84b-7604b077ef92-credential-socket\") pod \"e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj\" (UID: \"59246922-40fa-40ad-a84b-7604b077ef92\") " pod="openshift-ingress/e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj"
Oct 30 21:01:23.100035 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:23.099948    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59246922-40fa-40ad-a84b-7604b077ef92-workload-socket\") pod \"e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj\" (UID: \"59246922-40fa-40ad-a84b-7604b077ef92\") " pod="openshift-ingress/e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj"
Oct 30 21:01:23.100035 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:23.100030    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59246922-40fa-40ad-a84b-7604b077ef92-istiod-ca-cert\") pod \"e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj\" (UID: \"59246922-40fa-40ad-a84b-7604b077ef92\") " pod="openshift-ingress/e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj"
Oct 30 21:01:23.100285 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:23.100058    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59246922-40fa-40ad-a84b-7604b077ef92-istio-podinfo\") pod \"e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj\" (UID: \"59246922-40fa-40ad-a84b-7604b077ef92\") " pod="openshift-ingress/e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj"
Oct 30 21:01:23.100285 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:23.100193    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8bgd\" (UniqueName: \"kubernetes.io/projected/59246922-40fa-40ad-a84b-7604b077ef92-kube-api-access-j8bgd\") pod \"e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj\" (UID: \"59246922-40fa-40ad-a84b-7604b077ef92\") " pod="openshift-ingress/e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj"
Oct 30 21:01:23.100285 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:23.100223    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59246922-40fa-40ad-a84b-7604b077ef92-istio-data\") pod \"e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj\" (UID: \"59246922-40fa-40ad-a84b-7604b077ef92\") " pod="openshift-ingress/e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj"
Oct 30 21:01:23.100285 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:23.100245    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59246922-40fa-40ad-a84b-7604b077ef92-istio-token\") pod \"e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj\" (UID: \"59246922-40fa-40ad-a84b-7604b077ef92\") " pod="openshift-ingress/e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj"
Oct 30 21:01:23.101285 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:23.100967    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59246922-40fa-40ad-a84b-7604b077ef92-workload-certs\") pod \"e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj\" (UID: \"59246922-40fa-40ad-a84b-7604b077ef92\") " pod="openshift-ingress/e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj"
Oct 30 21:01:23.105196 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:23.103731    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59246922-40fa-40ad-a84b-7604b077ef92-istio-envoy\") pod \"e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj\" (UID: \"59246922-40fa-40ad-a84b-7604b077ef92\") " pod="openshift-ingress/e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj"
Oct 30 21:01:23.105196 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:23.104019    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59246922-40fa-40ad-a84b-7604b077ef92-credential-socket\") pod \"e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj\" (UID: \"59246922-40fa-40ad-a84b-7604b077ef92\") " pod="openshift-ingress/e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj"
Oct 30 21:01:23.105196 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:23.104263    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59246922-40fa-40ad-a84b-7604b077ef92-workload-socket\") pod \"e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj\" (UID: \"59246922-40fa-40ad-a84b-7604b077ef92\") " pod="openshift-ingress/e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj"
Oct 30 21:01:23.105196 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:23.104863    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59246922-40fa-40ad-a84b-7604b077ef92-istiod-ca-cert\") pod \"e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj\" (UID: \"59246922-40fa-40ad-a84b-7604b077ef92\") " pod="openshift-ingress/e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj"
Oct 30 21:01:23.129619 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:23.129586    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59246922-40fa-40ad-a84b-7604b077ef92-istio-data\") pod \"e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj\" (UID: \"59246922-40fa-40ad-a84b-7604b077ef92\") " pod="openshift-ingress/e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj"
Oct 30 21:01:23.150606 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:23.150519    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59246922-40fa-40ad-a84b-7604b077ef92-istio-podinfo\") pod \"e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj\" (UID: \"59246922-40fa-40ad-a84b-7604b077ef92\") " pod="openshift-ingress/e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj"
Oct 30 21:01:23.152083 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:23.152056    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj"]
Oct 30 21:01:23.157561 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:23.157536    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59246922-40fa-40ad-a84b-7604b077ef92-istio-token\") pod \"e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj\" (UID: \"59246922-40fa-40ad-a84b-7604b077ef92\") " pod="openshift-ingress/e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj"
Oct 30 21:01:23.170132 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:23.170100    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8bgd\" (UniqueName: \"kubernetes.io/projected/59246922-40fa-40ad-a84b-7604b077ef92-kube-api-access-j8bgd\") pod \"e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj\" (UID: \"59246922-40fa-40ad-a84b-7604b077ef92\") " pod="openshift-ingress/e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj"
Oct 30 21:01:23.217679 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:23.217646    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj"
Oct 30 21:01:23.320906 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:23.320701    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-83851760-6b5fd6887d-hcdv2"
Oct 30 21:01:23.669316 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:23.669292    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj"]
Oct 30 21:01:23.669423 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:01:23.669378    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59246922_40fa_40ad_a84b_7604b077ef92.slice/crio-15fe8edaee7b5e776ca9a6fdaff4a28806c21ee80902259d9b858a2ceb1bdf62 WatchSource:0}: Error finding container 15fe8edaee7b5e776ca9a6fdaff4a28806c21ee80902259d9b858a2ceb1bdf62: Status 404 returned error can't find the container with id 15fe8edaee7b5e776ca9a6fdaff4a28806c21ee80902259d9b858a2ceb1bdf62
Oct 30 21:01:23.672080 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:23.672051    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:01:23.672146 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:23.672126    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:01:23.672179 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:23.672155    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:01:24.460294 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:01:24.460250    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:01:24.641349 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:24.641317    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj" event={"ID":"59246922-40fa-40ad-a84b-7604b077ef92","Type":"ContainerStarted","Data":"cb1db293e5e6f79d85e87828737f890a585cef853602139c637a94e501426ffa"}
Oct 30 21:01:24.641566 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:24.641376    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj" event={"ID":"59246922-40fa-40ad-a84b-7604b077ef92","Type":"ContainerStarted","Data":"15fe8edaee7b5e776ca9a6fdaff4a28806c21ee80902259d9b858a2ceb1bdf62"}
Oct 30 21:01:24.766325 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:24.766216    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj" podStartSLOduration=2.766203038 podStartE2EDuration="2.766203038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:01:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:24.762088926 +0000 UTC m=+1367.826865348" watchObservedRunningTime="2026-10-30 21:01:24.766203038 +0000 UTC m=+1367.830979436"
Oct 30 21:01:25.218628 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:25.218587    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj"
Oct 30 21:01:25.224633 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:25.224607    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj"
Oct 30 21:01:25.650172 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:25.650093    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj"
Oct 30 21:01:25.651158 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:25.651140    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj"
Oct 30 21:01:30.511098 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:30.511064    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bc88b34e-b49b84d8c-hzzvc"]
Oct 30 21:01:30.515675 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:30.515656    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-bc88b34e-b49b84d8c-hzzvc"
Oct 30 21:01:30.572167 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:30.572136    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bc88b34e-b49b84d8c-hzzvc"]
Oct 30 21:01:30.636846 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:30.617478    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j7ng\" (UniqueName: \"kubernetes.io/projected/bb318efb-e29b-49fa-98b0-7ecf8cb93531-kube-api-access-6j7ng\") pod \"payload-pre-processing-e2e-ait-bc88b34e-b49b84d8c-hzzvc\" (UID: \"bb318efb-e29b-49fa-98b0-7ecf8cb93531\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bc88b34e-b49b84d8c-hzzvc"
Oct 30 21:01:30.636846 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:30.619019    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb318efb-e29b-49fa-98b0-7ecf8cb93531-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-bc88b34e-b49b84d8c-hzzvc\" (UID: \"bb318efb-e29b-49fa-98b0-7ecf8cb93531\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bc88b34e-b49b84d8c-hzzvc"
Oct 30 21:01:30.636846 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:30.624399    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-bc88b34e\""
Oct 30 21:01:30.636846 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:30.624618    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-bc88b34e-dockercfg-r5mtf\""
Oct 30 21:01:30.724015 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:30.722761    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j7ng\" (UniqueName: \"kubernetes.io/projected/bb318efb-e29b-49fa-98b0-7ecf8cb93531-kube-api-access-6j7ng\") pod \"payload-pre-processing-e2e-ait-bc88b34e-b49b84d8c-hzzvc\" (UID: \"bb318efb-e29b-49fa-98b0-7ecf8cb93531\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bc88b34e-b49b84d8c-hzzvc"
Oct 30 21:01:30.724015 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:30.723190    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb318efb-e29b-49fa-98b0-7ecf8cb93531-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-bc88b34e-b49b84d8c-hzzvc\" (UID: \"bb318efb-e29b-49fa-98b0-7ecf8cb93531\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bc88b34e-b49b84d8c-hzzvc"
Oct 30 21:01:30.756394 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:30.750337    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb318efb-e29b-49fa-98b0-7ecf8cb93531-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-bc88b34e-b49b84d8c-hzzvc\" (UID: \"bb318efb-e29b-49fa-98b0-7ecf8cb93531\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bc88b34e-b49b84d8c-hzzvc"
Oct 30 21:01:30.793873 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:30.793754    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j7ng\" (UniqueName: \"kubernetes.io/projected/bb318efb-e29b-49fa-98b0-7ecf8cb93531-kube-api-access-6j7ng\") pod \"payload-pre-processing-e2e-ait-bc88b34e-b49b84d8c-hzzvc\" (UID: \"bb318efb-e29b-49fa-98b0-7ecf8cb93531\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-bc88b34e-b49b84d8c-hzzvc"
Oct 30 21:01:30.831099 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:30.831059    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-bc88b34e-b49b84d8c-hzzvc"
Oct 30 21:01:31.230114 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:01:31.230081    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb318efb_e29b_49fa_98b0_7ecf8cb93531.slice/crio-33214b3c94dd3df76516b9c270c5ff61d19e1d00c4906268040b3d20c23e6b4c WatchSource:0}: Error finding container 33214b3c94dd3df76516b9c270c5ff61d19e1d00c4906268040b3d20c23e6b4c: Status 404 returned error can't find the container with id 33214b3c94dd3df76516b9c270c5ff61d19e1d00c4906268040b3d20c23e6b4c
Oct 30 21:01:31.231670 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:31.231640    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bc88b34e-b49b84d8c-hzzvc"]
Oct 30 21:01:31.775316 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:31.775277    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-bc88b34e-b49b84d8c-hzzvc" event={"ID":"bb318efb-e29b-49fa-98b0-7ecf8cb93531","Type":"ContainerStarted","Data":"58baed66ce07af43ed601a86e63d65d2dd91c6b67c2ea3e4abde3a2aeb7fab66"}
Oct 30 21:01:31.775729 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:31.775343    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-bc88b34e-b49b84d8c-hzzvc" event={"ID":"bb318efb-e29b-49fa-98b0-7ecf8cb93531","Type":"ContainerStarted","Data":"33214b3c94dd3df76516b9c270c5ff61d19e1d00c4906268040b3d20c23e6b4c"}
Oct 30 21:01:31.775729 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:31.775373    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-bc88b34e-b49b84d8c-hzzvc"
Oct 30 21:01:31.893691 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:31.893626    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-bc88b34e-b49b84d8c-hzzvc" podStartSLOduration=1.8936076229999999 podStartE2EDuration="1.893607623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:01:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:31.890180871 +0000 UTC m=+1374.954957269" watchObservedRunningTime="2026-10-30 21:01:31.893607623 +0000 UTC m=+1374.958384022"
Oct 30 21:01:37.425993 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:37.425962    5713 scope.go:122] "RemoveContainer" containerID="c24b023506187f7d2b23eedd5545c0e60acf4b338ef952f2a198f86fdcf5bff8"
Oct 30 21:01:37.437883 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:37.437860    5713 scope.go:122] "RemoveContainer" containerID="f8344bb7dc6199a55a12b9e777476a9bbc8b93dc5cf98e356493dfb0a73ec515"
Oct 30 21:01:37.448059 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:37.448037    5713 scope.go:122] "RemoveContainer" containerID="8fa00e8c613e38bc5d1770513f6157d96fd4f9021d1fae57a7249a67307190cc"
Oct 30 21:01:37.458164 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:37.458139    5713 scope.go:122] "RemoveContainer" containerID="747d94b4c6d169972205644b026e09606889201ec06c84c719d2165959dce273"
Oct 30 21:01:37.470075 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:37.470057    5713 scope.go:122] "RemoveContainer" containerID="e8ca99fafd4d8b072fde632768f0eee6c06dfaf96e7070ec5b1993531976c777"
Oct 30 21:01:37.481498 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:37.481475    5713 scope.go:122] "RemoveContainer" containerID="b59fa4862beb1d44536ef29572ec10aeaaa7c5a99a711fd4d3cc5df27f5d60a0"
Oct 30 21:01:37.491996 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:37.491984    5713 scope.go:122] "RemoveContainer" containerID="e220454cb4a55cfb39f5123ba87c80aba4f903ca022aac0c5c79257f4ff30fdc"
Oct 30 21:01:42.787344 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:42.787314    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-bc88b34e-b49b84d8c-hzzvc"
Oct 30 21:01:43.083567 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:43.083461    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-pmsc4"]
Oct 30 21:01:43.089634 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:43.089607    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-pmsc4"
Oct 30 21:01:43.144080 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:43.144048    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-qbqvq\""
Oct 30 21:01:43.186131 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:43.186095    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-pmsc4"]
Oct 30 21:01:43.229305 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:43.229270    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77cf9324-4316-4631-9d6f-cf42ff54290f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-pmsc4\" (UID: \"77cf9324-4316-4631-9d6f-cf42ff54290f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-pmsc4"
Oct 30 21:01:43.229513 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:43.229329    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77cf9324-4316-4631-9d6f-cf42ff54290f-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-pmsc4\" (UID: \"77cf9324-4316-4631-9d6f-cf42ff54290f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-pmsc4"
Oct 30 21:01:43.229513 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:43.229466    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxkhc\" (UniqueName: \"kubernetes.io/projected/77cf9324-4316-4631-9d6f-cf42ff54290f-kube-api-access-wxkhc\") pod \"maas-default-gateway-openshift-default-769df7846c-pmsc4\" (UID: \"77cf9324-4316-4631-9d6f-cf42ff54290f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-pmsc4"
Oct 30 21:01:43.229633 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:43.229525    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77cf9324-4316-4631-9d6f-cf42ff54290f-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-pmsc4\" (UID: \"77cf9324-4316-4631-9d6f-cf42ff54290f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-pmsc4"
Oct 30 21:01:43.229689 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:43.229635    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77cf9324-4316-4631-9d6f-cf42ff54290f-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-pmsc4\" (UID: \"77cf9324-4316-4631-9d6f-cf42ff54290f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-pmsc4"
Oct 30 21:01:43.229738 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:43.229702    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77cf9324-4316-4631-9d6f-cf42ff54290f-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-pmsc4\" (UID: \"77cf9324-4316-4631-9d6f-cf42ff54290f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-pmsc4"
Oct 30 21:01:43.229934 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:43.229904    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77cf9324-4316-4631-9d6f-cf42ff54290f-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-pmsc4\" (UID: \"77cf9324-4316-4631-9d6f-cf42ff54290f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-pmsc4"
Oct 30 21:01:43.230010 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:43.229994    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77cf9324-4316-4631-9d6f-cf42ff54290f-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-pmsc4\" (UID: \"77cf9324-4316-4631-9d6f-cf42ff54290f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-pmsc4"
Oct 30 21:01:43.230075 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:43.230060    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77cf9324-4316-4631-9d6f-cf42ff54290f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-pmsc4\" (UID: \"77cf9324-4316-4631-9d6f-cf42ff54290f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-pmsc4"
Oct 30 21:01:43.331143 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:43.331106    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77cf9324-4316-4631-9d6f-cf42ff54290f-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-pmsc4\" (UID: \"77cf9324-4316-4631-9d6f-cf42ff54290f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-pmsc4"
Oct 30 21:01:43.331316 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:43.331178    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77cf9324-4316-4631-9d6f-cf42ff54290f-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-pmsc4\" (UID: \"77cf9324-4316-4631-9d6f-cf42ff54290f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-pmsc4"
Oct 30 21:01:43.331316 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:43.331293    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77cf9324-4316-4631-9d6f-cf42ff54290f-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-pmsc4\" (UID: \"77cf9324-4316-4631-9d6f-cf42ff54290f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-pmsc4"
Oct 30 21:01:43.331452 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:43.331341    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77cf9324-4316-4631-9d6f-cf42ff54290f-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-pmsc4\" (UID: \"77cf9324-4316-4631-9d6f-cf42ff54290f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-pmsc4"
Oct 30 21:01:43.331452 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:43.331377    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77cf9324-4316-4631-9d6f-cf42ff54290f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-pmsc4\" (UID: \"77cf9324-4316-4631-9d6f-cf42ff54290f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-pmsc4"
Oct 30 21:01:43.331567 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:43.331471    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77cf9324-4316-4631-9d6f-cf42ff54290f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-pmsc4\" (UID: \"77cf9324-4316-4631-9d6f-cf42ff54290f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-pmsc4"
Oct 30 21:01:43.331567 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:43.331511    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77cf9324-4316-4631-9d6f-cf42ff54290f-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-pmsc4\" (UID: \"77cf9324-4316-4631-9d6f-cf42ff54290f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-pmsc4"
Oct 30 21:01:43.331567 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:43.331555    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxkhc\" (UniqueName: \"kubernetes.io/projected/77cf9324-4316-4631-9d6f-cf42ff54290f-kube-api-access-wxkhc\") pod \"maas-default-gateway-openshift-default-769df7846c-pmsc4\" (UID: \"77cf9324-4316-4631-9d6f-cf42ff54290f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-pmsc4"
Oct 30 21:01:43.331720 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:43.331600    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77cf9324-4316-4631-9d6f-cf42ff54290f-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-pmsc4\" (UID: \"77cf9324-4316-4631-9d6f-cf42ff54290f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-pmsc4"
Oct 30 21:01:43.331720 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:43.331624    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77cf9324-4316-4631-9d6f-cf42ff54290f-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-pmsc4\" (UID: \"77cf9324-4316-4631-9d6f-cf42ff54290f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-pmsc4"
Oct 30 21:01:43.331947 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:43.331921    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77cf9324-4316-4631-9d6f-cf42ff54290f-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-pmsc4\" (UID: \"77cf9324-4316-4631-9d6f-cf42ff54290f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-pmsc4"
Oct 30 21:01:43.332038 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:43.332022    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77cf9324-4316-4631-9d6f-cf42ff54290f-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-pmsc4\" (UID: \"77cf9324-4316-4631-9d6f-cf42ff54290f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-pmsc4"
Oct 30 21:01:43.332271 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:43.332248    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77cf9324-4316-4631-9d6f-cf42ff54290f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-pmsc4\" (UID: \"77cf9324-4316-4631-9d6f-cf42ff54290f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-pmsc4"
Oct 30 21:01:43.332342 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:43.332272    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77cf9324-4316-4631-9d6f-cf42ff54290f-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-pmsc4\" (UID: \"77cf9324-4316-4631-9d6f-cf42ff54290f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-pmsc4"
Oct 30 21:01:43.334159 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:43.334100    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77cf9324-4316-4631-9d6f-cf42ff54290f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-pmsc4\" (UID: \"77cf9324-4316-4631-9d6f-cf42ff54290f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-pmsc4"
Oct 30 21:01:43.334268 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:43.334250    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77cf9324-4316-4631-9d6f-cf42ff54290f-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-pmsc4\" (UID: \"77cf9324-4316-4631-9d6f-cf42ff54290f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-pmsc4"
Oct 30 21:01:43.389023 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:43.388982    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77cf9324-4316-4631-9d6f-cf42ff54290f-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-pmsc4\" (UID: \"77cf9324-4316-4631-9d6f-cf42ff54290f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-pmsc4"
Oct 30 21:01:43.389199 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:43.389034    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxkhc\" (UniqueName: \"kubernetes.io/projected/77cf9324-4316-4631-9d6f-cf42ff54290f-kube-api-access-wxkhc\") pod \"maas-default-gateway-openshift-default-769df7846c-pmsc4\" (UID: \"77cf9324-4316-4631-9d6f-cf42ff54290f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-pmsc4"
Oct 30 21:01:43.403998 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:43.403969    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-pmsc4"
Oct 30 21:01:43.850998 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:01:43.850955    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77cf9324_4316_4631_9d6f_cf42ff54290f.slice/crio-d256cac7d0b31804a90d3b1b29c4bc11120bbf5f7eb4db32c20c8e548ad88ace WatchSource:0}: Error finding container d256cac7d0b31804a90d3b1b29c4bc11120bbf5f7eb4db32c20c8e548ad88ace: Status 404 returned error can't find the container with id d256cac7d0b31804a90d3b1b29c4bc11120bbf5f7eb4db32c20c8e548ad88ace
Oct 30 21:01:43.852044 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:43.852020    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-pmsc4"]
Oct 30 21:01:43.853860 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:43.853780    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:01:43.853932 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:43.853902    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:01:43.853987 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:43.853935    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:01:43.882234 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:43.882193    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-pmsc4" event={"ID":"77cf9324-4316-4631-9d6f-cf42ff54290f","Type":"ContainerStarted","Data":"d256cac7d0b31804a90d3b1b29c4bc11120bbf5f7eb4db32c20c8e548ad88ace"}
Oct 30 21:01:44.892290 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:44.892255    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-pmsc4" event={"ID":"77cf9324-4316-4631-9d6f-cf42ff54290f","Type":"ContainerStarted","Data":"a94c6ee1f9ec11ad37270aed763b7bc43d52fe923af5bf3a45ae1c63dc4d0eac"}
Oct 30 21:01:45.071832 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:45.067804    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-pmsc4" podStartSLOduration=2.067768336 podStartE2EDuration="2.067768336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:01:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:45.058358929 +0000 UTC m=+1388.123135339" watchObservedRunningTime="2026-10-30 21:01:45.067768336 +0000 UTC m=+1388.132544736"
Oct 30 21:01:45.405059 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:45.405023    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-pmsc4"
Oct 30 21:01:45.406858 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:45.406833    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-pmsc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Oct 30 21:01:45.406981 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:45.406884    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-pmsc4" podUID="77cf9324-4316-4631-9d6f-cf42ff54290f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Oct 30 21:01:46.404922 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:46.404882    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-pmsc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Oct 30 21:01:46.405320 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:46.404938    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-pmsc4" podUID="77cf9324-4316-4631-9d6f-cf42ff54290f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Oct 30 21:01:47.405311 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:47.405274    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-pmsc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Oct 30 21:01:47.405840 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:47.405341    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-pmsc4" podUID="77cf9324-4316-4631-9d6f-cf42ff54290f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Oct 30 21:01:48.404583 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:48.404553    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-pmsc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Oct 30 21:01:48.404773 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:48.404602    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-pmsc4" podUID="77cf9324-4316-4631-9d6f-cf42ff54290f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Oct 30 21:01:49.404953 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:49.404917    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-pmsc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Oct 30 21:01:49.405411 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:49.404972    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-pmsc4" podUID="77cf9324-4316-4631-9d6f-cf42ff54290f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Oct 30 21:01:50.405291 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:50.405260    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-pmsc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Oct 30 21:01:50.405707 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:50.405313    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-pmsc4" podUID="77cf9324-4316-4631-9d6f-cf42ff54290f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Oct 30 21:01:51.405274 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:51.405233    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-pmsc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Oct 30 21:01:51.405458 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:51.405316    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-pmsc4" podUID="77cf9324-4316-4631-9d6f-cf42ff54290f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Oct 30 21:01:52.436179 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:52.436144    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-pmsc4"
Oct 30 21:01:52.437093 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:52.437065    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-pmsc4"
Oct 30 21:01:52.437825 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:01:52.437807    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-pmsc4"
Oct 30 21:01:54.460238 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:01:54.460200    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:02:02.744916 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:02.744870    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-682999/premium-w3-682999-kserve-5f47cf9b8d-xqnnr" podUID="a54cd33d-feef-42bb-a78b-7480634cb58d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:02.815847 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:02.815808    5713 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-682999/premium-w3-682999-kserve-5f47cf9b8d-xqnnr" podUID="a54cd33d-feef-42bb-a78b-7480634cb58d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:03.009992 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:03.009895    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b425d/simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk" podUID="e8108824-03ff-4c1d-9f5b-07be3a3a92d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Oct 30 21:02:03.026917 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:03.026876    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-781058/simulator-w0-781058-kserve-55b954cb4b-4k7z6" podUID="92c43343-2820-4097-b829-9aae96b670de" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:03.148108 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:03.148058    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l" podUID="fc9510c5-2180-41d6-90bc-9acdff0f3616" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:03.178962 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:03.178912    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f9f277/premium-w1-f9f277-kserve-5dbfd4d496-72w4n" podUID="1cc1f811-c149-4457-80f1-1a55621d73b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:03.890905 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:03.890851    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-682999/premium-w3-682999-kserve-5f47cf9b8d-xqnnr" podUID="a54cd33d-feef-42bb-a78b-7480634cb58d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 30 21:02:03.897808 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:03.891684    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-682999/premium-w3-682999-kserve-5f47cf9b8d-xqnnr"
Oct 30 21:02:03.897808 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:03.893498    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-682999/distinct-2-w3-682999-kserve-7bc687885b-gq5x7" podUID="0dcf0422-081f-46d7-b246-7a49b074c88c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Oct 30 21:02:04.029015 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:04.028954    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-781058/simulator-w0-781058-kserve-55b954cb4b-4k7z6" podUID="92c43343-2820-4097-b829-9aae96b670de" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:04.029738 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:04.029701    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-781058/simulator-w0-781058-kserve-55b954cb4b-4k7z6"
Oct 30 21:02:04.159761 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:04.159656    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l" podUID="fc9510c5-2180-41d6-90bc-9acdff0f3616" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:02:04.160475 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:04.160450    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l"
Oct 30 21:02:04.207661 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:04.194736    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-682999/premium-w3-682999-kserve-5f47cf9b8d-xqnnr"
Oct 30 21:02:04.251746 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:04.251717    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-781058/simulator-w0-781058-kserve-55b954cb4b-4k7z6"
Oct 30 21:02:04.392131 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:04.392090    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l"
Oct 30 21:02:12.395209 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:12.395168    5713 patch_prober.go:29] interesting pod/e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": read tcp 10.134.0.2:47122->10.134.0.115:15021: read: connection reset by peer" start-of-body=
Oct 30 21:02:12.395745 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:12.395254    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb" podUID="d335bbaf-1e6b-49cb-8bb1-89d987d18a90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": read tcp 10.134.0.2:47122->10.134.0.115:15021: read: connection reset by peer"
Oct 30 21:02:13.444917 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:13.444881    5713 generic.go:356] "Generic (PLEG): container finished" podID="d335bbaf-1e6b-49cb-8bb1-89d987d18a90" containerID="a1f615299e1b3df5d35d745da95a15e5ee3e605e0fb9efb025bd729eb3163eab" exitCode=0
Oct 30 21:02:13.445428 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:13.444955    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb" event={"ID":"d335bbaf-1e6b-49cb-8bb1-89d987d18a90","Type":"ContainerDied","Data":"a1f615299e1b3df5d35d745da95a15e5ee3e605e0fb9efb025bd729eb3163eab"}
Oct 30 21:02:13.445537 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:13.445519    5713 scope.go:122] "RemoveContainer" containerID="a1f615299e1b3df5d35d745da95a15e5ee3e605e0fb9efb025bd729eb3163eab"
Oct 30 21:02:13.447231 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:13.447204    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:02:13.447316 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:13.447274    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:02:13.447316 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:13.447308    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:02:14.467856 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:14.467822    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb" event={"ID":"d335bbaf-1e6b-49cb-8bb1-89d987d18a90","Type":"ContainerStarted","Data":"962dbe672b45e3853f7af7675cd368c7a5eeff5fcd923839bbe76855a6814b80"}
Oct 30 21:02:14.548292 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:14.548255    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb"
Oct 30 21:02:14.548688 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:14.548595    5713 patch_prober.go:29] interesting pod/e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 30 21:02:14.548891 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:14.548867    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb" podUID="d335bbaf-1e6b-49cb-8bb1-89d987d18a90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 30 21:02:15.552817 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:15.547235    5713 patch_prober.go:29] interesting pod/e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 30 21:02:15.552817 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:15.547335    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb" podUID="d335bbaf-1e6b-49cb-8bb1-89d987d18a90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 30 21:02:16.557814 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:16.557231    5713 patch_prober.go:29] interesting pod/e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 30 21:02:16.557814 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:16.557314    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb" podUID="d335bbaf-1e6b-49cb-8bb1-89d987d18a90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 30 21:02:17.552816 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:17.550272    5713 patch_prober.go:29] interesting pod/e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 30 21:02:17.552816 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:17.550355    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb" podUID="d335bbaf-1e6b-49cb-8bb1-89d987d18a90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 30 21:02:18.554295 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:18.554249    5713 patch_prober.go:29] interesting pod/e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 30 21:02:18.555628 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:18.555593    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb" podUID="d335bbaf-1e6b-49cb-8bb1-89d987d18a90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 30 21:02:19.546775 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:19.546736    5713 patch_prober.go:29] interesting pod/e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 30 21:02:19.547000 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:19.546841    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb" podUID="d335bbaf-1e6b-49cb-8bb1-89d987d18a90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 30 21:02:20.550323 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:20.550296    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb"
Oct 30 21:02:20.550812 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:20.550777    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb"
Oct 30 21:02:20.551546 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:20.551526    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb"
Oct 30 21:02:24.460040 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:02:24.460004    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:02:28.444654 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:28.444619    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5hlrx"]
Oct 30 21:02:28.448147 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:28.448129    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5hlrx"
Oct 30 21:02:28.494169 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:28.494133    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6nvgb"]
Oct 30 21:02:28.497594 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:28.497570    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6nvgb"
Oct 30 21:02:28.529971 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:28.529939    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/454f552c-5306-44d0-8d5d-df9555872183-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-5hlrx\" (UID: \"454f552c-5306-44d0-8d5d-df9555872183\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5hlrx"
Oct 30 21:02:28.530128 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:28.529978    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/454f552c-5306-44d0-8d5d-df9555872183-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-5hlrx\" (UID: \"454f552c-5306-44d0-8d5d-df9555872183\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5hlrx"
Oct 30 21:02:28.530128 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:28.530064    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/454f552c-5306-44d0-8d5d-df9555872183-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-5hlrx\" (UID: \"454f552c-5306-44d0-8d5d-df9555872183\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5hlrx"
Oct 30 21:02:28.530199 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:28.530145    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a166cd-7a71-449c-9ec1-68950b90e678-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-6nvgb\" (UID: \"74a166cd-7a71-449c-9ec1-68950b90e678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6nvgb"
Oct 30 21:02:28.530199 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:28.530172    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74a166cd-7a71-449c-9ec1-68950b90e678-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-6nvgb\" (UID: \"74a166cd-7a71-449c-9ec1-68950b90e678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6nvgb"
Oct 30 21:02:28.530271 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:28.530206    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74a166cd-7a71-449c-9ec1-68950b90e678-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-6nvgb\" (UID: \"74a166cd-7a71-449c-9ec1-68950b90e678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6nvgb"
Oct 30 21:02:28.530271 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:28.530229    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/454f552c-5306-44d0-8d5d-df9555872183-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-5hlrx\" (UID: \"454f552c-5306-44d0-8d5d-df9555872183\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5hlrx"
Oct 30 21:02:28.530271 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:28.530249    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74a166cd-7a71-449c-9ec1-68950b90e678-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-6nvgb\" (UID: \"74a166cd-7a71-449c-9ec1-68950b90e678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6nvgb"
Oct 30 21:02:28.530381 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:28.530366    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/454f552c-5306-44d0-8d5d-df9555872183-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-5hlrx\" (UID: \"454f552c-5306-44d0-8d5d-df9555872183\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5hlrx"
Oct 30 21:02:28.530418 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:28.530409    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsrb7\" (UniqueName: \"kubernetes.io/projected/74a166cd-7a71-449c-9ec1-68950b90e678-kube-api-access-tsrb7\") pod \"maas-default-gateway-openshift-default-769df7846c-6nvgb\" (UID: \"74a166cd-7a71-449c-9ec1-68950b90e678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6nvgb"
Oct 30 21:02:28.530458 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:28.530442    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74a166cd-7a71-449c-9ec1-68950b90e678-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-6nvgb\" (UID: \"74a166cd-7a71-449c-9ec1-68950b90e678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6nvgb"
Oct 30 21:02:28.530502 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:28.530490    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/454f552c-5306-44d0-8d5d-df9555872183-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-5hlrx\" (UID: \"454f552c-5306-44d0-8d5d-df9555872183\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5hlrx"
Oct 30 21:02:28.530537 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:28.530516    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/454f552c-5306-44d0-8d5d-df9555872183-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-5hlrx\" (UID: \"454f552c-5306-44d0-8d5d-df9555872183\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5hlrx"
Oct 30 21:02:28.530575 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:28.530556    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74a166cd-7a71-449c-9ec1-68950b90e678-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-6nvgb\" (UID: \"74a166cd-7a71-449c-9ec1-68950b90e678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6nvgb"
Oct 30 21:02:28.530627 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:28.530609    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a166cd-7a71-449c-9ec1-68950b90e678-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-6nvgb\" (UID: \"74a166cd-7a71-449c-9ec1-68950b90e678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6nvgb"
Oct 30 21:02:28.530676 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:28.530663    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/454f552c-5306-44d0-8d5d-df9555872183-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-5hlrx\" (UID: \"454f552c-5306-44d0-8d5d-df9555872183\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5hlrx"
Oct 30 21:02:28.530761 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:28.530750    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s8pb\" (UniqueName: \"kubernetes.io/projected/454f552c-5306-44d0-8d5d-df9555872183-kube-api-access-8s8pb\") pod \"maas-default-gateway-openshift-default-769df7846c-5hlrx\" (UID: \"454f552c-5306-44d0-8d5d-df9555872183\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5hlrx"
Oct 30 21:02:28.530830 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:28.530774    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74a166cd-7a71-449c-9ec1-68950b90e678-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-6nvgb\" (UID: \"74a166cd-7a71-449c-9ec1-68950b90e678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6nvgb"
Oct 30 21:02:28.544844 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:28.544813    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5hlrx"]
Oct 30 21:02:28.597210 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:28.597173    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6nvgb"]
Oct 30 21:02:28.631843 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:28.631766    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/454f552c-5306-44d0-8d5d-df9555872183-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-5hlrx\" (UID: \"454f552c-5306-44d0-8d5d-df9555872183\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5hlrx"
Oct 30 21:02:28.632059 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:28.631892    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsrb7\" (UniqueName: \"kubernetes.io/projected/74a166cd-7a71-449c-9ec1-68950b90e678-kube-api-access-tsrb7\") pod \"maas-default-gateway-openshift-default-769df7846c-6nvgb\" (UID: \"74a166cd-7a71-449c-9ec1-68950b90e678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6nvgb"
Oct 30 21:02:28.632059 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:28.631932    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74a166cd-7a71-449c-9ec1-68950b90e678-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-6nvgb\" (UID: \"74a166cd-7a71-449c-9ec1-68950b90e678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6nvgb"
Oct 30 21:02:28.632059 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:28.631957    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/454f552c-5306-44d0-8d5d-df9555872183-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-5hlrx\" (UID: \"454f552c-5306-44d0-8d5d-df9555872183\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5hlrx"
Oct 30 21:02:28.632249 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:28.632105    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/454f552c-5306-44d0-8d5d-df9555872183-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-5hlrx\" (UID: \"454f552c-5306-44d0-8d5d-df9555872183\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5hlrx"
Oct 30 21:02:28.632249 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:28.632155    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74a166cd-7a71-449c-9ec1-68950b90e678-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-6nvgb\" (UID: \"74a166cd-7a71-449c-9ec1-68950b90e678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6nvgb"
Oct 30 21:02:28.632249 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:28.632189    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a166cd-7a71-449c-9ec1-68950b90e678-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-6nvgb\" (UID: \"74a166cd-7a71-449c-9ec1-68950b90e678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6nvgb"
Oct 30 21:02:28.632249 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:28.632229    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/454f552c-5306-44d0-8d5d-df9555872183-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-5hlrx\" (UID: \"454f552c-5306-44d0-8d5d-df9555872183\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5hlrx"
Oct 30 21:02:28.632249 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:28.632239    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/454f552c-5306-44d0-8d5d-df9555872183-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-5hlrx\" (UID: \"454f552c-5306-44d0-8d5d-df9555872183\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5hlrx"
Oct 30 21:02:28.632507 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:28.632249    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/454f552c-5306-44d0-8d5d-df9555872183-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-5hlrx\" (UID: \"454f552c-5306-44d0-8d5d-df9555872183\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5hlrx"
Oct 30 21:02:28.632507 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:28.632441    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s8pb\" (UniqueName: \"kubernetes.io/projected/454f552c-5306-44d0-8d5d-df9555872183-kube-api-access-8s8pb\") pod \"maas-default-gateway-openshift-default-769df7846c-5hlrx\" (UID: \"454f552c-5306-44d0-8d5d-df9555872183\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5hlrx"
Oct 30 21:02:28.632507 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:28.632479    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74a166cd-7a71-449c-9ec1-68950b90e678-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-6nvgb\" (UID: \"74a166cd-7a71-449c-9ec1-68950b90e678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6nvgb"
Oct 30 21:02:28.632685 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:28.632511    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/454f552c-5306-44d0-8d5d-df9555872183-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-5hlrx\" (UID: \"454f552c-5306-44d0-8d5d-df9555872183\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5hlrx"
Oct 30 21:02:28.632685 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:28.632614    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/454f552c-5306-44d0-8d5d-df9555872183-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-5hlrx\" (UID: \"454f552c-5306-44d0-8d5d-df9555872183\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5hlrx"
Oct 30 21:02:28.632685 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:28.632650    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/454f552c-5306-44d0-8d5d-df9555872183-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-5hlrx\" (UID: \"454f552c-5306-44d0-8d5d-df9555872183\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5hlrx"
Oct 30 21:02:28.632895 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:28.632697    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/454f552c-5306-44d0-8d5d-df9555872183-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-5hlrx\" (UID: \"454f552c-5306-44d0-8d5d-df9555872183\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5hlrx"
Oct 30 21:02:28.632895 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:28.632758    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74a166cd-7a71-449c-9ec1-68950b90e678-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-6nvgb\" (UID: \"74a166cd-7a71-449c-9ec1-68950b90e678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6nvgb"
Oct 30 21:02:28.632895 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:28.632762    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a166cd-7a71-449c-9ec1-68950b90e678-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-6nvgb\" (UID: \"74a166cd-7a71-449c-9ec1-68950b90e678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6nvgb"
Oct 30 21:02:28.632895 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:28.632764    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a166cd-7a71-449c-9ec1-68950b90e678-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-6nvgb\" (UID: \"74a166cd-7a71-449c-9ec1-68950b90e678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6nvgb"
Oct 30 21:02:28.632895 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:28.632850    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74a166cd-7a71-449c-9ec1-68950b90e678-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-6nvgb\" (UID: \"74a166cd-7a71-449c-9ec1-68950b90e678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6nvgb"
Oct 30 21:02:28.633138 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:28.632900    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74a166cd-7a71-449c-9ec1-68950b90e678-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-6nvgb\" (UID: \"74a166cd-7a71-449c-9ec1-68950b90e678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6nvgb"
Oct 30 21:02:28.633138 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:28.632944    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/454f552c-5306-44d0-8d5d-df9555872183-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-5hlrx\" (UID: \"454f552c-5306-44d0-8d5d-df9555872183\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5hlrx"
Oct 30 21:02:28.633138 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:28.632969    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74a166cd-7a71-449c-9ec1-68950b90e678-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-6nvgb\" (UID: \"74a166cd-7a71-449c-9ec1-68950b90e678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6nvgb"
Oct 30 21:02:28.633301 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:28.633251    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74a166cd-7a71-449c-9ec1-68950b90e678-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-6nvgb\" (UID: \"74a166cd-7a71-449c-9ec1-68950b90e678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6nvgb"
Oct 30 21:02:28.633301 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:28.633276    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/454f552c-5306-44d0-8d5d-df9555872183-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-5hlrx\" (UID: \"454f552c-5306-44d0-8d5d-df9555872183\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5hlrx"
Oct 30 21:02:28.633401 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:28.633324    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74a166cd-7a71-449c-9ec1-68950b90e678-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-6nvgb\" (UID: \"74a166cd-7a71-449c-9ec1-68950b90e678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6nvgb"
Oct 30 21:02:28.633660 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:28.633632    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/454f552c-5306-44d0-8d5d-df9555872183-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-5hlrx\" (UID: \"454f552c-5306-44d0-8d5d-df9555872183\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5hlrx"
Oct 30 21:02:28.633766 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:28.633725    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a166cd-7a71-449c-9ec1-68950b90e678-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-6nvgb\" (UID: \"74a166cd-7a71-449c-9ec1-68950b90e678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6nvgb"
Oct 30 21:02:28.637962 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:28.637937    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74a166cd-7a71-449c-9ec1-68950b90e678-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-6nvgb\" (UID: \"74a166cd-7a71-449c-9ec1-68950b90e678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6nvgb"
Oct 30 21:02:28.638385 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:28.638346    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/454f552c-5306-44d0-8d5d-df9555872183-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-5hlrx\" (UID: \"454f552c-5306-44d0-8d5d-df9555872183\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5hlrx"
Oct 30 21:02:28.638832 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:28.638773    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74a166cd-7a71-449c-9ec1-68950b90e678-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-6nvgb\" (UID: \"74a166cd-7a71-449c-9ec1-68950b90e678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6nvgb"
Oct 30 21:02:28.640206 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:28.640178    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/454f552c-5306-44d0-8d5d-df9555872183-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-5hlrx\" (UID: \"454f552c-5306-44d0-8d5d-df9555872183\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5hlrx"
Oct 30 21:02:28.695545 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:28.695456    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74a166cd-7a71-449c-9ec1-68950b90e678-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-6nvgb\" (UID: \"74a166cd-7a71-449c-9ec1-68950b90e678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6nvgb"
Oct 30 21:02:28.697908 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:28.697885    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsrb7\" (UniqueName: \"kubernetes.io/projected/74a166cd-7a71-449c-9ec1-68950b90e678-kube-api-access-tsrb7\") pod \"maas-default-gateway-openshift-default-769df7846c-6nvgb\" (UID: \"74a166cd-7a71-449c-9ec1-68950b90e678\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6nvgb"
Oct 30 21:02:28.698045 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:28.697931    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/454f552c-5306-44d0-8d5d-df9555872183-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-5hlrx\" (UID: \"454f552c-5306-44d0-8d5d-df9555872183\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5hlrx"
Oct 30 21:02:28.699045 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:28.699026    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s8pb\" (UniqueName: \"kubernetes.io/projected/454f552c-5306-44d0-8d5d-df9555872183-kube-api-access-8s8pb\") pod \"maas-default-gateway-openshift-default-769df7846c-5hlrx\" (UID: \"454f552c-5306-44d0-8d5d-df9555872183\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5hlrx"
Oct 30 21:02:28.761423 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:28.761385    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5hlrx"
Oct 30 21:02:28.811254 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:28.811202    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6nvgb"
Oct 30 21:02:29.209302 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:02:29.209268    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod454f552c_5306_44d0_8d5d_df9555872183.slice/crio-fedf1e6aa57b558e333b025c93d60ce263d124783b5ffb141614307b5d219ff5 WatchSource:0}: Error finding container fedf1e6aa57b558e333b025c93d60ce263d124783b5ffb141614307b5d219ff5: Status 404 returned error can't find the container with id fedf1e6aa57b558e333b025c93d60ce263d124783b5ffb141614307b5d219ff5
Oct 30 21:02:29.209875 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:29.209849    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5hlrx"]
Oct 30 21:02:29.211956 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:29.211923    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:02:29.212036 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:29.211990    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:02:29.212036 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:29.212023    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:02:29.266251 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:02:29.266211    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74a166cd_7a71_449c_9ec1_68950b90e678.slice/crio-6c80ca056eab2c6b1f1ef5f7e965500abc2a94e69e129d32a9f610f0cdaeb58b WatchSource:0}: Error finding container 6c80ca056eab2c6b1f1ef5f7e965500abc2a94e69e129d32a9f610f0cdaeb58b: Status 404 returned error can't find the container with id 6c80ca056eab2c6b1f1ef5f7e965500abc2a94e69e129d32a9f610f0cdaeb58b
Oct 30 21:02:29.267527 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:29.267497    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6nvgb"]
Oct 30 21:02:29.269130 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:29.269101    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:02:29.269194 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:29.269184    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:02:29.269237 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:29.269225    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:02:29.606289 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:29.606190    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6nvgb" event={"ID":"74a166cd-7a71-449c-9ec1-68950b90e678","Type":"ContainerStarted","Data":"ee2b358442f429882ecde2493a4d04ffe513f309b2d2bdf43fd7429a3965a5e4"}
Oct 30 21:02:29.606289 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:29.606276    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6nvgb" event={"ID":"74a166cd-7a71-449c-9ec1-68950b90e678","Type":"ContainerStarted","Data":"6c80ca056eab2c6b1f1ef5f7e965500abc2a94e69e129d32a9f610f0cdaeb58b"}
Oct 30 21:02:29.608404 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:29.608372    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5hlrx" event={"ID":"454f552c-5306-44d0-8d5d-df9555872183","Type":"ContainerStarted","Data":"953db0957b097a6cfee8cd7bee85597a6d1982031c8480cc7ce33894bf1d89cd"}
Oct 30 21:02:29.608549 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:29.608426    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5hlrx" event={"ID":"454f552c-5306-44d0-8d5d-df9555872183","Type":"ContainerStarted","Data":"fedf1e6aa57b558e333b025c93d60ce263d124783b5ffb141614307b5d219ff5"}
Oct 30 21:02:29.736001 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:29.735931    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6nvgb" podStartSLOduration=1.735911378 podStartE2EDuration="1.735911378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:02:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:02:29.727455431 +0000 UTC m=+1432.792231830" watchObservedRunningTime="2026-10-30 21:02:29.735911378 +0000 UTC m=+1432.800687776"
Oct 30 21:02:29.762474 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:29.762445    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5hlrx"
Oct 30 21:02:29.811651 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:29.811614    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6nvgb"
Oct 30 21:02:29.849548 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:29.849493    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5hlrx" podStartSLOduration=1.849478425 podStartE2EDuration="1.849478425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:02:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:02:29.843415765 +0000 UTC m=+1432.908192165" watchObservedRunningTime="2026-10-30 21:02:29.849478425 +0000 UTC m=+1432.914254823"
Oct 30 21:02:30.781814 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:30.779934    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-5hlrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body=
Oct 30 21:02:30.781814 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:30.780006    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5hlrx" podUID="454f552c-5306-44d0-8d5d-df9555872183" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused"
Oct 30 21:02:30.813465 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:30.813428    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-6nvgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body=
Oct 30 21:02:30.813645 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:30.813486    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6nvgb" podUID="74a166cd-7a71-449c-9ec1-68950b90e678" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused"
Oct 30 21:02:31.772813 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:31.772269    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-5hlrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body=
Oct 30 21:02:31.772813 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:31.772331    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5hlrx" podUID="454f552c-5306-44d0-8d5d-df9555872183" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused"
Oct 30 21:02:31.812501 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:31.812469    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-6nvgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body=
Oct 30 21:02:31.812931 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:31.812522    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6nvgb" podUID="74a166cd-7a71-449c-9ec1-68950b90e678" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused"
Oct 30 21:02:32.771813 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:32.765343    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-5hlrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body=
Oct 30 21:02:32.771813 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:32.765426    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5hlrx" podUID="454f552c-5306-44d0-8d5d-df9555872183" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused"
Oct 30 21:02:32.812815 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:32.812749    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-6nvgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body=
Oct 30 21:02:32.813208 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:32.812837    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6nvgb" podUID="74a166cd-7a71-449c-9ec1-68950b90e678" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused"
Oct 30 21:02:33.778815 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:33.765320    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-5hlrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body=
Oct 30 21:02:33.778815 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:33.765383    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5hlrx" podUID="454f552c-5306-44d0-8d5d-df9555872183" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused"
Oct 30 21:02:33.812740 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:33.812707    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-6nvgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body=
Oct 30 21:02:33.812925 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:33.812761    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6nvgb" podUID="74a166cd-7a71-449c-9ec1-68950b90e678" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused"
Oct 30 21:02:34.785815 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:34.765910    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-5hlrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body=
Oct 30 21:02:34.785815 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:34.765978    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5hlrx" podUID="454f552c-5306-44d0-8d5d-df9555872183" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused"
Oct 30 21:02:34.811796 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:34.811757    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-6nvgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body=
Oct 30 21:02:34.811963 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:34.811828    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6nvgb" podUID="74a166cd-7a71-449c-9ec1-68950b90e678" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused"
Oct 30 21:02:35.769827 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:35.762015    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-5hlrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body=
Oct 30 21:02:35.769827 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:35.762083    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5hlrx" podUID="454f552c-5306-44d0-8d5d-df9555872183" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused"
Oct 30 21:02:35.812779 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:35.812743    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-6nvgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body=
Oct 30 21:02:35.812986 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:35.812830    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6nvgb" podUID="74a166cd-7a71-449c-9ec1-68950b90e678" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused"
Oct 30 21:02:36.762065 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:36.762028    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-5hlrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body=
Oct 30 21:02:36.762274 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:36.762120    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5hlrx" podUID="454f552c-5306-44d0-8d5d-df9555872183" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused"
Oct 30 21:02:36.812682 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:36.812649    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-6nvgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body=
Oct 30 21:02:36.813108 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:36.812707    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6nvgb" podUID="74a166cd-7a71-449c-9ec1-68950b90e678" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused"
Oct 30 21:02:38.750949 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:38.750259    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6nvgb"
Oct 30 21:02:38.752872 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:38.751924    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6nvgb"
Oct 30 21:02:38.754708 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:38.753271    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5hlrx"
Oct 30 21:02:38.754708 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:38.754498    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5hlrx"
Oct 30 21:02:38.755296 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:38.755249    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6nvgb"
Oct 30 21:02:38.757250 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:38.757232    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5hlrx"
Oct 30 21:02:43.564375 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:43.564337    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7k46d"]
Oct 30 21:02:43.569363 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:43.569340    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7k46d"
Oct 30 21:02:43.690113 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:43.690074    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4242f534-4d88-40d1-a4e0-e45493a00d7a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-7k46d\" (UID: \"4242f534-4d88-40d1-a4e0-e45493a00d7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7k46d"
Oct 30 21:02:43.690329 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:43.690146    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4242f534-4d88-40d1-a4e0-e45493a00d7a-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-7k46d\" (UID: \"4242f534-4d88-40d1-a4e0-e45493a00d7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7k46d"
Oct 30 21:02:43.690329 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:43.690268    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4242f534-4d88-40d1-a4e0-e45493a00d7a-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-7k46d\" (UID: \"4242f534-4d88-40d1-a4e0-e45493a00d7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7k46d"
Oct 30 21:02:43.690329 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:43.690313    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4242f534-4d88-40d1-a4e0-e45493a00d7a-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-7k46d\" (UID: \"4242f534-4d88-40d1-a4e0-e45493a00d7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7k46d"
Oct 30 21:02:43.690499 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:43.690393    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4242f534-4d88-40d1-a4e0-e45493a00d7a-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-7k46d\" (UID: \"4242f534-4d88-40d1-a4e0-e45493a00d7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7k46d"
Oct 30 21:02:43.690499 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:43.690420    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4242f534-4d88-40d1-a4e0-e45493a00d7a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-7k46d\" (UID: \"4242f534-4d88-40d1-a4e0-e45493a00d7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7k46d"
Oct 30 21:02:43.690499 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:43.690482    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4242f534-4d88-40d1-a4e0-e45493a00d7a-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-7k46d\" (UID: \"4242f534-4d88-40d1-a4e0-e45493a00d7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7k46d"
Oct 30 21:02:43.690667 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:43.690543    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6sjv\" (UniqueName: \"kubernetes.io/projected/4242f534-4d88-40d1-a4e0-e45493a00d7a-kube-api-access-q6sjv\") pod \"maas-default-gateway-openshift-default-769df7846c-7k46d\" (UID: \"4242f534-4d88-40d1-a4e0-e45493a00d7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7k46d"
Oct 30 21:02:43.690667 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:43.690570    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4242f534-4d88-40d1-a4e0-e45493a00d7a-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-7k46d\" (UID: \"4242f534-4d88-40d1-a4e0-e45493a00d7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7k46d"
Oct 30 21:02:43.694737 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:43.694706    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7k46d"]
Oct 30 21:02:43.792865 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:43.792822    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4242f534-4d88-40d1-a4e0-e45493a00d7a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-7k46d\" (UID: \"4242f534-4d88-40d1-a4e0-e45493a00d7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7k46d"
Oct 30 21:02:43.793178 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:43.793136    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4242f534-4d88-40d1-a4e0-e45493a00d7a-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-7k46d\" (UID: \"4242f534-4d88-40d1-a4e0-e45493a00d7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7k46d"
Oct 30 21:02:43.793333 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:43.793257    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4242f534-4d88-40d1-a4e0-e45493a00d7a-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-7k46d\" (UID: \"4242f534-4d88-40d1-a4e0-e45493a00d7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7k46d"
Oct 30 21:02:43.793333 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:43.793301    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4242f534-4d88-40d1-a4e0-e45493a00d7a-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-7k46d\" (UID: \"4242f534-4d88-40d1-a4e0-e45493a00d7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7k46d"
Oct 30 21:02:43.793451 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:43.793356    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4242f534-4d88-40d1-a4e0-e45493a00d7a-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-7k46d\" (UID: \"4242f534-4d88-40d1-a4e0-e45493a00d7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7k46d"
Oct 30 21:02:43.793451 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:43.793383    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4242f534-4d88-40d1-a4e0-e45493a00d7a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-7k46d\" (UID: \"4242f534-4d88-40d1-a4e0-e45493a00d7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7k46d"
Oct 30 21:02:43.793570 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:43.793459    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4242f534-4d88-40d1-a4e0-e45493a00d7a-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-7k46d\" (UID: \"4242f534-4d88-40d1-a4e0-e45493a00d7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7k46d"
Oct 30 21:02:43.793570 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:43.793526    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6sjv\" (UniqueName: \"kubernetes.io/projected/4242f534-4d88-40d1-a4e0-e45493a00d7a-kube-api-access-q6sjv\") pod \"maas-default-gateway-openshift-default-769df7846c-7k46d\" (UID: \"4242f534-4d88-40d1-a4e0-e45493a00d7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7k46d"
Oct 30 21:02:43.793570 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:43.793561    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4242f534-4d88-40d1-a4e0-e45493a00d7a-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-7k46d\" (UID: \"4242f534-4d88-40d1-a4e0-e45493a00d7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7k46d"
Oct 30 21:02:43.793760 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:43.793735    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4242f534-4d88-40d1-a4e0-e45493a00d7a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-7k46d\" (UID: \"4242f534-4d88-40d1-a4e0-e45493a00d7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7k46d"
Oct 30 21:02:43.794426 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:43.794394    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4242f534-4d88-40d1-a4e0-e45493a00d7a-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-7k46d\" (UID: \"4242f534-4d88-40d1-a4e0-e45493a00d7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7k46d"
Oct 30 21:02:43.794566 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:43.794543    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4242f534-4d88-40d1-a4e0-e45493a00d7a-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-7k46d\" (UID: \"4242f534-4d88-40d1-a4e0-e45493a00d7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7k46d"
Oct 30 21:02:43.795886 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:43.795832    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4242f534-4d88-40d1-a4e0-e45493a00d7a-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-7k46d\" (UID: \"4242f534-4d88-40d1-a4e0-e45493a00d7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7k46d"
Oct 30 21:02:43.799052 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:43.796041    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4242f534-4d88-40d1-a4e0-e45493a00d7a-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-7k46d\" (UID: \"4242f534-4d88-40d1-a4e0-e45493a00d7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7k46d"
Oct 30 21:02:43.799052 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:43.797643    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4242f534-4d88-40d1-a4e0-e45493a00d7a-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-7k46d\" (UID: \"4242f534-4d88-40d1-a4e0-e45493a00d7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7k46d"
Oct 30 21:02:43.806317 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:43.806293    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4242f534-4d88-40d1-a4e0-e45493a00d7a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-7k46d\" (UID: \"4242f534-4d88-40d1-a4e0-e45493a00d7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7k46d"
Oct 30 21:02:43.852560 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:43.852495    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4242f534-4d88-40d1-a4e0-e45493a00d7a-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-7k46d\" (UID: \"4242f534-4d88-40d1-a4e0-e45493a00d7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7k46d"
Oct 30 21:02:43.852560 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:43.852544    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6sjv\" (UniqueName: \"kubernetes.io/projected/4242f534-4d88-40d1-a4e0-e45493a00d7a-kube-api-access-q6sjv\") pod \"maas-default-gateway-openshift-default-769df7846c-7k46d\" (UID: \"4242f534-4d88-40d1-a4e0-e45493a00d7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7k46d"
Oct 30 21:02:43.884240 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:43.884213    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7k46d"
Oct 30 21:02:44.319509 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:02:44.319468    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4242f534_4d88_40d1_a4e0_e45493a00d7a.slice/crio-db4cf48a8463d20dab36b103979e3763f51e44ac944767c0c2ac7aa88b418874 WatchSource:0}: Error finding container db4cf48a8463d20dab36b103979e3763f51e44ac944767c0c2ac7aa88b418874: Status 404 returned error can't find the container with id db4cf48a8463d20dab36b103979e3763f51e44ac944767c0c2ac7aa88b418874
Oct 30 21:02:44.321236 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:44.321178    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7k46d"]
Oct 30 21:02:44.322367 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:44.322331    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:02:44.322438 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:44.322419    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:02:44.322474 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:44.322460    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:02:44.821098 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:44.821062    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7k46d" event={"ID":"4242f534-4d88-40d1-a4e0-e45493a00d7a","Type":"ContainerStarted","Data":"940268c02595e8de006da84aabe30fe4c50150ccc152264561e4d80fbb6b2bca"}
Oct 30 21:02:44.821502 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:44.821119    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7k46d" event={"ID":"4242f534-4d88-40d1-a4e0-e45493a00d7a","Type":"ContainerStarted","Data":"db4cf48a8463d20dab36b103979e3763f51e44ac944767c0c2ac7aa88b418874"}
Oct 30 21:02:44.885353 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:44.885324    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7k46d"
Oct 30 21:02:44.941135 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:44.941087    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7k46d" podStartSLOduration=1.9410724940000001 podStartE2EDuration="1.941072494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:02:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:02:44.939084416 +0000 UTC m=+1448.003860813" watchObservedRunningTime="2026-10-30 21:02:44.941072494 +0000 UTC m=+1448.005848892"
Oct 30 21:02:45.886469 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:45.886436    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-7k46d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 30 21:02:45.886917 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:45.886494    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7k46d" podUID="4242f534-4d88-40d1-a4e0-e45493a00d7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 30 21:02:46.885628 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:46.885541    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-7k46d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 30 21:02:46.885628 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:46.885604    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7k46d" podUID="4242f534-4d88-40d1-a4e0-e45493a00d7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 30 21:02:47.885160 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:47.885129    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-7k46d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 30 21:02:47.885682 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:47.885177    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7k46d" podUID="4242f534-4d88-40d1-a4e0-e45493a00d7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 30 21:02:48.884695 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:48.884661    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-7k46d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 30 21:02:48.884899 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:48.884716    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7k46d" podUID="4242f534-4d88-40d1-a4e0-e45493a00d7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 30 21:02:49.898308 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:49.898261    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-7k46d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 30 21:02:49.898902 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:49.898359    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7k46d" podUID="4242f534-4d88-40d1-a4e0-e45493a00d7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 30 21:02:50.885089 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:50.885047    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-7k46d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 30 21:02:50.885277 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:50.885112    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7k46d" podUID="4242f534-4d88-40d1-a4e0-e45493a00d7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 30 21:02:51.181550 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:51.181517    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f55142ef-9s5nv"]
Oct 30 21:02:51.186607 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:51.186583    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f55142ef-9s5nv"
Oct 30 21:02:51.234527 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:51.234481    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d50e751c-bb2b-46a0-b7f3-7b79b8cc088a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f55142ef-9s5nv\" (UID: \"d50e751c-bb2b-46a0-b7f3-7b79b8cc088a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f55142ef-9s5nv"
Oct 30 21:02:51.234719 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:51.234548    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg45m\" (UniqueName: \"kubernetes.io/projected/d50e751c-bb2b-46a0-b7f3-7b79b8cc088a-kube-api-access-cg45m\") pod \"maas-api-revoke-keys-e2e-migrate-f55142ef-9s5nv\" (UID: \"d50e751c-bb2b-46a0-b7f3-7b79b8cc088a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f55142ef-9s5nv"
Oct 30 21:02:51.234901 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:51.234877    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d50e751c-bb2b-46a0-b7f3-7b79b8cc088a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f55142ef-9s5nv\" (UID: \"d50e751c-bb2b-46a0-b7f3-7b79b8cc088a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f55142ef-9s5nv"
Oct 30 21:02:51.242501 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:51.242470    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f55142ef-9s5nv"]
Oct 30 21:02:51.285621 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:51.285591    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:02:51.285621 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:51.285591    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:02:51.285860 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:51.285597    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:02:51.288522 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:51.288506    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:02:51.335957 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:51.335927    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d50e751c-bb2b-46a0-b7f3-7b79b8cc088a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f55142ef-9s5nv\" (UID: \"d50e751c-bb2b-46a0-b7f3-7b79b8cc088a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f55142ef-9s5nv"
Oct 30 21:02:51.336128 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:51.336022    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d50e751c-bb2b-46a0-b7f3-7b79b8cc088a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f55142ef-9s5nv\" (UID: \"d50e751c-bb2b-46a0-b7f3-7b79b8cc088a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f55142ef-9s5nv"
Oct 30 21:02:51.336128 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:51.336054    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg45m\" (UniqueName: \"kubernetes.io/projected/d50e751c-bb2b-46a0-b7f3-7b79b8cc088a-kube-api-access-cg45m\") pod \"maas-api-revoke-keys-e2e-migrate-f55142ef-9s5nv\" (UID: \"d50e751c-bb2b-46a0-b7f3-7b79b8cc088a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f55142ef-9s5nv"
Oct 30 21:02:51.336624 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:51.336603    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d50e751c-bb2b-46a0-b7f3-7b79b8cc088a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f55142ef-9s5nv\" (UID: \"d50e751c-bb2b-46a0-b7f3-7b79b8cc088a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f55142ef-9s5nv"
Oct 30 21:02:51.338649 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:51.338624    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d50e751c-bb2b-46a0-b7f3-7b79b8cc088a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f55142ef-9s5nv\" (UID: \"d50e751c-bb2b-46a0-b7f3-7b79b8cc088a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f55142ef-9s5nv"
Oct 30 21:02:51.395105 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:51.395072    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg45m\" (UniqueName: \"kubernetes.io/projected/d50e751c-bb2b-46a0-b7f3-7b79b8cc088a-kube-api-access-cg45m\") pod \"maas-api-revoke-keys-e2e-migrate-f55142ef-9s5nv\" (UID: \"d50e751c-bb2b-46a0-b7f3-7b79b8cc088a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f55142ef-9s5nv"
Oct 30 21:02:51.503356 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:51.503276    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f55142ef-9s5nv"
Oct 30 21:02:51.680297 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:51.680251    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5fc0962-5mcg4"]
Oct 30 21:02:51.686040 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:51.686011    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5fc0962-5mcg4"
Oct 30 21:02:51.742858 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:51.742765    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5104496-d047-41dc-9ed4-550e8c89725b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5fc0962-5mcg4\" (UID: \"a5104496-d047-41dc-9ed4-550e8c89725b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5fc0962-5mcg4"
Oct 30 21:02:51.743155 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:51.743124    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx975\" (UniqueName: \"kubernetes.io/projected/a5104496-d047-41dc-9ed4-550e8c89725b-kube-api-access-cx975\") pod \"maas-api-revoke-sub-d5fc0962-5mcg4\" (UID: \"a5104496-d047-41dc-9ed4-550e8c89725b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5fc0962-5mcg4"
Oct 30 21:02:51.743338 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:51.743322    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5104496-d047-41dc-9ed4-550e8c89725b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5fc0962-5mcg4\" (UID: \"a5104496-d047-41dc-9ed4-550e8c89725b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5fc0962-5mcg4"
Oct 30 21:02:51.744173 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:51.744151    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5fc0962-5mcg4"]
Oct 30 21:02:51.844388 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:51.844304    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5104496-d047-41dc-9ed4-550e8c89725b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5fc0962-5mcg4\" (UID: \"a5104496-d047-41dc-9ed4-550e8c89725b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5fc0962-5mcg4"
Oct 30 21:02:51.844564 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:51.844506    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5104496-d047-41dc-9ed4-550e8c89725b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5fc0962-5mcg4\" (UID: \"a5104496-d047-41dc-9ed4-550e8c89725b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5fc0962-5mcg4"
Oct 30 21:02:51.844631 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:51.844569    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx975\" (UniqueName: \"kubernetes.io/projected/a5104496-d047-41dc-9ed4-550e8c89725b-kube-api-access-cx975\") pod \"maas-api-revoke-sub-d5fc0962-5mcg4\" (UID: \"a5104496-d047-41dc-9ed4-550e8c89725b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5fc0962-5mcg4"
Oct 30 21:02:51.845112 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:51.845086    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5104496-d047-41dc-9ed4-550e8c89725b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5fc0962-5mcg4\" (UID: \"a5104496-d047-41dc-9ed4-550e8c89725b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5fc0962-5mcg4"
Oct 30 21:02:51.847086 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:51.847057    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5104496-d047-41dc-9ed4-550e8c89725b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5fc0962-5mcg4\" (UID: \"a5104496-d047-41dc-9ed4-550e8c89725b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5fc0962-5mcg4"
Oct 30 21:02:51.885065 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:51.885028    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-7k46d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 30 21:02:51.885199 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:51.885080    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7k46d" podUID="4242f534-4d88-40d1-a4e0-e45493a00d7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 30 21:02:51.902427 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:51.902394    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx975\" (UniqueName: \"kubernetes.io/projected/a5104496-d047-41dc-9ed4-550e8c89725b-kube-api-access-cx975\") pod \"maas-api-revoke-sub-d5fc0962-5mcg4\" (UID: \"a5104496-d047-41dc-9ed4-550e8c89725b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5fc0962-5mcg4"
Oct 30 21:02:51.917260 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:51.917231    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f55142ef-9s5nv"]
Oct 30 21:02:51.917581 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:02:51.917555    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd50e751c_bb2b_46a0_b7f3_7b79b8cc088a.slice/crio-f64f8538680f6042cdf92b93dc49dab1bf4ec7e23ba73d4dd6d99012a989d8c5 WatchSource:0}: Error finding container f64f8538680f6042cdf92b93dc49dab1bf4ec7e23ba73d4dd6d99012a989d8c5: Status 404 returned error can't find the container with id f64f8538680f6042cdf92b93dc49dab1bf4ec7e23ba73d4dd6d99012a989d8c5
Oct 30 21:02:52.001384 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:52.001356    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5fc0962-5mcg4"
Oct 30 21:02:52.416006 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:02:52.415969    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5104496_d047_41dc_9ed4_550e8c89725b.slice/crio-bfee0142476fada029dbef3e687ef496b21c98e56cc34388717672fc929e6e12 WatchSource:0}: Error finding container bfee0142476fada029dbef3e687ef496b21c98e56cc34388717672fc929e6e12: Status 404 returned error can't find the container with id bfee0142476fada029dbef3e687ef496b21c98e56cc34388717672fc929e6e12
Oct 30 21:02:52.416006 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:52.415992    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5fc0962-5mcg4"]
Oct 30 21:02:52.885311 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:52.885279    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-7k46d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 30 21:02:52.885468 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:52.885331    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7k46d" podUID="4242f534-4d88-40d1-a4e0-e45493a00d7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 30 21:02:52.934502 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:52.934468    5713 generic.go:356] "Generic (PLEG): container finished" podID="d50e751c-bb2b-46a0-b7f3-7b79b8cc088a" containerID="6ecb5620cfd99d8e90af5d7586a1de5de17fd2c7ea1f14f7a347f1c9e27ba3a0" exitCode=0
Oct 30 21:02:52.934662 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:52.934559    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f55142ef-9s5nv" event={"ID":"d50e751c-bb2b-46a0-b7f3-7b79b8cc088a","Type":"ContainerDied","Data":"6ecb5620cfd99d8e90af5d7586a1de5de17fd2c7ea1f14f7a347f1c9e27ba3a0"}
Oct 30 21:02:52.934662 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:52.934656    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f55142ef-9s5nv" event={"ID":"d50e751c-bb2b-46a0-b7f3-7b79b8cc088a","Type":"ContainerStarted","Data":"f64f8538680f6042cdf92b93dc49dab1bf4ec7e23ba73d4dd6d99012a989d8c5"}
Oct 30 21:02:52.936942 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:52.936905    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5fc0962-5mcg4" event={"ID":"a5104496-d047-41dc-9ed4-550e8c89725b","Type":"ContainerStarted","Data":"b981cd0d93974018b3b4f6f27068e7d9663fc47d94df51a35f1de1c233ede85b"}
Oct 30 21:02:52.937057 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:52.936970    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5fc0962-5mcg4" event={"ID":"a5104496-d047-41dc-9ed4-550e8c89725b","Type":"ContainerStarted","Data":"bfee0142476fada029dbef3e687ef496b21c98e56cc34388717672fc929e6e12"}
Oct 30 21:02:53.170292 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:53.170178    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5fc0962-5mcg4" podStartSLOduration=2.170158038 podStartE2EDuration="2.170158038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:02:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:02:53.160849665 +0000 UTC m=+1456.225626055" watchObservedRunningTime="2026-10-30 21:02:53.170158038 +0000 UTC m=+1456.234934436"
Oct 30 21:02:53.352202 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:53.352166    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83851760-n5v5c"]
Oct 30 21:02:53.358095 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:53.358071    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83851760-n5v5c"
Oct 30 21:02:53.409268 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:53.409236    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83851760-n5v5c"]
Oct 30 21:02:53.471389 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:53.471355    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-955m4\" (UniqueName: \"kubernetes.io/projected/6b920142-6a6c-4028-b919-6c736245ae93-kube-api-access-955m4\") pod \"maas-api-revoke-keys-e2e-ait-83851760-n5v5c\" (UID: \"6b920142-6a6c-4028-b919-6c736245ae93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83851760-n5v5c"
Oct 30 21:02:53.471954 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:53.471527    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b920142-6a6c-4028-b919-6c736245ae93-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-83851760-n5v5c\" (UID: \"6b920142-6a6c-4028-b919-6c736245ae93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83851760-n5v5c"
Oct 30 21:02:53.471954 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:53.471669    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b920142-6a6c-4028-b919-6c736245ae93-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-83851760-n5v5c\" (UID: \"6b920142-6a6c-4028-b919-6c736245ae93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83851760-n5v5c"
Oct 30 21:02:53.572580 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:53.572547    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b920142-6a6c-4028-b919-6c736245ae93-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-83851760-n5v5c\" (UID: \"6b920142-6a6c-4028-b919-6c736245ae93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83851760-n5v5c"
Oct 30 21:02:53.572775 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:53.572608    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-955m4\" (UniqueName: \"kubernetes.io/projected/6b920142-6a6c-4028-b919-6c736245ae93-kube-api-access-955m4\") pod \"maas-api-revoke-keys-e2e-ait-83851760-n5v5c\" (UID: \"6b920142-6a6c-4028-b919-6c736245ae93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83851760-n5v5c"
Oct 30 21:02:53.572897 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:53.572802    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b920142-6a6c-4028-b919-6c736245ae93-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-83851760-n5v5c\" (UID: \"6b920142-6a6c-4028-b919-6c736245ae93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83851760-n5v5c"
Oct 30 21:02:53.573347 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:53.573320    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b920142-6a6c-4028-b919-6c736245ae93-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-83851760-n5v5c\" (UID: \"6b920142-6a6c-4028-b919-6c736245ae93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83851760-n5v5c"
Oct 30 21:02:53.575449 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:53.575423    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b920142-6a6c-4028-b919-6c736245ae93-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-83851760-n5v5c\" (UID: \"6b920142-6a6c-4028-b919-6c736245ae93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83851760-n5v5c"
Oct 30 21:02:53.630215 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:53.630180    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-955m4\" (UniqueName: \"kubernetes.io/projected/6b920142-6a6c-4028-b919-6c736245ae93-kube-api-access-955m4\") pod \"maas-api-revoke-keys-e2e-ait-83851760-n5v5c\" (UID: \"6b920142-6a6c-4028-b919-6c736245ae93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83851760-n5v5c"
Oct 30 21:02:53.673138 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:53.673100    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83851760-n5v5c"
Oct 30 21:02:53.888941 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:53.888914    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7k46d"
Oct 30 21:02:53.889433 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:53.889416    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7k46d"
Oct 30 21:02:53.890071 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:53.890057    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7k46d"
Oct 30 21:02:53.947578 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:53.947541    5713 generic.go:356] "Generic (PLEG): container finished" podID="a5104496-d047-41dc-9ed4-550e8c89725b" containerID="b981cd0d93974018b3b4f6f27068e7d9663fc47d94df51a35f1de1c233ede85b" exitCode=0
Oct 30 21:02:53.947767 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:53.947635    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5fc0962-5mcg4" event={"ID":"a5104496-d047-41dc-9ed4-550e8c89725b","Type":"ContainerDied","Data":"b981cd0d93974018b3b4f6f27068e7d9663fc47d94df51a35f1de1c233ede85b"}
Oct 30 21:02:54.082020 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:54.081982    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83851760-n5v5c"]
Oct 30 21:02:54.092167 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:02:54.092136    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b920142_6a6c_4028_b919_6c736245ae93.slice/crio-bedad3a877b7c324cf32669b98b6f762f0aa8c3d95e316912db637d862f28105 WatchSource:0}: Error finding container bedad3a877b7c324cf32669b98b6f762f0aa8c3d95e316912db637d862f28105: Status 404 returned error can't find the container with id bedad3a877b7c324cf32669b98b6f762f0aa8c3d95e316912db637d862f28105
Oct 30 21:02:54.105172 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:54.105149    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f55142ef-9s5nv"
Oct 30 21:02:54.184139 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:54.184092    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d50e751c-bb2b-46a0-b7f3-7b79b8cc088a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d50e751c-bb2b-46a0-b7f3-7b79b8cc088a-maas-api-openshift-ca\") pod \"d50e751c-bb2b-46a0-b7f3-7b79b8cc088a\" (UID: \"d50e751c-bb2b-46a0-b7f3-7b79b8cc088a\") "
Oct 30 21:02:54.184320 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:54.184305    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d50e751c-bb2b-46a0-b7f3-7b79b8cc088a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d50e751c-bb2b-46a0-b7f3-7b79b8cc088a-maas-api-xks-ca\") pod \"d50e751c-bb2b-46a0-b7f3-7b79b8cc088a\" (UID: \"d50e751c-bb2b-46a0-b7f3-7b79b8cc088a\") "
Oct 30 21:02:54.184473 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:54.184457    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d50e751c-bb2b-46a0-b7f3-7b79b8cc088a-kube-api-access-cg45m\" (UniqueName: \"kubernetes.io/projected/d50e751c-bb2b-46a0-b7f3-7b79b8cc088a-kube-api-access-cg45m\") pod \"d50e751c-bb2b-46a0-b7f3-7b79b8cc088a\" (UID: \"d50e751c-bb2b-46a0-b7f3-7b79b8cc088a\") "
Oct 30 21:02:54.184550 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:54.184496    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d50e751c-bb2b-46a0-b7f3-7b79b8cc088a-maas-api-openshift-ca" pod "d50e751c-bb2b-46a0-b7f3-7b79b8cc088a" (UID: "d50e751c-bb2b-46a0-b7f3-7b79b8cc088a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:02:54.185039 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:54.185017    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d50e751c-bb2b-46a0-b7f3-7b79b8cc088a-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:02:54.186454 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:54.186433    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d50e751c-bb2b-46a0-b7f3-7b79b8cc088a-maas-api-xks-ca" pod "d50e751c-bb2b-46a0-b7f3-7b79b8cc088a" (UID: "d50e751c-bb2b-46a0-b7f3-7b79b8cc088a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:02:54.186731 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:54.186698    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d50e751c-bb2b-46a0-b7f3-7b79b8cc088a-kube-api-access-cg45m" pod "d50e751c-bb2b-46a0-b7f3-7b79b8cc088a" (UID: "d50e751c-bb2b-46a0-b7f3-7b79b8cc088a"). InnerVolumeSpecName "kube-api-access-cg45m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:02:54.287121 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:54.287041    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cg45m\" (UniqueName: \"kubernetes.io/projected/d50e751c-bb2b-46a0-b7f3-7b79b8cc088a-kube-api-access-cg45m\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:02:54.287121 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:54.287069    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d50e751c-bb2b-46a0-b7f3-7b79b8cc088a-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:02:54.464418 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:02:54.464381    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:02:54.959055 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:54.959024    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f55142ef-9s5nv"
Oct 30 21:02:54.959055 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:54.959040    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f55142ef-9s5nv" event={"ID":"d50e751c-bb2b-46a0-b7f3-7b79b8cc088a","Type":"ContainerDied","Data":"f64f8538680f6042cdf92b93dc49dab1bf4ec7e23ba73d4dd6d99012a989d8c5"}
Oct 30 21:02:54.959469 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:54.959098    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f64f8538680f6042cdf92b93dc49dab1bf4ec7e23ba73d4dd6d99012a989d8c5"
Oct 30 21:02:54.960906 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:54.960883    5713 generic.go:356] "Generic (PLEG): container finished" podID="6b920142-6a6c-4028-b919-6c736245ae93" containerID="9953ef86dc0084bbe4cdbfc8869b7630538de952f56d92d6d10a0f10d3095f5e" exitCode=0
Oct 30 21:02:54.961338 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:54.961304    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83851760-n5v5c" event={"ID":"6b920142-6a6c-4028-b919-6c736245ae93","Type":"ContainerDied","Data":"9953ef86dc0084bbe4cdbfc8869b7630538de952f56d92d6d10a0f10d3095f5e"}
Oct 30 21:02:54.961457 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:54.961351    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83851760-n5v5c" event={"ID":"6b920142-6a6c-4028-b919-6c736245ae93","Type":"ContainerStarted","Data":"bedad3a877b7c324cf32669b98b6f762f0aa8c3d95e316912db637d862f28105"}
Oct 30 21:02:55.141465 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:55.141443    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5fc0962-5mcg4"
Oct 30 21:02:55.210606 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:55.210527    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5104496-d047-41dc-9ed4-550e8c89725b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5104496-d047-41dc-9ed4-550e8c89725b-maas-api-xks-ca\") pod \"a5104496-d047-41dc-9ed4-550e8c89725b\" (UID: \"a5104496-d047-41dc-9ed4-550e8c89725b\") "
Oct 30 21:02:55.210606 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:55.210586    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5104496-d047-41dc-9ed4-550e8c89725b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5104496-d047-41dc-9ed4-550e8c89725b-maas-api-openshift-ca\") pod \"a5104496-d047-41dc-9ed4-550e8c89725b\" (UID: \"a5104496-d047-41dc-9ed4-550e8c89725b\") "
Oct 30 21:02:55.210780 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:55.210727    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5104496-d047-41dc-9ed4-550e8c89725b-kube-api-access-cx975\" (UniqueName: \"kubernetes.io/projected/a5104496-d047-41dc-9ed4-550e8c89725b-kube-api-access-cx975\") pod \"a5104496-d047-41dc-9ed4-550e8c89725b\" (UID: \"a5104496-d047-41dc-9ed4-550e8c89725b\") "
Oct 30 21:02:55.210947 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:55.210917    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5104496-d047-41dc-9ed4-550e8c89725b-maas-api-openshift-ca" pod "a5104496-d047-41dc-9ed4-550e8c89725b" (UID: "a5104496-d047-41dc-9ed4-550e8c89725b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:02:55.211335 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:55.211313    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5104496-d047-41dc-9ed4-550e8c89725b-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:02:55.212757 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:55.212730    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5104496-d047-41dc-9ed4-550e8c89725b-maas-api-xks-ca" pod "a5104496-d047-41dc-9ed4-550e8c89725b" (UID: "a5104496-d047-41dc-9ed4-550e8c89725b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:02:55.213044 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:55.213024    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5104496-d047-41dc-9ed4-550e8c89725b-kube-api-access-cx975" pod "a5104496-d047-41dc-9ed4-550e8c89725b" (UID: "a5104496-d047-41dc-9ed4-550e8c89725b"). InnerVolumeSpecName "kube-api-access-cx975". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:02:55.313024 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:55.312991    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5104496-d047-41dc-9ed4-550e8c89725b-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:02:55.313024 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:55.313018    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cx975\" (UniqueName: \"kubernetes.io/projected/a5104496-d047-41dc-9ed4-550e8c89725b-kube-api-access-cx975\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:02:55.979038 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:55.979007    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5fc0962-5mcg4"
Oct 30 21:02:55.979576 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:55.979039    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5fc0962-5mcg4" event={"ID":"a5104496-d047-41dc-9ed4-550e8c89725b","Type":"ContainerDied","Data":"bfee0142476fada029dbef3e687ef496b21c98e56cc34388717672fc929e6e12"}
Oct 30 21:02:55.979576 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:55.979092    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bfee0142476fada029dbef3e687ef496b21c98e56cc34388717672fc929e6e12"
Oct 30 21:02:56.148866 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:56.148839    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83851760-n5v5c"
Oct 30 21:02:56.230720 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:56.230621    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b920142-6a6c-4028-b919-6c736245ae93-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b920142-6a6c-4028-b919-6c736245ae93-maas-api-xks-ca\") pod \"6b920142-6a6c-4028-b919-6c736245ae93\" (UID: \"6b920142-6a6c-4028-b919-6c736245ae93\") "
Oct 30 21:02:56.230975 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:56.230955    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b920142-6a6c-4028-b919-6c736245ae93-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b920142-6a6c-4028-b919-6c736245ae93-maas-api-openshift-ca\") pod \"6b920142-6a6c-4028-b919-6c736245ae93\" (UID: \"6b920142-6a6c-4028-b919-6c736245ae93\") "
Oct 30 21:02:56.231047 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:56.230993    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b920142-6a6c-4028-b919-6c736245ae93-kube-api-access-955m4\" (UniqueName: \"kubernetes.io/projected/6b920142-6a6c-4028-b919-6c736245ae93-kube-api-access-955m4\") pod \"6b920142-6a6c-4028-b919-6c736245ae93\" (UID: \"6b920142-6a6c-4028-b919-6c736245ae93\") "
Oct 30 21:02:56.231291 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:56.231267    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b920142-6a6c-4028-b919-6c736245ae93-maas-api-openshift-ca" pod "6b920142-6a6c-4028-b919-6c736245ae93" (UID: "6b920142-6a6c-4028-b919-6c736245ae93"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:02:56.231626 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:56.231607    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b920142-6a6c-4028-b919-6c736245ae93-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:02:56.233282 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:56.233258    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b920142-6a6c-4028-b919-6c736245ae93-maas-api-xks-ca" pod "6b920142-6a6c-4028-b919-6c736245ae93" (UID: "6b920142-6a6c-4028-b919-6c736245ae93"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:02:56.233350 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:56.233305    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b920142-6a6c-4028-b919-6c736245ae93-kube-api-access-955m4" pod "6b920142-6a6c-4028-b919-6c736245ae93" (UID: "6b920142-6a6c-4028-b919-6c736245ae93"). InnerVolumeSpecName "kube-api-access-955m4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:02:56.332911 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:56.332874    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-955m4\" (UniqueName: \"kubernetes.io/projected/6b920142-6a6c-4028-b919-6c736245ae93-kube-api-access-955m4\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:02:56.332911 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:56.332905    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b920142-6a6c-4028-b919-6c736245ae93-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:02:56.989595 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:56.989570    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83851760-n5v5c"
Oct 30 21:02:56.990002 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:56.989606    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83851760-n5v5c" event={"ID":"6b920142-6a6c-4028-b919-6c736245ae93","Type":"ContainerDied","Data":"bedad3a877b7c324cf32669b98b6f762f0aa8c3d95e316912db637d862f28105"}
Oct 30 21:02:56.990002 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:56.989663    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bedad3a877b7c324cf32669b98b6f762f0aa8c3d95e316912db637d862f28105"
Oct 30 21:02:59.048718 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:59.048685    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c72e675-npr68"]
Oct 30 21:02:59.098290 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:02:59.098256    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c72e675-npr68"]
Oct 30 21:03:01.065864 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:01.065832    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bfeafa0-144d-4b07-acf4-800dbb9a8072" path="/var/lib/kubelet/pods/6bfeafa0-144d-4b07-acf4-800dbb9a8072/volumes"
Oct 30 21:03:05.446074 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:05.446040    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-83851760-6b5fd6887d-hcdv2"]
Oct 30 21:03:10.447286 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:10.447233    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-83851760-6b5fd6887d-hcdv2" podUID="ed3bfc15-452e-45a3-b7bf-545c615fcb34" containerName="payload-processing" containerID="cri-o://690479b2aa2bc4d7b1d12c65902960bbb247d29d0a1fc78a12a1a908ab61cf55" gracePeriod=25
Oct 30 21:03:10.729841 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:10.729816    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-83851760-6b5fd6887d-hcdv2"
Oct 30 21:03:10.773870 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:10.773835    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed3bfc15-452e-45a3-b7bf-545c615fcb34-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed3bfc15-452e-45a3-b7bf-545c615fcb34-plugins-config-volume\") pod \"ed3bfc15-452e-45a3-b7bf-545c615fcb34\" (UID: \"ed3bfc15-452e-45a3-b7bf-545c615fcb34\") "
Oct 30 21:03:10.774083 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:10.774061    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed3bfc15-452e-45a3-b7bf-545c615fcb34-kube-api-access-7v4mh\" (UniqueName: \"kubernetes.io/projected/ed3bfc15-452e-45a3-b7bf-545c615fcb34-kube-api-access-7v4mh\") pod \"ed3bfc15-452e-45a3-b7bf-545c615fcb34\" (UID: \"ed3bfc15-452e-45a3-b7bf-545c615fcb34\") "
Oct 30 21:03:10.774245 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:10.774225    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed3bfc15-452e-45a3-b7bf-545c615fcb34-plugins-config-volume" pod "ed3bfc15-452e-45a3-b7bf-545c615fcb34" (UID: "ed3bfc15-452e-45a3-b7bf-545c615fcb34"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:03:10.774687 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:10.774667    5713 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed3bfc15-452e-45a3-b7bf-545c615fcb34-plugins-config-volume\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:10.776335 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:10.776312    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed3bfc15-452e-45a3-b7bf-545c615fcb34-kube-api-access-7v4mh" pod "ed3bfc15-452e-45a3-b7bf-545c615fcb34" (UID: "ed3bfc15-452e-45a3-b7bf-545c615fcb34"). InnerVolumeSpecName "kube-api-access-7v4mh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:03:10.876584 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:10.876533    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v4mh\" (UniqueName: \"kubernetes.io/projected/ed3bfc15-452e-45a3-b7bf-545c615fcb34-kube-api-access-7v4mh\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:11.148698 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:11.148660    5713 generic.go:356] "Generic (PLEG): container finished" podID="ed3bfc15-452e-45a3-b7bf-545c615fcb34" containerID="690479b2aa2bc4d7b1d12c65902960bbb247d29d0a1fc78a12a1a908ab61cf55" exitCode=0
Oct 30 21:03:11.148945 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:11.148711    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-83851760-6b5fd6887d-hcdv2"
Oct 30 21:03:11.148945 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:11.148751    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-83851760-6b5fd6887d-hcdv2" event={"ID":"ed3bfc15-452e-45a3-b7bf-545c615fcb34","Type":"ContainerDied","Data":"690479b2aa2bc4d7b1d12c65902960bbb247d29d0a1fc78a12a1a908ab61cf55"}
Oct 30 21:03:11.148945 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:11.148828    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-83851760-6b5fd6887d-hcdv2" event={"ID":"ed3bfc15-452e-45a3-b7bf-545c615fcb34","Type":"ContainerDied","Data":"374572b3b12eabcd82711a1f091d2fa5b386cd4433768fe96a10ac7800a0190f"}
Oct 30 21:03:11.148945 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:11.148858    5713 scope.go:122] "RemoveContainer" containerID="690479b2aa2bc4d7b1d12c65902960bbb247d29d0a1fc78a12a1a908ab61cf55"
Oct 30 21:03:11.162164 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:11.162138    5713 scope.go:122] "RemoveContainer" containerID="690479b2aa2bc4d7b1d12c65902960bbb247d29d0a1fc78a12a1a908ab61cf55"
Oct 30 21:03:11.162518 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:03:11.162494    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"690479b2aa2bc4d7b1d12c65902960bbb247d29d0a1fc78a12a1a908ab61cf55\": container with ID starting with 690479b2aa2bc4d7b1d12c65902960bbb247d29d0a1fc78a12a1a908ab61cf55 not found: ID does not exist" containerID="690479b2aa2bc4d7b1d12c65902960bbb247d29d0a1fc78a12a1a908ab61cf55"
Oct 30 21:03:11.162616 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:11.162528    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"690479b2aa2bc4d7b1d12c65902960bbb247d29d0a1fc78a12a1a908ab61cf55"} err="failed to get container status \"690479b2aa2bc4d7b1d12c65902960bbb247d29d0a1fc78a12a1a908ab61cf55\": rpc error: code = NotFound desc = could not find container \"690479b2aa2bc4d7b1d12c65902960bbb247d29d0a1fc78a12a1a908ab61cf55\": container with ID starting with 690479b2aa2bc4d7b1d12c65902960bbb247d29d0a1fc78a12a1a908ab61cf55 not found: ID does not exist"
Oct 30 21:03:11.319391 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:11.319361    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-83851760-6b5fd6887d-hcdv2"]
Oct 30 21:03:11.324309 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:11.324248    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-83851760-6b5fd6887d-hcdv2"]
Oct 30 21:03:13.067589 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:13.067551    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed3bfc15-452e-45a3-b7bf-545c615fcb34" path="/var/lib/kubelet/pods/ed3bfc15-452e-45a3-b7bf-545c615fcb34/volumes"
Oct 30 21:03:14.048959 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:14.048925    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4278716-rj7l7"]
Oct 30 21:03:14.097724 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:14.097688    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4278716-rj7l7"]
Oct 30 21:03:15.066651 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:15.066617    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8481d84b-2a48-4160-8f0e-b239d27d2d2b" path="/var/lib/kubelet/pods/8481d84b-2a48-4160-8f0e-b239d27d2d2b/volumes"
Oct 30 21:03:24.484427 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:03:24.484391    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:03:25.872225 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:25.872194    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc88b34e-mh9pn"]
Oct 30 21:03:25.872926 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:25.872911    5713 cpu_manager.go:414] "containerMap: removing container" podUID="6b920142-6a6c-4028-b919-6c736245ae93" containerName="revoke-keys"
Oct 30 21:03:25.872974 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:25.872927    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b920142-6a6c-4028-b919-6c736245ae93" containerName="revoke-keys"
Oct 30 21:03:25.872974 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:25.872948    5713 cpu_manager.go:414] "containerMap: removing container" podUID="ed3bfc15-452e-45a3-b7bf-545c615fcb34" containerName="payload-processing"
Oct 30 21:03:25.872974 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:25.872953    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed3bfc15-452e-45a3-b7bf-545c615fcb34" containerName="payload-processing"
Oct 30 21:03:25.872974 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:25.872963    5713 cpu_manager.go:414] "containerMap: removing container" podUID="d50e751c-bb2b-46a0-b7f3-7b79b8cc088a" containerName="revoke-keys"
Oct 30 21:03:25.872974 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:25.872969    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d50e751c-bb2b-46a0-b7f3-7b79b8cc088a" containerName="revoke-keys"
Oct 30 21:03:25.873128 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:25.872982    5713 cpu_manager.go:414] "containerMap: removing container" podUID="a5104496-d047-41dc-9ed4-550e8c89725b" containerName="revoke-keys"
Oct 30 21:03:25.873128 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:25.872986    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5104496-d047-41dc-9ed4-550e8c89725b" containerName="revoke-keys"
Oct 30 21:03:25.873128 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:25.873093    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b920142-6a6c-4028-b919-6c736245ae93" containerName="revoke-keys"
Oct 30 21:03:25.873128 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:25.873103    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed3bfc15-452e-45a3-b7bf-545c615fcb34" containerName="payload-processing"
Oct 30 21:03:25.873128 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:25.873109    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="d50e751c-bb2b-46a0-b7f3-7b79b8cc088a" containerName="revoke-keys"
Oct 30 21:03:25.873128 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:25.873116    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5104496-d047-41dc-9ed4-550e8c89725b" containerName="revoke-keys"
Oct 30 21:03:25.876951 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:25.876931    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc88b34e-mh9pn"
Oct 30 21:03:25.936959 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:25.936924    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc88b34e-mh9pn"]
Oct 30 21:03:25.972667 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:25.972639    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/664ca418-ae7c-4106-8bfb-43e1255bb087-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-bc88b34e-mh9pn\" (UID: \"664ca418-ae7c-4106-8bfb-43e1255bb087\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc88b34e-mh9pn"
Oct 30 21:03:25.972853 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:25.972682    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/664ca418-ae7c-4106-8bfb-43e1255bb087-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-bc88b34e-mh9pn\" (UID: \"664ca418-ae7c-4106-8bfb-43e1255bb087\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc88b34e-mh9pn"
Oct 30 21:03:25.972853 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:25.972810    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgcsj\" (UniqueName: \"kubernetes.io/projected/664ca418-ae7c-4106-8bfb-43e1255bb087-kube-api-access-fgcsj\") pod \"maas-api-revoke-keys-e2e-ait-bc88b34e-mh9pn\" (UID: \"664ca418-ae7c-4106-8bfb-43e1255bb087\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc88b34e-mh9pn"
Oct 30 21:03:25.977661 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:25.977640    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:03:25.977661 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:25.977648    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:03:25.979435 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:25.979416    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:03:25.979627 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:25.979611    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:03:26.074369 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:26.074335    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/664ca418-ae7c-4106-8bfb-43e1255bb087-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-bc88b34e-mh9pn\" (UID: \"664ca418-ae7c-4106-8bfb-43e1255bb087\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc88b34e-mh9pn"
Oct 30 21:03:26.074571 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:26.074385    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/664ca418-ae7c-4106-8bfb-43e1255bb087-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-bc88b34e-mh9pn\" (UID: \"664ca418-ae7c-4106-8bfb-43e1255bb087\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc88b34e-mh9pn"
Oct 30 21:03:26.074571 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:26.074434    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgcsj\" (UniqueName: \"kubernetes.io/projected/664ca418-ae7c-4106-8bfb-43e1255bb087-kube-api-access-fgcsj\") pod \"maas-api-revoke-keys-e2e-ait-bc88b34e-mh9pn\" (UID: \"664ca418-ae7c-4106-8bfb-43e1255bb087\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc88b34e-mh9pn"
Oct 30 21:03:26.075095 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:26.075065    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/664ca418-ae7c-4106-8bfb-43e1255bb087-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-bc88b34e-mh9pn\" (UID: \"664ca418-ae7c-4106-8bfb-43e1255bb087\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc88b34e-mh9pn"
Oct 30 21:03:26.077075 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:26.077051    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/664ca418-ae7c-4106-8bfb-43e1255bb087-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-bc88b34e-mh9pn\" (UID: \"664ca418-ae7c-4106-8bfb-43e1255bb087\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc88b34e-mh9pn"
Oct 30 21:03:26.131620 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:26.131539    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgcsj\" (UniqueName: \"kubernetes.io/projected/664ca418-ae7c-4106-8bfb-43e1255bb087-kube-api-access-fgcsj\") pod \"maas-api-revoke-keys-e2e-ait-bc88b34e-mh9pn\" (UID: \"664ca418-ae7c-4106-8bfb-43e1255bb087\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc88b34e-mh9pn"
Oct 30 21:03:26.191641 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:26.191613    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc88b34e-mh9pn"
Oct 30 21:03:26.605299 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:03:26.605263    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod664ca418_ae7c_4106_8bfb_43e1255bb087.slice/crio-b14ca65e5eea3a6d22496fe8f7ce821a51f0b9d2e727de277a2fb62fb4b8aeb9 WatchSource:0}: Error finding container b14ca65e5eea3a6d22496fe8f7ce821a51f0b9d2e727de277a2fb62fb4b8aeb9: Status 404 returned error can't find the container with id b14ca65e5eea3a6d22496fe8f7ce821a51f0b9d2e727de277a2fb62fb4b8aeb9
Oct 30 21:03:26.609580 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:26.609557    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc88b34e-mh9pn"]
Oct 30 21:03:27.358622 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:27.358520    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc88b34e-mh9pn" event={"ID":"664ca418-ae7c-4106-8bfb-43e1255bb087","Type":"ContainerStarted","Data":"eedb447238bd70fcb9505a3d2556fc0d5221fc0e7e006962e6b51e11637c657e"}
Oct 30 21:03:27.358622 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:27.358591    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc88b34e-mh9pn" event={"ID":"664ca418-ae7c-4106-8bfb-43e1255bb087","Type":"ContainerStarted","Data":"b14ca65e5eea3a6d22496fe8f7ce821a51f0b9d2e727de277a2fb62fb4b8aeb9"}
Oct 30 21:03:27.474559 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:27.474493    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc88b34e-mh9pn" podStartSLOduration=2.474480239 podStartE2EDuration="2.474480239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:03:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:03:27.473909446 +0000 UTC m=+1490.538685846" watchObservedRunningTime="2026-10-30 21:03:27.474480239 +0000 UTC m=+1490.539256637"
Oct 30 21:03:28.369622 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:28.369594    5713 generic.go:356] "Generic (PLEG): container finished" podID="664ca418-ae7c-4106-8bfb-43e1255bb087" containerID="eedb447238bd70fcb9505a3d2556fc0d5221fc0e7e006962e6b51e11637c657e" exitCode=0
Oct 30 21:03:28.370025 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:28.369689    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc88b34e-mh9pn" event={"ID":"664ca418-ae7c-4106-8bfb-43e1255bb087","Type":"ContainerDied","Data":"eedb447238bd70fcb9505a3d2556fc0d5221fc0e7e006962e6b51e11637c657e"}
Oct 30 21:03:29.549240 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:29.549217    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc88b34e-mh9pn"
Oct 30 21:03:29.636003 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:29.635940    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/664ca418-ae7c-4106-8bfb-43e1255bb087-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/664ca418-ae7c-4106-8bfb-43e1255bb087-maas-api-openshift-ca\") pod \"664ca418-ae7c-4106-8bfb-43e1255bb087\" (UID: \"664ca418-ae7c-4106-8bfb-43e1255bb087\") "
Oct 30 21:03:29.637301 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:29.636105    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/664ca418-ae7c-4106-8bfb-43e1255bb087-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/664ca418-ae7c-4106-8bfb-43e1255bb087-maas-api-xks-ca\") pod \"664ca418-ae7c-4106-8bfb-43e1255bb087\" (UID: \"664ca418-ae7c-4106-8bfb-43e1255bb087\") "
Oct 30 21:03:29.637301 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:29.636279    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/664ca418-ae7c-4106-8bfb-43e1255bb087-kube-api-access-fgcsj\" (UniqueName: \"kubernetes.io/projected/664ca418-ae7c-4106-8bfb-43e1255bb087-kube-api-access-fgcsj\") pod \"664ca418-ae7c-4106-8bfb-43e1255bb087\" (UID: \"664ca418-ae7c-4106-8bfb-43e1255bb087\") "
Oct 30 21:03:29.637301 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:29.636349    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/664ca418-ae7c-4106-8bfb-43e1255bb087-maas-api-openshift-ca" pod "664ca418-ae7c-4106-8bfb-43e1255bb087" (UID: "664ca418-ae7c-4106-8bfb-43e1255bb087"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:03:29.637301 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:29.636905    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/664ca418-ae7c-4106-8bfb-43e1255bb087-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:29.638574 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:29.638544    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/664ca418-ae7c-4106-8bfb-43e1255bb087-maas-api-xks-ca" pod "664ca418-ae7c-4106-8bfb-43e1255bb087" (UID: "664ca418-ae7c-4106-8bfb-43e1255bb087"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:03:29.638670 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:29.638548    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/664ca418-ae7c-4106-8bfb-43e1255bb087-kube-api-access-fgcsj" pod "664ca418-ae7c-4106-8bfb-43e1255bb087" (UID: "664ca418-ae7c-4106-8bfb-43e1255bb087"). InnerVolumeSpecName "kube-api-access-fgcsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:03:29.738381 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:29.738346    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/664ca418-ae7c-4106-8bfb-43e1255bb087-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:29.738489 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:29.738386    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgcsj\" (UniqueName: \"kubernetes.io/projected/664ca418-ae7c-4106-8bfb-43e1255bb087-kube-api-access-fgcsj\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:30.388942 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:30.388907    5713 generic.go:356] "Generic (PLEG): container finished" podID="59246922-40fa-40ad-a84b-7604b077ef92" containerID="cb1db293e5e6f79d85e87828737f890a585cef853602139c637a94e501426ffa" exitCode=0
Oct 30 21:03:30.389135 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:30.388977    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj" event={"ID":"59246922-40fa-40ad-a84b-7604b077ef92","Type":"ContainerDied","Data":"cb1db293e5e6f79d85e87828737f890a585cef853602139c637a94e501426ffa"}
Oct 30 21:03:30.389459 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:30.389444    5713 scope.go:122] "RemoveContainer" containerID="cb1db293e5e6f79d85e87828737f890a585cef853602139c637a94e501426ffa"
Oct 30 21:03:30.390627 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:30.390612    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc88b34e-mh9pn"
Oct 30 21:03:30.390683 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:30.390634    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc88b34e-mh9pn" event={"ID":"664ca418-ae7c-4106-8bfb-43e1255bb087","Type":"ContainerDied","Data":"b14ca65e5eea3a6d22496fe8f7ce821a51f0b9d2e727de277a2fb62fb4b8aeb9"}
Oct 30 21:03:30.390683 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:30.390668    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b14ca65e5eea3a6d22496fe8f7ce821a51f0b9d2e727de277a2fb62fb4b8aeb9"
Oct 30 21:03:30.391110 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:30.391084    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:03:30.391200 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:30.391150    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:03:30.391200 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:30.391193    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:03:31.402315 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:31.402275    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj" event={"ID":"59246922-40fa-40ad-a84b-7604b077ef92","Type":"ContainerStarted","Data":"3a2fcbaa407c72cdbd80eeb60a83c8b9e3a0a20f2ff6a90294f077bf5720a6c9"}
Oct 30 21:03:32.218891 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:32.218853    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj"
Oct 30 21:03:32.220983 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:32.220951    5713 patch_prober.go:29] interesting pod/e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Oct 30 21:03:32.221138 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:32.221013    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj" podUID="59246922-40fa-40ad-a84b-7604b077ef92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Oct 30 21:03:33.222816 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:33.222218    5713 patch_prober.go:29] interesting pod/e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Oct 30 21:03:33.222816 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:33.222305    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj" podUID="59246922-40fa-40ad-a84b-7604b077ef92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Oct 30 21:03:33.955173 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:33.953892    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-781058/premium-w0-781058-kserve-7f5c8964c9-z5fmk" podUID="57626f4f-844e-4ec9-ab89-359d015923af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:03:33.955492 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:33.955458    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-781058/distinct-2-w0-781058-kserve-7f6c654757-ppmhm" podUID="5e818303-0e9f-4ba0-a635-b80b92c9e36d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:03:34.011049 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:34.011009    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b425d/simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk" podUID="e8108824-03ff-4c1d-9f5b-07be3a3a92d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:03:34.031920 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:34.031873    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-682999/distinct-w3-682999-kserve-74d49d4955-nndlc" podUID="2a35314b-31ea-42e6-b49a-15ab1305b338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:03:34.054816 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:34.051892    5713 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2b425d/simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk" podUID="e8108824-03ff-4c1d-9f5b-07be3a3a92d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:03:34.054816 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:34.052013    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b425d/premium-w5-2b425d-kserve-78756b6c5b-9x5xb" podUID="abf7b512-50e5-4400-9ca3-9ffccaead3d7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 30 21:03:34.084925 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:34.084878    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7" podUID="b2193213-ebae-498a-911a-6056a8e94312" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:03:34.085482 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:34.085447    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm" podUID="cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Oct 30 21:03:34.181813 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:34.171895    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b425d/distinct-2-w5-2b425d-kserve-6694867c68-bzpvn" podUID="b9d391e6-de6f-476d-a926-d1b333474109" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:03:34.182467 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:34.182426    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f9f277/premium-w1-f9f277-kserve-5dbfd4d496-72w4n" podUID="1cc1f811-c149-4457-80f1-1a55621d73b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 30 21:03:34.191920 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:34.191877    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-682999/premium-w3-682999-kserve-5f47cf9b8d-xqnnr" podUID="a54cd33d-feef-42bb-a78b-7480634cb58d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:03:34.229883 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:34.229248    5713 patch_prober.go:29] interesting pod/e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Oct 30 21:03:34.229883 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:34.229314    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj" podUID="59246922-40fa-40ad-a84b-7604b077ef92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Oct 30 21:03:34.255462 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:34.255414    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-781058/simulator-w0-781058-kserve-55b954cb4b-4k7z6" podUID="92c43343-2820-4097-b829-9aae96b670de" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:03:34.398945 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:34.398894    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l" podUID="fc9510c5-2180-41d6-90bc-9acdff0f3616" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded"
Oct 30 21:03:34.425934 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:34.425875    5713 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l" podUID="fc9510c5-2180-41d6-90bc-9acdff0f3616" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:03:34.692961 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:34.692905    5713 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-781058/premium-w0-781058-kserve-7f5c8964c9-z5fmk" podUID="57626f4f-844e-4ec9-ab89-359d015923af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:03:34.761826 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:34.757323    5713 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-f9f277/premium-w1-f9f277-kserve-5dbfd4d496-72w4n" podUID="1cc1f811-c149-4457-80f1-1a55621d73b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:03:34.856714 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:34.855883    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-682999/distinct-2-w3-682999-kserve-7bc687885b-gq5x7" podUID="0dcf0422-081f-46d7-b246-7a49b074c88c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Oct 30 21:03:34.871028 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:34.870936    5713 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7" podUID="b2193213-ebae-498a-911a-6056a8e94312" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:03:34.937652 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:34.936842    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-682999/simulator-w3-682999-kserve-57cfcb6f8f-6mgrg" podUID="53296e7f-c6bd-45f2-ac31-004feddc63a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:03:34.961882 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:34.960026    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-781058/distinct-2-w0-781058-kserve-7f6c654757-ppmhm" podUID="5e818303-0e9f-4ba0-a635-b80b92c9e36d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 30 21:03:34.961882 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:34.961702    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-781058/distinct-2-w0-781058-kserve-7f6c654757-ppmhm"
Oct 30 21:03:34.964902 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:34.964868    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-781058/premium-w0-781058-kserve-7f5c8964c9-z5fmk" podUID="57626f4f-844e-4ec9-ab89-359d015923af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:03:34.967062 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:34.967043    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-781058/premium-w0-781058-kserve-7f5c8964c9-z5fmk"
Oct 30 21:03:35.012941 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:35.012888    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b425d/simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk" podUID="e8108824-03ff-4c1d-9f5b-07be3a3a92d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:03:35.013624 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:35.013601    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2b425d/simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk"
Oct 30 21:03:35.032998 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:35.032947    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-682999/distinct-w3-682999-kserve-74d49d4955-nndlc" podUID="2a35314b-31ea-42e6-b49a-15ab1305b338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:03:35.033741 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:35.033715    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-682999/distinct-w3-682999-kserve-74d49d4955-nndlc"
Oct 30 21:03:35.054405 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:35.054364    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b425d/premium-w5-2b425d-kserve-78756b6c5b-9x5xb" podUID="abf7b512-50e5-4400-9ca3-9ffccaead3d7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 30 21:03:35.055104 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:35.055079    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2b425d/premium-w5-2b425d-kserve-78756b6c5b-9x5xb"
Oct 30 21:03:35.085512 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:35.085472    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7" podUID="b2193213-ebae-498a-911a-6056a8e94312" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded"
Oct 30 21:03:35.086465 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:35.086430    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm" podUID="cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:03:35.133822 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:35.115043    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7"
Oct 30 21:03:35.133822 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:35.115104    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm"
Oct 30 21:03:35.219871 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:35.201021    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f9f277/premium-w1-f9f277-kserve-5dbfd4d496-72w4n" podUID="1cc1f811-c149-4457-80f1-1a55621d73b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 30 21:03:35.219871 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:35.201555    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f9f277/premium-w1-f9f277-kserve-5dbfd4d496-72w4n"
Oct 30 21:03:35.284821 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:35.221689    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-682999/premium-w3-682999-kserve-5f47cf9b8d-xqnnr" podUID="a54cd33d-feef-42bb-a78b-7480634cb58d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:03:35.284821 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:35.268937    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-682999/premium-w3-682999-kserve-5f47cf9b8d-xqnnr"
Oct 30 21:03:35.284821 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:35.272880    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-781058/simulator-w0-781058-kserve-55b954cb4b-4k7z6" podUID="92c43343-2820-4097-b829-9aae96b670de" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:03:35.284821 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:35.273331    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-781058/simulator-w0-781058-kserve-55b954cb4b-4k7z6"
Oct 30 21:03:35.324775 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:35.321328    5713 patch_prober.go:29] interesting pod/e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Oct 30 21:03:35.324775 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:35.321391    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj" podUID="59246922-40fa-40ad-a84b-7604b077ef92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Oct 30 21:03:35.447832 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:35.444177    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l" podUID="fc9510c5-2180-41d6-90bc-9acdff0f3616" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:03:35.447832 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:35.444573    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l"
Oct 30 21:03:35.811040 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:35.810979    5713 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-781058/simulator-w0-781058-kserve-55b954cb4b-4k7z6" podUID="92c43343-2820-4097-b829-9aae96b670de" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:03:35.837810 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:35.816176    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-781058/distinct-2-w0-781058-kserve-7f6c654757-ppmhm"
Oct 30 21:03:35.884438 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:35.884297    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-682999/distinct-2-w3-682999-kserve-7bc687885b-gq5x7" podUID="0dcf0422-081f-46d7-b246-7a49b074c88c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Oct 30 21:03:35.920894 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:35.915956    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-682999/distinct-2-w3-682999-kserve-7bc687885b-gq5x7"
Oct 30 21:03:35.920894 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:35.916087    5713 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2b425d/distinct-2-w5-2b425d-kserve-6694867c68-bzpvn" podUID="b9d391e6-de6f-476d-a926-d1b333474109" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": context deadline exceeded"
Oct 30 21:03:35.920894 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:35.916499    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-682999/distinct-w3-682999-kserve-74d49d4955-nndlc"
Oct 30 21:03:35.938857 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:35.928599    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7"
Oct 30 21:03:35.979849 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:35.976911    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-781058/premium-w0-781058-kserve-7f5c8964c9-z5fmk" podUID="57626f4f-844e-4ec9-ab89-359d015923af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 30 21:03:35.995838 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:35.981584    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm"
Oct 30 21:03:36.012086 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:36.011319    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-682999/simulator-w3-682999-kserve-57cfcb6f8f-6mgrg" podUID="53296e7f-c6bd-45f2-ac31-004feddc63a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:03:36.013842 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:36.012485    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-682999/simulator-w3-682999-kserve-57cfcb6f8f-6mgrg"
Oct 30 21:03:36.013842 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:36.013405    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b425d/simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk" podUID="e8108824-03ff-4c1d-9f5b-07be3a3a92d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:03:36.023574 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:36.023531    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-682999/premium-w3-682999-kserve-5f47cf9b8d-xqnnr"
Oct 30 21:03:36.023992 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:36.023977    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2b425d/premium-w5-2b425d-kserve-78756b6c5b-9x5xb"
Oct 30 21:03:36.024816 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:36.024780    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-781058/simulator-w0-781058-kserve-55b954cb4b-4k7z6"
Oct 30 21:03:36.087871 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:36.068873    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f9f277/premium-w1-f9f277-kserve-5dbfd4d496-72w4n"
Oct 30 21:03:36.190400 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:36.174172    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-682999/distinct-2-w3-682999-kserve-7bc687885b-gq5x7"
Oct 30 21:03:36.190400 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:36.174228    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-781058/premium-w0-781058-kserve-7f5c8964c9-z5fmk"
Oct 30 21:03:36.190400 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:36.189738    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-682999/simulator-w3-682999-kserve-57cfcb6f8f-6mgrg"
Oct 30 21:03:36.192006 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:36.190851    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l"
Oct 30 21:03:36.202348 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:36.201966    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2b425d/simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk"
Oct 30 21:03:36.218424 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:36.218398    5713 patch_prober.go:29] interesting pod/e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Oct 30 21:03:36.218620 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:36.218601    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj" podUID="59246922-40fa-40ad-a84b-7604b077ef92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Oct 30 21:03:37.230267 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:37.230234    5713 patch_prober.go:29] interesting pod/e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Oct 30 21:03:37.233758 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:37.233712    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj" podUID="59246922-40fa-40ad-a84b-7604b077ef92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Oct 30 21:03:37.412177 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:37.403648    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d4597073-8ff0-4a4e-933c-c64e934fe977/prometheus/0.log"
Oct 30 21:03:37.421816 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:37.419145    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d4597073-8ff0-4a4e-933c-c64e934fe977/prometheus/0.log"
Oct 30 21:03:37.446221 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:37.446196    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r7lpf_764461d5-e0b2-4856-a97e-0c5d3d34a497/ovn-acl-logging/0.log"
Oct 30 21:03:37.448153 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:37.448132    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r7lpf_764461d5-e0b2-4856-a97e-0c5d3d34a497/ovn-acl-logging/0.log"
Oct 30 21:03:37.612883 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:37.612800    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bc88b34e-b49b84d8c-hzzvc"]
Oct 30 21:03:38.071035 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:38.071003    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb"]
Oct 30 21:03:38.071479 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:38.071453    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb" podUID="d335bbaf-1e6b-49cb-8bb1-89d987d18a90" containerName="istio-proxy" containerID="cri-o://962dbe672b45e3853f7af7675cd368c7a5eeff5fcd923839bbe76855a6814b80" gracePeriod=30
Oct 30 21:03:38.086302 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:38.086276    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f55142ef-9s5nv"]
Oct 30 21:03:38.097238 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:38.097171    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f55142ef-9s5nv"]
Oct 30 21:03:38.232814 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:38.232504    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj"
Oct 30 21:03:38.241819 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:38.236681    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj"
Oct 30 21:03:38.241819 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:38.237232    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj"
Oct 30 21:03:39.065037 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:39.065005    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d50e751c-bb2b-46a0-b7f3-7b79b8cc088a" path="/var/lib/kubelet/pods/d50e751c-bb2b-46a0-b7f3-7b79b8cc088a/volumes"
Oct 30 21:03:39.065377 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:39.065360    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d562e94-59xbh"]
Oct 30 21:03:39.065448 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:39.065396    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d562e94-59xbh"]
Oct 30 21:03:39.511370 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:39.511335    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83851760-n5v5c"]
Oct 30 21:03:39.519907 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:39.519861    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-83851760-n5v5c"]
Oct 30 21:03:41.065548 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:41.065509    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b920142-6a6c-4028-b919-6c736245ae93" path="/var/lib/kubelet/pods/6b920142-6a6c-4028-b919-6c736245ae93/volumes"
Oct 30 21:03:41.066763 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:41.066733    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e7f7cf4-9ed5-40d4-aa69-2e086b1854d1" path="/var/lib/kubelet/pods/7e7f7cf4-9ed5-40d4-aa69-2e086b1854d1/volumes"
Oct 30 21:03:42.613565 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:42.613501    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-bc88b34e-b49b84d8c-hzzvc" podUID="bb318efb-e29b-49fa-98b0-7ecf8cb93531" containerName="payload-pre-processing" containerID="cri-o://58baed66ce07af43ed601a86e63d65d2dd91c6b67c2ea3e4abde3a2aeb7fab66" gracePeriod=25
Oct 30 21:03:42.786458 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:03:42.786431    5713 logging.go:55] [core] [Channel #1754 SubChannel #1755]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.123:50052", ServerName: "10.134.0.123:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.123:50052: connect: connection refused"
Oct 30 21:03:42.908332 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:42.908305    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-bc88b34e-b49b84d8c-hzzvc"
Oct 30 21:03:42.980772 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:42.980737    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb318efb-e29b-49fa-98b0-7ecf8cb93531-kube-api-access-6j7ng\" (UniqueName: \"kubernetes.io/projected/bb318efb-e29b-49fa-98b0-7ecf8cb93531-kube-api-access-6j7ng\") pod \"bb318efb-e29b-49fa-98b0-7ecf8cb93531\" (UID: \"bb318efb-e29b-49fa-98b0-7ecf8cb93531\") "
Oct 30 21:03:42.981009 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:42.980826    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb318efb-e29b-49fa-98b0-7ecf8cb93531-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb318efb-e29b-49fa-98b0-7ecf8cb93531-plugins-config-volume\") pod \"bb318efb-e29b-49fa-98b0-7ecf8cb93531\" (UID: \"bb318efb-e29b-49fa-98b0-7ecf8cb93531\") "
Oct 30 21:03:42.981149 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:42.981127    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb318efb-e29b-49fa-98b0-7ecf8cb93531-plugins-config-volume" pod "bb318efb-e29b-49fa-98b0-7ecf8cb93531" (UID: "bb318efb-e29b-49fa-98b0-7ecf8cb93531"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:03:42.981619 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:42.981596    5713 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb318efb-e29b-49fa-98b0-7ecf8cb93531-plugins-config-volume\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:42.983252 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:42.983229    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb318efb-e29b-49fa-98b0-7ecf8cb93531-kube-api-access-6j7ng" pod "bb318efb-e29b-49fa-98b0-7ecf8cb93531" (UID: "bb318efb-e29b-49fa-98b0-7ecf8cb93531"). InnerVolumeSpecName "kube-api-access-6j7ng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:03:43.083194 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:43.083165    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6j7ng\" (UniqueName: \"kubernetes.io/projected/bb318efb-e29b-49fa-98b0-7ecf8cb93531-kube-api-access-6j7ng\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:43.314514 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:43.314478    5713 generic.go:356] "Generic (PLEG): container finished" podID="d335bbaf-1e6b-49cb-8bb1-89d987d18a90" containerID="962dbe672b45e3853f7af7675cd368c7a5eeff5fcd923839bbe76855a6814b80" exitCode=0
Oct 30 21:03:43.314908 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:43.314882    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb" event={"ID":"d335bbaf-1e6b-49cb-8bb1-89d987d18a90","Type":"ContainerDied","Data":"962dbe672b45e3853f7af7675cd368c7a5eeff5fcd923839bbe76855a6814b80"}
Oct 30 21:03:43.315536 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:43.315514    5713 scope.go:122] "RemoveContainer" containerID="a1f615299e1b3df5d35d745da95a15e5ee3e605e0fb9efb025bd729eb3163eab"
Oct 30 21:03:43.316473 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:43.316442    5713 generic.go:356] "Generic (PLEG): container finished" podID="bb318efb-e29b-49fa-98b0-7ecf8cb93531" containerID="58baed66ce07af43ed601a86e63d65d2dd91c6b67c2ea3e4abde3a2aeb7fab66" exitCode=0
Oct 30 21:03:43.316582 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:43.316486    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-bc88b34e-b49b84d8c-hzzvc" event={"ID":"bb318efb-e29b-49fa-98b0-7ecf8cb93531","Type":"ContainerDied","Data":"58baed66ce07af43ed601a86e63d65d2dd91c6b67c2ea3e4abde3a2aeb7fab66"}
Oct 30 21:03:43.316582 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:43.316505    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-bc88b34e-b49b84d8c-hzzvc"
Oct 30 21:03:43.316582 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:43.316524    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-bc88b34e-b49b84d8c-hzzvc" event={"ID":"bb318efb-e29b-49fa-98b0-7ecf8cb93531","Type":"ContainerDied","Data":"33214b3c94dd3df76516b9c270c5ff61d19e1d00c4906268040b3d20c23e6b4c"}
Oct 30 21:03:43.335610 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:43.335582    5713 scope.go:122] "RemoveContainer" containerID="58baed66ce07af43ed601a86e63d65d2dd91c6b67c2ea3e4abde3a2aeb7fab66"
Oct 30 21:03:43.352703 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:43.352678    5713 scope.go:122] "RemoveContainer" containerID="58baed66ce07af43ed601a86e63d65d2dd91c6b67c2ea3e4abde3a2aeb7fab66"
Oct 30 21:03:43.355430 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:03:43.353328    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58baed66ce07af43ed601a86e63d65d2dd91c6b67c2ea3e4abde3a2aeb7fab66\": container with ID starting with 58baed66ce07af43ed601a86e63d65d2dd91c6b67c2ea3e4abde3a2aeb7fab66 not found: ID does not exist" containerID="58baed66ce07af43ed601a86e63d65d2dd91c6b67c2ea3e4abde3a2aeb7fab66"
Oct 30 21:03:43.355430 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:43.353855    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58baed66ce07af43ed601a86e63d65d2dd91c6b67c2ea3e4abde3a2aeb7fab66"} err="failed to get container status \"58baed66ce07af43ed601a86e63d65d2dd91c6b67c2ea3e4abde3a2aeb7fab66\": rpc error: code = NotFound desc = could not find container \"58baed66ce07af43ed601a86e63d65d2dd91c6b67c2ea3e4abde3a2aeb7fab66\": container with ID starting with 58baed66ce07af43ed601a86e63d65d2dd91c6b67c2ea3e4abde3a2aeb7fab66 not found: ID does not exist"
Oct 30 21:03:43.407226 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:43.407202    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb"
Oct 30 21:03:43.483933 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:43.483831    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bc88b34e-b49b84d8c-hzzvc"]
Oct 30 21:03:43.488905 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:43.488880    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-bc88b34e-b49b84d8c-hzzvc"]
Oct 30 21:03:43.489261 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:43.489241    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d335bbaf-1e6b-49cb-8bb1-89d987d18a90-istio-token\" (UniqueName: \"kubernetes.io/projected/d335bbaf-1e6b-49cb-8bb1-89d987d18a90-istio-token\") pod \"d335bbaf-1e6b-49cb-8bb1-89d987d18a90\" (UID: \"d335bbaf-1e6b-49cb-8bb1-89d987d18a90\") "
Oct 30 21:03:43.489301 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:43.489277    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d335bbaf-1e6b-49cb-8bb1-89d987d18a90-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d335bbaf-1e6b-49cb-8bb1-89d987d18a90-credential-socket\") pod \"d335bbaf-1e6b-49cb-8bb1-89d987d18a90\" (UID: \"d335bbaf-1e6b-49cb-8bb1-89d987d18a90\") "
Oct 30 21:03:43.489375 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:43.489362    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d335bbaf-1e6b-49cb-8bb1-89d987d18a90-kube-api-access-4fqqq\" (UniqueName: \"kubernetes.io/projected/d335bbaf-1e6b-49cb-8bb1-89d987d18a90-kube-api-access-4fqqq\") pod \"d335bbaf-1e6b-49cb-8bb1-89d987d18a90\" (UID: \"d335bbaf-1e6b-49cb-8bb1-89d987d18a90\") "
Oct 30 21:03:43.489410 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:43.489397    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d335bbaf-1e6b-49cb-8bb1-89d987d18a90-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d335bbaf-1e6b-49cb-8bb1-89d987d18a90-istio-envoy\") pod \"d335bbaf-1e6b-49cb-8bb1-89d987d18a90\" (UID: \"d335bbaf-1e6b-49cb-8bb1-89d987d18a90\") "
Oct 30 21:03:43.489478 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:43.489459    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d335bbaf-1e6b-49cb-8bb1-89d987d18a90-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d335bbaf-1e6b-49cb-8bb1-89d987d18a90-istio-data\") pod \"d335bbaf-1e6b-49cb-8bb1-89d987d18a90\" (UID: \"d335bbaf-1e6b-49cb-8bb1-89d987d18a90\") "
Oct 30 21:03:43.489551 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:43.489502    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d335bbaf-1e6b-49cb-8bb1-89d987d18a90-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d335bbaf-1e6b-49cb-8bb1-89d987d18a90-workload-certs\") pod \"d335bbaf-1e6b-49cb-8bb1-89d987d18a90\" (UID: \"d335bbaf-1e6b-49cb-8bb1-89d987d18a90\") "
Oct 30 21:03:43.489642 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:43.489625    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d335bbaf-1e6b-49cb-8bb1-89d987d18a90-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d335bbaf-1e6b-49cb-8bb1-89d987d18a90-workload-socket\") pod \"d335bbaf-1e6b-49cb-8bb1-89d987d18a90\" (UID: \"d335bbaf-1e6b-49cb-8bb1-89d987d18a90\") "
Oct 30 21:03:43.489715 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:43.489632    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d335bbaf-1e6b-49cb-8bb1-89d987d18a90-credential-socket" pod "d335bbaf-1e6b-49cb-8bb1-89d987d18a90" (UID: "d335bbaf-1e6b-49cb-8bb1-89d987d18a90"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:43.489853 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:43.489818    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d335bbaf-1e6b-49cb-8bb1-89d987d18a90-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d335bbaf-1e6b-49cb-8bb1-89d987d18a90-istiod-ca-cert\") pod \"d335bbaf-1e6b-49cb-8bb1-89d987d18a90\" (UID: \"d335bbaf-1e6b-49cb-8bb1-89d987d18a90\") "
Oct 30 21:03:43.490000 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:43.489893    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d335bbaf-1e6b-49cb-8bb1-89d987d18a90-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d335bbaf-1e6b-49cb-8bb1-89d987d18a90-istio-podinfo\") pod \"d335bbaf-1e6b-49cb-8bb1-89d987d18a90\" (UID: \"d335bbaf-1e6b-49cb-8bb1-89d987d18a90\") "
Oct 30 21:03:43.490000 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:43.489837    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d335bbaf-1e6b-49cb-8bb1-89d987d18a90-istio-data" pod "d335bbaf-1e6b-49cb-8bb1-89d987d18a90" (UID: "d335bbaf-1e6b-49cb-8bb1-89d987d18a90"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:43.490000 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:43.489918    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d335bbaf-1e6b-49cb-8bb1-89d987d18a90-workload-certs" pod "d335bbaf-1e6b-49cb-8bb1-89d987d18a90" (UID: "d335bbaf-1e6b-49cb-8bb1-89d987d18a90"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:43.490000 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:43.489878    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d335bbaf-1e6b-49cb-8bb1-89d987d18a90-workload-socket" pod "d335bbaf-1e6b-49cb-8bb1-89d987d18a90" (UID: "d335bbaf-1e6b-49cb-8bb1-89d987d18a90"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:43.490361 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:43.490332    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d335bbaf-1e6b-49cb-8bb1-89d987d18a90-istiod-ca-cert" pod "d335bbaf-1e6b-49cb-8bb1-89d987d18a90" (UID: "d335bbaf-1e6b-49cb-8bb1-89d987d18a90"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:03:43.490821 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:43.490764    5713 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d335bbaf-1e6b-49cb-8bb1-89d987d18a90-istiod-ca-cert\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:43.490821 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:43.490811    5713 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d335bbaf-1e6b-49cb-8bb1-89d987d18a90-credential-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:43.490914 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:43.490827    5713 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d335bbaf-1e6b-49cb-8bb1-89d987d18a90-istio-data\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:43.490914 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:43.490840    5713 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d335bbaf-1e6b-49cb-8bb1-89d987d18a90-workload-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:43.490914 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:43.490854    5713 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d335bbaf-1e6b-49cb-8bb1-89d987d18a90-workload-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:43.491739 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:43.491721    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d335bbaf-1e6b-49cb-8bb1-89d987d18a90-istio-token" pod "d335bbaf-1e6b-49cb-8bb1-89d987d18a90" (UID: "d335bbaf-1e6b-49cb-8bb1-89d987d18a90"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:03:43.492014 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:43.491990    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d335bbaf-1e6b-49cb-8bb1-89d987d18a90-istio-podinfo" pod "d335bbaf-1e6b-49cb-8bb1-89d987d18a90" (UID: "d335bbaf-1e6b-49cb-8bb1-89d987d18a90"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:03:43.492314 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:43.492294    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d335bbaf-1e6b-49cb-8bb1-89d987d18a90-kube-api-access-4fqqq" pod "d335bbaf-1e6b-49cb-8bb1-89d987d18a90" (UID: "d335bbaf-1e6b-49cb-8bb1-89d987d18a90"). InnerVolumeSpecName "kube-api-access-4fqqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:03:43.492383 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:43.492294    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d335bbaf-1e6b-49cb-8bb1-89d987d18a90-istio-envoy" pod "d335bbaf-1e6b-49cb-8bb1-89d987d18a90" (UID: "d335bbaf-1e6b-49cb-8bb1-89d987d18a90"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:43.592547 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:43.592516    5713 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d335bbaf-1e6b-49cb-8bb1-89d987d18a90-istio-token\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:43.592547 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:43.592544    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fqqq\" (UniqueName: \"kubernetes.io/projected/d335bbaf-1e6b-49cb-8bb1-89d987d18a90-kube-api-access-4fqqq\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:43.592547 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:43.592554    5713 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d335bbaf-1e6b-49cb-8bb1-89d987d18a90-istio-envoy\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:43.592775 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:43.592566    5713 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d335bbaf-1e6b-49cb-8bb1-89d987d18a90-istio-podinfo\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:03:43.785204 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:43.785119    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ait-bc88b34e-b49b84d8c-hzzvc" podUID="bb318efb-e29b-49fa-98b0-7ecf8cb93531" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.123:50052\" within 1s: context deadline exceeded"
Oct 30 21:03:44.326348 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:44.326319    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb"
Oct 30 21:03:44.326509 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:44.326354    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb" event={"ID":"d335bbaf-1e6b-49cb-8bb1-89d987d18a90","Type":"ContainerDied","Data":"4652051b1da9d2f8943e3908498219514c9c45a7e789359195c4eb9d5f66ad03"}
Oct 30 21:03:44.326509 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:44.326413    5713 scope.go:122] "RemoveContainer" containerID="962dbe672b45e3853f7af7675cd368c7a5eeff5fcd923839bbe76855a6814b80"
Oct 30 21:03:44.508914 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:44.508879    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb"]
Oct 30 21:03:44.557050 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:44.556999    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-f55142ef-gw-openshift-default-654b97b657-v4fcb"]
Oct 30 21:03:45.065782 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:45.065745    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb318efb-e29b-49fa-98b0-7ecf8cb93531" path="/var/lib/kubelet/pods/bb318efb-e29b-49fa-98b0-7ecf8cb93531/volumes"
Oct 30 21:03:45.066421 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:45.066403    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d335bbaf-1e6b-49cb-8bb1-89d987d18a90" path="/var/lib/kubelet/pods/d335bbaf-1e6b-49cb-8bb1-89d987d18a90/volumes"
Oct 30 21:03:47.987931 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:47.987892    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2cf21005-55df98b7dc-h466s"]
Oct 30 21:03:47.988752 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:47.988730    5713 cpu_manager.go:414] "containerMap: removing container" podUID="bb318efb-e29b-49fa-98b0-7ecf8cb93531" containerName="payload-pre-processing"
Oct 30 21:03:47.988899 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:47.988753    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb318efb-e29b-49fa-98b0-7ecf8cb93531" containerName="payload-pre-processing"
Oct 30 21:03:47.988899 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:47.988780    5713 cpu_manager.go:414] "containerMap: removing container" podUID="d335bbaf-1e6b-49cb-8bb1-89d987d18a90" containerName="istio-proxy"
Oct 30 21:03:47.988899 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:47.988812    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d335bbaf-1e6b-49cb-8bb1-89d987d18a90" containerName="istio-proxy"
Oct 30 21:03:47.988899 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:47.988889    5713 cpu_manager.go:414] "containerMap: removing container" podUID="664ca418-ae7c-4106-8bfb-43e1255bb087" containerName="revoke-keys"
Oct 30 21:03:47.988899 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:47.988896    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="664ca418-ae7c-4106-8bfb-43e1255bb087" containerName="revoke-keys"
Oct 30 21:03:47.989148 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:47.989094    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="664ca418-ae7c-4106-8bfb-43e1255bb087" containerName="revoke-keys"
Oct 30 21:03:47.989148 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:47.989112    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb318efb-e29b-49fa-98b0-7ecf8cb93531" containerName="payload-pre-processing"
Oct 30 21:03:47.989148 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:47.989126    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="d335bbaf-1e6b-49cb-8bb1-89d987d18a90" containerName="istio-proxy"
Oct 30 21:03:47.989148 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:47.989137    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="d335bbaf-1e6b-49cb-8bb1-89d987d18a90" containerName="istio-proxy"
Oct 30 21:03:47.993064 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:47.993042    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2cf21005-55df98b7dc-h466s"
Oct 30 21:03:48.050692 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:48.050646    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2cf21005-55df98b7dc-h466s"]
Oct 30 21:03:48.085821 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:48.081230    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65hck\" (UniqueName: \"kubernetes.io/projected/bdc39409-a6df-4bed-8017-a6f502bb7b44-kube-api-access-65hck\") pod \"payload-pre-processing-e2e-ait-2cf21005-55df98b7dc-h466s\" (UID: \"bdc39409-a6df-4bed-8017-a6f502bb7b44\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2cf21005-55df98b7dc-h466s"
Oct 30 21:03:48.085821 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:48.081453    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdc39409-a6df-4bed-8017-a6f502bb7b44-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2cf21005-55df98b7dc-h466s\" (UID: \"bdc39409-a6df-4bed-8017-a6f502bb7b44\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2cf21005-55df98b7dc-h466s"
Oct 30 21:03:48.109583 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:48.107204    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-2cf21005\""
Oct 30 21:03:48.109583 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:48.107364    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-2cf21005-dockercfg-4zk88\""
Oct 30 21:03:48.152267 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:48.152232    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2cf21005-548d5fccfb-kzfwl"]
Oct 30 21:03:48.153372 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:48.153347    5713 cpu_manager.go:414] "containerMap: removing container" podUID="d335bbaf-1e6b-49cb-8bb1-89d987d18a90" containerName="istio-proxy"
Oct 30 21:03:48.153497 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:48.153383    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d335bbaf-1e6b-49cb-8bb1-89d987d18a90" containerName="istio-proxy"
Oct 30 21:03:48.157987 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:48.157949    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2cf21005-548d5fccfb-kzfwl"
Oct 30 21:03:48.210813 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:48.199129    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65hck\" (UniqueName: \"kubernetes.io/projected/bdc39409-a6df-4bed-8017-a6f502bb7b44-kube-api-access-65hck\") pod \"payload-pre-processing-e2e-ait-2cf21005-55df98b7dc-h466s\" (UID: \"bdc39409-a6df-4bed-8017-a6f502bb7b44\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2cf21005-55df98b7dc-h466s"
Oct 30 21:03:48.210813 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:48.199369    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdc39409-a6df-4bed-8017-a6f502bb7b44-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2cf21005-55df98b7dc-h466s\" (UID: \"bdc39409-a6df-4bed-8017-a6f502bb7b44\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2cf21005-55df98b7dc-h466s"
Oct 30 21:03:48.242922 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:48.241739    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdc39409-a6df-4bed-8017-a6f502bb7b44-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2cf21005-55df98b7dc-h466s\" (UID: \"bdc39409-a6df-4bed-8017-a6f502bb7b44\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2cf21005-55df98b7dc-h466s"
Oct 30 21:03:48.292201 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:48.292133    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65hck\" (UniqueName: \"kubernetes.io/projected/bdc39409-a6df-4bed-8017-a6f502bb7b44-kube-api-access-65hck\") pod \"payload-pre-processing-e2e-ait-2cf21005-55df98b7dc-h466s\" (UID: \"bdc39409-a6df-4bed-8017-a6f502bb7b44\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2cf21005-55df98b7dc-h466s"
Oct 30 21:03:48.295805 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:48.295756    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2cf21005-548d5fccfb-kzfwl"]
Oct 30 21:03:48.301447 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:48.301416    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/427927da-96f1-464c-9a58-1116d61bee68-plugins-config-volume\") pod \"payload-processing-e2e-ait-2cf21005-548d5fccfb-kzfwl\" (UID: \"427927da-96f1-464c-9a58-1116d61bee68\") " pod="openshift-ingress/payload-processing-e2e-ait-2cf21005-548d5fccfb-kzfwl"
Oct 30 21:03:48.301659 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:48.301642    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd6k4\" (UniqueName: \"kubernetes.io/projected/427927da-96f1-464c-9a58-1116d61bee68-kube-api-access-fd6k4\") pod \"payload-processing-e2e-ait-2cf21005-548d5fccfb-kzfwl\" (UID: \"427927da-96f1-464c-9a58-1116d61bee68\") " pod="openshift-ingress/payload-processing-e2e-ait-2cf21005-548d5fccfb-kzfwl"
Oct 30 21:03:48.315423 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:48.315397    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2cf21005-55df98b7dc-h466s"
Oct 30 21:03:48.405062 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:48.405033    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/427927da-96f1-464c-9a58-1116d61bee68-plugins-config-volume\") pod \"payload-processing-e2e-ait-2cf21005-548d5fccfb-kzfwl\" (UID: \"427927da-96f1-464c-9a58-1116d61bee68\") " pod="openshift-ingress/payload-processing-e2e-ait-2cf21005-548d5fccfb-kzfwl"
Oct 30 21:03:48.405175 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:48.405072    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd6k4\" (UniqueName: \"kubernetes.io/projected/427927da-96f1-464c-9a58-1116d61bee68-kube-api-access-fd6k4\") pod \"payload-processing-e2e-ait-2cf21005-548d5fccfb-kzfwl\" (UID: \"427927da-96f1-464c-9a58-1116d61bee68\") " pod="openshift-ingress/payload-processing-e2e-ait-2cf21005-548d5fccfb-kzfwl"
Oct 30 21:03:48.406198 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:48.406155    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/427927da-96f1-464c-9a58-1116d61bee68-plugins-config-volume\") pod \"payload-processing-e2e-ait-2cf21005-548d5fccfb-kzfwl\" (UID: \"427927da-96f1-464c-9a58-1116d61bee68\") " pod="openshift-ingress/payload-processing-e2e-ait-2cf21005-548d5fccfb-kzfwl"
Oct 30 21:03:48.464892 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:48.464860    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd6k4\" (UniqueName: \"kubernetes.io/projected/427927da-96f1-464c-9a58-1116d61bee68-kube-api-access-fd6k4\") pod \"payload-processing-e2e-ait-2cf21005-548d5fccfb-kzfwl\" (UID: \"427927da-96f1-464c-9a58-1116d61bee68\") " pod="openshift-ingress/payload-processing-e2e-ait-2cf21005-548d5fccfb-kzfwl"
Oct 30 21:03:48.591689 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:48.591603    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2cf21005-548d5fccfb-kzfwl"
Oct 30 21:03:48.727828 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:03:48.727772    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdc39409_a6df_4bed_8017_a6f502bb7b44.slice/crio-1d97e9d169e39adf96109033c17843b04f64e356d3712b36ab0c9b2672955663 WatchSource:0}: Error finding container 1d97e9d169e39adf96109033c17843b04f64e356d3712b36ab0c9b2672955663: Status 404 returned error can't find the container with id 1d97e9d169e39adf96109033c17843b04f64e356d3712b36ab0c9b2672955663
Oct 30 21:03:48.728537 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:48.728512    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2cf21005-55df98b7dc-h466s"]
Oct 30 21:03:48.973079 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:03:48.973034    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod427927da_96f1_464c_9a58_1116d61bee68.slice/crio-cb3edbd7c4de3c9192509e0d1a200d479a058681f1d058ff4d4ac79e177c58ee WatchSource:0}: Error finding container cb3edbd7c4de3c9192509e0d1a200d479a058681f1d058ff4d4ac79e177c58ee: Status 404 returned error can't find the container with id cb3edbd7c4de3c9192509e0d1a200d479a058681f1d058ff4d4ac79e177c58ee
Oct 30 21:03:48.974125 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:48.974101    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2cf21005-548d5fccfb-kzfwl"]
Oct 30 21:03:49.379572 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:49.379472    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2cf21005-55df98b7dc-h466s" event={"ID":"bdc39409-a6df-4bed-8017-a6f502bb7b44","Type":"ContainerStarted","Data":"a179c70d401dc1bbd0a6d76798061d5d26c509f32d643f4e83035290ac0af0ed"}
Oct 30 21:03:49.379572 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:49.379537    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2cf21005-55df98b7dc-h466s" event={"ID":"bdc39409-a6df-4bed-8017-a6f502bb7b44","Type":"ContainerStarted","Data":"1d97e9d169e39adf96109033c17843b04f64e356d3712b36ab0c9b2672955663"}
Oct 30 21:03:49.380210 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:49.379582    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-2cf21005-55df98b7dc-h466s"
Oct 30 21:03:49.381407 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:49.381380    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2cf21005-548d5fccfb-kzfwl" event={"ID":"427927da-96f1-464c-9a58-1116d61bee68","Type":"ContainerStarted","Data":"cd3ef5c457b8ec624cad2c4be51f2531bd02bf77f1cdff4439ef9b03453e0cb2"}
Oct 30 21:03:49.381487 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:49.381427    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2cf21005-548d5fccfb-kzfwl" event={"ID":"427927da-96f1-464c-9a58-1116d61bee68","Type":"ContainerStarted","Data":"cb3edbd7c4de3c9192509e0d1a200d479a058681f1d058ff4d4ac79e177c58ee"}
Oct 30 21:03:49.381487 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:49.381451    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-2cf21005-548d5fccfb-kzfwl"
Oct 30 21:03:49.496747 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:49.496687    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-2cf21005-55df98b7dc-h466s" podStartSLOduration=2.496673693 podStartE2EDuration="2.496673693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:03:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:03:49.494983036 +0000 UTC m=+1512.559759438" watchObservedRunningTime="2026-10-30 21:03:49.496673693 +0000 UTC m=+1512.561450091"
Oct 30 21:03:49.619339 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:49.619274    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-2cf21005-548d5fccfb-kzfwl" podStartSLOduration=1.6192530550000002 podStartE2EDuration="1.619253055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:03:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:03:49.609156304 +0000 UTC m=+1512.673932703" watchObservedRunningTime="2026-10-30 21:03:49.619253055 +0000 UTC m=+1512.684029454"
Oct 30 21:03:53.069867 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:53.069838    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb31ab64-57b5l"]
Oct 30 21:03:53.124888 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:53.123640    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb31ab64-57b5l"]
Oct 30 21:03:53.227698 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:53.227661    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-bf9c0727-8d8698f4d-hjmdn"]
Oct 30 21:03:53.231769 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:53.231744    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-bf9c0727-8d8698f4d-hjmdn"
Oct 30 21:03:53.284241 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:53.284210    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-bf9c0727\""
Oct 30 21:03:53.284241 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:53.284210    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-bf9c0727-dockercfg-wz5r9\""
Oct 30 21:03:53.288759 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:53.288733    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-bf9c0727-8d8698f4d-hjmdn"]
Oct 30 21:03:53.398821 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:53.398704    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5478d17-b619-43ac-8f68-2675932b4abe-plugins-config-volume\") pod \"payload-processing-e2e-derive-bf9c0727-8d8698f4d-hjmdn\" (UID: \"f5478d17-b619-43ac-8f68-2675932b4abe\") " pod="openshift-ingress/payload-processing-e2e-derive-bf9c0727-8d8698f4d-hjmdn"
Oct 30 21:03:53.399014 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:53.398849    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjlj8\" (UniqueName: \"kubernetes.io/projected/f5478d17-b619-43ac-8f68-2675932b4abe-kube-api-access-fjlj8\") pod \"payload-processing-e2e-derive-bf9c0727-8d8698f4d-hjmdn\" (UID: \"f5478d17-b619-43ac-8f68-2675932b4abe\") " pod="openshift-ingress/payload-processing-e2e-derive-bf9c0727-8d8698f4d-hjmdn"
Oct 30 21:03:53.500823 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:53.500745    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5478d17-b619-43ac-8f68-2675932b4abe-plugins-config-volume\") pod \"payload-processing-e2e-derive-bf9c0727-8d8698f4d-hjmdn\" (UID: \"f5478d17-b619-43ac-8f68-2675932b4abe\") " pod="openshift-ingress/payload-processing-e2e-derive-bf9c0727-8d8698f4d-hjmdn"
Oct 30 21:03:53.501023 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:53.500894    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjlj8\" (UniqueName: \"kubernetes.io/projected/f5478d17-b619-43ac-8f68-2675932b4abe-kube-api-access-fjlj8\") pod \"payload-processing-e2e-derive-bf9c0727-8d8698f4d-hjmdn\" (UID: \"f5478d17-b619-43ac-8f68-2675932b4abe\") " pod="openshift-ingress/payload-processing-e2e-derive-bf9c0727-8d8698f4d-hjmdn"
Oct 30 21:03:53.501661 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:53.501633    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5478d17-b619-43ac-8f68-2675932b4abe-plugins-config-volume\") pod \"payload-processing-e2e-derive-bf9c0727-8d8698f4d-hjmdn\" (UID: \"f5478d17-b619-43ac-8f68-2675932b4abe\") " pod="openshift-ingress/payload-processing-e2e-derive-bf9c0727-8d8698f4d-hjmdn"
Oct 30 21:03:53.558999 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:53.558971    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjlj8\" (UniqueName: \"kubernetes.io/projected/f5478d17-b619-43ac-8f68-2675932b4abe-kube-api-access-fjlj8\") pod \"payload-processing-e2e-derive-bf9c0727-8d8698f4d-hjmdn\" (UID: \"f5478d17-b619-43ac-8f68-2675932b4abe\") " pod="openshift-ingress/payload-processing-e2e-derive-bf9c0727-8d8698f4d-hjmdn"
Oct 30 21:03:53.847258 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:53.847229    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-bf9c0727-8d8698f4d-hjmdn"
Oct 30 21:03:54.226910 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:03:54.226877    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5478d17_b619_43ac_8f68_2675932b4abe.slice/crio-c347c5721e9c4b1fba7c6e8ca8306b359036ffc1b714bf8af80d6fc5469981c0 WatchSource:0}: Error finding container c347c5721e9c4b1fba7c6e8ca8306b359036ffc1b714bf8af80d6fc5469981c0: Status 404 returned error can't find the container with id c347c5721e9c4b1fba7c6e8ca8306b359036ffc1b714bf8af80d6fc5469981c0
Oct 30 21:03:54.229498 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:54.229474    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-bf9c0727-8d8698f4d-hjmdn"]
Oct 30 21:03:54.432444 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:54.432396    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-bf9c0727-8d8698f4d-hjmdn" event={"ID":"f5478d17-b619-43ac-8f68-2675932b4abe","Type":"ContainerStarted","Data":"905415ba78668542403721985f468689cdc2ab6d86bcf9767bafe371996f364b"}
Oct 30 21:03:54.432965 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:54.432460    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-bf9c0727-8d8698f4d-hjmdn"
Oct 30 21:03:54.432965 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:54.432484    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-bf9c0727-8d8698f4d-hjmdn" event={"ID":"f5478d17-b619-43ac-8f68-2675932b4abe","Type":"ContainerStarted","Data":"c347c5721e9c4b1fba7c6e8ca8306b359036ffc1b714bf8af80d6fc5469981c0"}
Oct 30 21:03:54.459277 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:03:54.459246    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:03:54.550153 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:54.550040    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-bf9c0727-8d8698f4d-hjmdn" podStartSLOduration=1.550025647 podStartE2EDuration="1.550025647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:03:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:03:54.546084644 +0000 UTC m=+1517.610861055" watchObservedRunningTime="2026-10-30 21:03:54.550025647 +0000 UTC m=+1517.614802045"
Oct 30 21:03:55.065718 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:03:55.065680    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b4cefda-3fd3-4394-aeac-e6594ce167a0" path="/var/lib/kubelet/pods/7b4cefda-3fd3-4394-aeac-e6594ce167a0/volumes"
Oct 30 21:04:00.392445 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:00.392418    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-2cf21005-55df98b7dc-h466s"
Oct 30 21:04:00.392989 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:00.392503    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-2cf21005-548d5fccfb-kzfwl"
Oct 30 21:04:00.897091 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:00.897054    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2e56076-87sh8"]
Oct 30 21:04:00.900749 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:00.900730    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2e56076-87sh8"
Oct 30 21:04:00.953616 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:00.953522    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:04:00.953616 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:00.953531    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:04:00.953616 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:00.953564    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:04:00.953616 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:00.953580    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:04:00.958327 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:00.958301    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2e56076-87sh8"]
Oct 30 21:04:01.025959 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:01.025919    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6124e97b-a754-4495-a0d9-61b63533a579-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2e56076-87sh8\" (UID: \"6124e97b-a754-4495-a0d9-61b63533a579\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2e56076-87sh8"
Oct 30 21:04:01.026157 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:01.026060    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdktd\" (UniqueName: \"kubernetes.io/projected/6124e97b-a754-4495-a0d9-61b63533a579-kube-api-access-cdktd\") pod \"maas-api-revoke-sub-b2e56076-87sh8\" (UID: \"6124e97b-a754-4495-a0d9-61b63533a579\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2e56076-87sh8"
Oct 30 21:04:01.026157 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:01.026094    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6124e97b-a754-4495-a0d9-61b63533a579-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2e56076-87sh8\" (UID: \"6124e97b-a754-4495-a0d9-61b63533a579\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2e56076-87sh8"
Oct 30 21:04:01.127252 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:01.127219    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6124e97b-a754-4495-a0d9-61b63533a579-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2e56076-87sh8\" (UID: \"6124e97b-a754-4495-a0d9-61b63533a579\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2e56076-87sh8"
Oct 30 21:04:01.127453 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:01.127343    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdktd\" (UniqueName: \"kubernetes.io/projected/6124e97b-a754-4495-a0d9-61b63533a579-kube-api-access-cdktd\") pod \"maas-api-revoke-sub-b2e56076-87sh8\" (UID: \"6124e97b-a754-4495-a0d9-61b63533a579\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2e56076-87sh8"
Oct 30 21:04:01.127453 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:01.127394    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6124e97b-a754-4495-a0d9-61b63533a579-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2e56076-87sh8\" (UID: \"6124e97b-a754-4495-a0d9-61b63533a579\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2e56076-87sh8"
Oct 30 21:04:01.128010 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:01.127986    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6124e97b-a754-4495-a0d9-61b63533a579-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2e56076-87sh8\" (UID: \"6124e97b-a754-4495-a0d9-61b63533a579\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2e56076-87sh8"
Oct 30 21:04:01.130122 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:01.130099    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6124e97b-a754-4495-a0d9-61b63533a579-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2e56076-87sh8\" (UID: \"6124e97b-a754-4495-a0d9-61b63533a579\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2e56076-87sh8"
Oct 30 21:04:01.184399 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:01.184373    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdktd\" (UniqueName: \"kubernetes.io/projected/6124e97b-a754-4495-a0d9-61b63533a579-kube-api-access-cdktd\") pod \"maas-api-revoke-sub-b2e56076-87sh8\" (UID: \"6124e97b-a754-4495-a0d9-61b63533a579\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2e56076-87sh8"
Oct 30 21:04:01.214147 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:01.214100    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2e56076-87sh8"
Oct 30 21:04:01.595039 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:04:01.595006    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6124e97b_a754_4495_a0d9_61b63533a579.slice/crio-6421f69ca5f210df267b507c3fcafc5d1dfe9e86afcd651a3c327a27a4d9ccb9 WatchSource:0}: Error finding container 6421f69ca5f210df267b507c3fcafc5d1dfe9e86afcd651a3c327a27a4d9ccb9: Status 404 returned error can't find the container with id 6421f69ca5f210df267b507c3fcafc5d1dfe9e86afcd651a3c327a27a4d9ccb9
Oct 30 21:04:01.596302 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:01.596272    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2e56076-87sh8"]
Oct 30 21:04:02.506946 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:02.506905    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2e56076-87sh8" event={"ID":"6124e97b-a754-4495-a0d9-61b63533a579","Type":"ContainerStarted","Data":"240ed7d821f503693218f530d2cdfe559f91b26fb3ffbe8a59394e387156a040"}
Oct 30 21:04:02.507147 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:02.506965    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2e56076-87sh8" event={"ID":"6124e97b-a754-4495-a0d9-61b63533a579","Type":"ContainerStarted","Data":"6421f69ca5f210df267b507c3fcafc5d1dfe9e86afcd651a3c327a27a4d9ccb9"}
Oct 30 21:04:02.622976 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:02.622930    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2e56076-87sh8" podStartSLOduration=2.62291661 podStartE2EDuration="2.62291661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:04:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:04:02.620026458 +0000 UTC m=+1525.684802858" watchObservedRunningTime="2026-10-30 21:04:02.62291661 +0000 UTC m=+1525.687693007"
Oct 30 21:04:03.517136 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:03.517101    5713 generic.go:356] "Generic (PLEG): container finished" podID="6124e97b-a754-4495-a0d9-61b63533a579" containerID="240ed7d821f503693218f530d2cdfe559f91b26fb3ffbe8a59394e387156a040" exitCode=0
Oct 30 21:04:03.517306 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:03.517190    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2e56076-87sh8" event={"ID":"6124e97b-a754-4495-a0d9-61b63533a579","Type":"ContainerDied","Data":"240ed7d821f503693218f530d2cdfe559f91b26fb3ffbe8a59394e387156a040"}
Oct 30 21:04:04.048704 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:04.048670    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ce05cc9-dtkj5"]
Oct 30 21:04:04.053442 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:04.053419    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ce05cc9-dtkj5"]
Oct 30 21:04:04.708336 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:04.708300    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2e56076-87sh8"
Oct 30 21:04:04.791930 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:04.791840    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6124e97b-a754-4495-a0d9-61b63533a579-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6124e97b-a754-4495-a0d9-61b63533a579-maas-api-openshift-ca\") pod \"6124e97b-a754-4495-a0d9-61b63533a579\" (UID: \"6124e97b-a754-4495-a0d9-61b63533a579\") "
Oct 30 21:04:04.792121 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:04.791964    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6124e97b-a754-4495-a0d9-61b63533a579-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6124e97b-a754-4495-a0d9-61b63533a579-maas-api-xks-ca\") pod \"6124e97b-a754-4495-a0d9-61b63533a579\" (UID: \"6124e97b-a754-4495-a0d9-61b63533a579\") "
Oct 30 21:04:04.792121 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:04.792035    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6124e97b-a754-4495-a0d9-61b63533a579-kube-api-access-cdktd\" (UniqueName: \"kubernetes.io/projected/6124e97b-a754-4495-a0d9-61b63533a579-kube-api-access-cdktd\") pod \"6124e97b-a754-4495-a0d9-61b63533a579\" (UID: \"6124e97b-a754-4495-a0d9-61b63533a579\") "
Oct 30 21:04:04.792255 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:04.792190    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6124e97b-a754-4495-a0d9-61b63533a579-maas-api-openshift-ca" pod "6124e97b-a754-4495-a0d9-61b63533a579" (UID: "6124e97b-a754-4495-a0d9-61b63533a579"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:04:04.792841 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:04.792730    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6124e97b-a754-4495-a0d9-61b63533a579-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:04.794280 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:04.794254    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6124e97b-a754-4495-a0d9-61b63533a579-maas-api-xks-ca" pod "6124e97b-a754-4495-a0d9-61b63533a579" (UID: "6124e97b-a754-4495-a0d9-61b63533a579"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:04:04.794434 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:04.794415    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6124e97b-a754-4495-a0d9-61b63533a579-kube-api-access-cdktd" pod "6124e97b-a754-4495-a0d9-61b63533a579" (UID: "6124e97b-a754-4495-a0d9-61b63533a579"). InnerVolumeSpecName "kube-api-access-cdktd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:04:04.894425 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:04.894341    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6124e97b-a754-4495-a0d9-61b63533a579-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:04.895811 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:04.895764    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdktd\" (UniqueName: \"kubernetes.io/projected/6124e97b-a754-4495-a0d9-61b63533a579-kube-api-access-cdktd\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:05.067086 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:05.066995    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81c89aa9-2696-4e30-aa16-a4af8c4235c5" path="/var/lib/kubelet/pods/81c89aa9-2696-4e30-aa16-a4af8c4235c5/volumes"
Oct 30 21:04:05.481812 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:05.472765    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-bf9c0727-8d8698f4d-hjmdn"
Oct 30 21:04:05.587778 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:05.587739    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2e56076-87sh8" event={"ID":"6124e97b-a754-4495-a0d9-61b63533a579","Type":"ContainerDied","Data":"6421f69ca5f210df267b507c3fcafc5d1dfe9e86afcd651a3c327a27a4d9ccb9"}
Oct 30 21:04:05.588217 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:05.588096    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2e56076-87sh8"
Oct 30 21:04:05.588481 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:05.588200    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6421f69ca5f210df267b507c3fcafc5d1dfe9e86afcd651a3c327a27a4d9ccb9"
Oct 30 21:04:05.767020 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:05.766913    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jg6f5"]
Oct 30 21:04:05.768159 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:05.768133    5713 cpu_manager.go:414] "containerMap: removing container" podUID="6124e97b-a754-4495-a0d9-61b63533a579" containerName="revoke-keys"
Oct 30 21:04:05.768159 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:05.768158    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6124e97b-a754-4495-a0d9-61b63533a579" containerName="revoke-keys"
Oct 30 21:04:05.768395 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:05.768372    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="6124e97b-a754-4495-a0d9-61b63533a579" containerName="revoke-keys"
Oct 30 21:04:05.771874 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:05.771854    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jg6f5"
Oct 30 21:04:05.823335 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:05.823299    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jg6f5"]
Oct 30 21:04:05.824006 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:05.823975    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 21:04:05.824144 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:05.824012    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 21:04:05.824144 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:05.824015    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 21:04:05.824248 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:05.824156    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-pwj8h\""
Oct 30 21:04:05.915333 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:05.915298    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76z6f\" (UniqueName: \"kubernetes.io/projected/4271c173-e39b-4207-b710-e2298c3f7b11-kube-api-access-76z6f\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jg6f5\" (UID: \"4271c173-e39b-4207-b710-e2298c3f7b11\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jg6f5"
Oct 30 21:04:05.915501 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:05.915345    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4271c173-e39b-4207-b710-e2298c3f7b11-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jg6f5\" (UID: \"4271c173-e39b-4207-b710-e2298c3f7b11\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jg6f5"
Oct 30 21:04:05.915501 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:05.915454    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4271c173-e39b-4207-b710-e2298c3f7b11-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jg6f5\" (UID: \"4271c173-e39b-4207-b710-e2298c3f7b11\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jg6f5"
Oct 30 21:04:05.915629 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:05.915581    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4271c173-e39b-4207-b710-e2298c3f7b11-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jg6f5\" (UID: \"4271c173-e39b-4207-b710-e2298c3f7b11\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jg6f5"
Oct 30 21:04:05.915744 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:05.915727    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4271c173-e39b-4207-b710-e2298c3f7b11-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jg6f5\" (UID: \"4271c173-e39b-4207-b710-e2298c3f7b11\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jg6f5"
Oct 30 21:04:05.915835 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:05.915822    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4271c173-e39b-4207-b710-e2298c3f7b11-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jg6f5\" (UID: \"4271c173-e39b-4207-b710-e2298c3f7b11\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jg6f5"
Oct 30 21:04:05.915906 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:05.915858    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4271c173-e39b-4207-b710-e2298c3f7b11-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jg6f5\" (UID: \"4271c173-e39b-4207-b710-e2298c3f7b11\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jg6f5"
Oct 30 21:04:06.017683 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:06.017605    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4271c173-e39b-4207-b710-e2298c3f7b11-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jg6f5\" (UID: \"4271c173-e39b-4207-b710-e2298c3f7b11\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jg6f5"
Oct 30 21:04:06.017683 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:06.017650    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4271c173-e39b-4207-b710-e2298c3f7b11-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jg6f5\" (UID: \"4271c173-e39b-4207-b710-e2298c3f7b11\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jg6f5"
Oct 30 21:04:06.018008 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:06.017729    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76z6f\" (UniqueName: \"kubernetes.io/projected/4271c173-e39b-4207-b710-e2298c3f7b11-kube-api-access-76z6f\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jg6f5\" (UID: \"4271c173-e39b-4207-b710-e2298c3f7b11\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jg6f5"
Oct 30 21:04:06.018008 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:06.017756    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4271c173-e39b-4207-b710-e2298c3f7b11-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jg6f5\" (UID: \"4271c173-e39b-4207-b710-e2298c3f7b11\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jg6f5"
Oct 30 21:04:06.018008 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:06.017843    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4271c173-e39b-4207-b710-e2298c3f7b11-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jg6f5\" (UID: \"4271c173-e39b-4207-b710-e2298c3f7b11\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jg6f5"
Oct 30 21:04:06.018008 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:06.017941    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4271c173-e39b-4207-b710-e2298c3f7b11-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jg6f5\" (UID: \"4271c173-e39b-4207-b710-e2298c3f7b11\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jg6f5"
Oct 30 21:04:06.018213 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:06.018085    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4271c173-e39b-4207-b710-e2298c3f7b11-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jg6f5\" (UID: \"4271c173-e39b-4207-b710-e2298c3f7b11\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jg6f5"
Oct 30 21:04:06.018627 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:06.018571    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4271c173-e39b-4207-b710-e2298c3f7b11-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jg6f5\" (UID: \"4271c173-e39b-4207-b710-e2298c3f7b11\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jg6f5"
Oct 30 21:04:06.020362 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:06.020339    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4271c173-e39b-4207-b710-e2298c3f7b11-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jg6f5\" (UID: \"4271c173-e39b-4207-b710-e2298c3f7b11\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jg6f5"
Oct 30 21:04:06.020466 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:06.020412    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4271c173-e39b-4207-b710-e2298c3f7b11-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jg6f5\" (UID: \"4271c173-e39b-4207-b710-e2298c3f7b11\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jg6f5"
Oct 30 21:04:06.020764 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:06.020739    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4271c173-e39b-4207-b710-e2298c3f7b11-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jg6f5\" (UID: \"4271c173-e39b-4207-b710-e2298c3f7b11\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jg6f5"
Oct 30 21:04:06.020871 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:06.020764    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4271c173-e39b-4207-b710-e2298c3f7b11-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jg6f5\" (UID: \"4271c173-e39b-4207-b710-e2298c3f7b11\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jg6f5"
Oct 30 21:04:06.075971 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:06.075946    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4271c173-e39b-4207-b710-e2298c3f7b11-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jg6f5\" (UID: \"4271c173-e39b-4207-b710-e2298c3f7b11\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jg6f5"
Oct 30 21:04:06.076445 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:06.075945    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76z6f\" (UniqueName: \"kubernetes.io/projected/4271c173-e39b-4207-b710-e2298c3f7b11-kube-api-access-76z6f\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jg6f5\" (UID: \"4271c173-e39b-4207-b710-e2298c3f7b11\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jg6f5"
Oct 30 21:04:06.086183 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:06.086158    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jg6f5"
Oct 30 21:04:06.528844 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:04:06.528767    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4271c173_e39b_4207_b710_e2298c3f7b11.slice/crio-35c1fcde4270eece5f8b5eb53af80d881b768c31c88668d00ec47ea6f2318d82 WatchSource:0}: Error finding container 35c1fcde4270eece5f8b5eb53af80d881b768c31c88668d00ec47ea6f2318d82: Status 404 returned error can't find the container with id 35c1fcde4270eece5f8b5eb53af80d881b768c31c88668d00ec47ea6f2318d82
Oct 30 21:04:06.530103 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:06.530074    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jg6f5"]
Oct 30 21:04:06.531581 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:06.531566    5713 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:04:06.598476 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:06.598439    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jg6f5" event={"ID":"4271c173-e39b-4207-b710-e2298c3f7b11","Type":"ContainerStarted","Data":"35c1fcde4270eece5f8b5eb53af80d881b768c31c88668d00ec47ea6f2318d82"}
Oct 30 21:04:06.772997 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:06.772968    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc88b34e-mh9pn"]
Oct 30 21:04:06.777507 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:06.777451    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bc88b34e-mh9pn"]
Oct 30 21:04:07.065086 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:07.065018    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="664ca418-ae7c-4106-8bfb-43e1255bb087" path="/var/lib/kubelet/pods/664ca418-ae7c-4106-8bfb-43e1255bb087/volumes"
Oct 30 21:04:08.275149 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:08.275118    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj"]
Oct 30 21:04:08.275623 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:08.275349    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj" podUID="59246922-40fa-40ad-a84b-7604b077ef92" containerName="istio-proxy" containerID="cri-o://3a2fcbaa407c72cdbd80eeb60a83c8b9e3a0a20f2ff6a90294f077bf5720a6c9" gracePeriod=30
Oct 30 21:04:11.001662 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:11.001630    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:04:11.002112 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:11.001696    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:04:11.648404 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:11.648364    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jg6f5" event={"ID":"4271c173-e39b-4207-b710-e2298c3f7b11","Type":"ContainerStarted","Data":"fba8c41dcd2043058ffbd6f39f88270988bb445ce37710b4caab64de94b61e08"}
Oct 30 21:04:11.648603 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:11.648465    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jg6f5"
Oct 30 21:04:12.659561 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:12.659527    5713 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-jg6f5 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:04:12.659964 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:12.659589    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jg6f5" podUID="4271c173-e39b-4207-b710-e2298c3f7b11" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:04:13.552166 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:13.552086    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj"
Oct 30 21:04:13.668385 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:13.668351    5713 generic.go:356] "Generic (PLEG): container finished" podID="59246922-40fa-40ad-a84b-7604b077ef92" containerID="3a2fcbaa407c72cdbd80eeb60a83c8b9e3a0a20f2ff6a90294f077bf5720a6c9" exitCode=0
Oct 30 21:04:13.668924 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:13.668396    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj"
Oct 30 21:04:13.668924 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:13.668416    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59246922-40fa-40ad-a84b-7604b077ef92-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59246922-40fa-40ad-a84b-7604b077ef92-istio-data\") pod \"59246922-40fa-40ad-a84b-7604b077ef92\" (UID: \"59246922-40fa-40ad-a84b-7604b077ef92\") "
Oct 30 21:04:13.668924 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:13.668437    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj" event={"ID":"59246922-40fa-40ad-a84b-7604b077ef92","Type":"ContainerDied","Data":"3a2fcbaa407c72cdbd80eeb60a83c8b9e3a0a20f2ff6a90294f077bf5720a6c9"}
Oct 30 21:04:13.668924 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:13.668504    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj" event={"ID":"59246922-40fa-40ad-a84b-7604b077ef92","Type":"ContainerDied","Data":"15fe8edaee7b5e776ca9a6fdaff4a28806c21ee80902259d9b858a2ceb1bdf62"}
Oct 30 21:04:13.668924 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:13.668537    5713 scope.go:122] "RemoveContainer" containerID="3a2fcbaa407c72cdbd80eeb60a83c8b9e3a0a20f2ff6a90294f077bf5720a6c9"
Oct 30 21:04:13.668924 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:13.668538    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59246922-40fa-40ad-a84b-7604b077ef92-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59246922-40fa-40ad-a84b-7604b077ef92-workload-socket\") pod \"59246922-40fa-40ad-a84b-7604b077ef92\" (UID: \"59246922-40fa-40ad-a84b-7604b077ef92\") "
Oct 30 21:04:13.668924 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:13.668666    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59246922-40fa-40ad-a84b-7604b077ef92-istio-data" pod "59246922-40fa-40ad-a84b-7604b077ef92" (UID: "59246922-40fa-40ad-a84b-7604b077ef92"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:04:13.668924 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:13.668703    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59246922-40fa-40ad-a84b-7604b077ef92-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59246922-40fa-40ad-a84b-7604b077ef92-istio-envoy\") pod \"59246922-40fa-40ad-a84b-7604b077ef92\" (UID: \"59246922-40fa-40ad-a84b-7604b077ef92\") "
Oct 30 21:04:13.668924 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:13.668749    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59246922-40fa-40ad-a84b-7604b077ef92-workload-socket" pod "59246922-40fa-40ad-a84b-7604b077ef92" (UID: "59246922-40fa-40ad-a84b-7604b077ef92"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:04:13.668924 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:13.668840    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59246922-40fa-40ad-a84b-7604b077ef92-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59246922-40fa-40ad-a84b-7604b077ef92-credential-socket\") pod \"59246922-40fa-40ad-a84b-7604b077ef92\" (UID: \"59246922-40fa-40ad-a84b-7604b077ef92\") "
Oct 30 21:04:13.668924 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:13.668879    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/59246922-40fa-40ad-a84b-7604b077ef92-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59246922-40fa-40ad-a84b-7604b077ef92-istio-podinfo\") pod \"59246922-40fa-40ad-a84b-7604b077ef92\" (UID: \"59246922-40fa-40ad-a84b-7604b077ef92\") "
Oct 30 21:04:13.668924 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:13.668905    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59246922-40fa-40ad-a84b-7604b077ef92-kube-api-access-j8bgd\" (UniqueName: \"kubernetes.io/projected/59246922-40fa-40ad-a84b-7604b077ef92-kube-api-access-j8bgd\") pod \"59246922-40fa-40ad-a84b-7604b077ef92\" (UID: \"59246922-40fa-40ad-a84b-7604b077ef92\") "
Oct 30 21:04:13.669579 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:13.668951    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59246922-40fa-40ad-a84b-7604b077ef92-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59246922-40fa-40ad-a84b-7604b077ef92-workload-certs\") pod \"59246922-40fa-40ad-a84b-7604b077ef92\" (UID: \"59246922-40fa-40ad-a84b-7604b077ef92\") "
Oct 30 21:04:13.669579 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:13.669037    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59246922-40fa-40ad-a84b-7604b077ef92-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59246922-40fa-40ad-a84b-7604b077ef92-istiod-ca-cert\") pod \"59246922-40fa-40ad-a84b-7604b077ef92\" (UID: \"59246922-40fa-40ad-a84b-7604b077ef92\") "
Oct 30 21:04:13.669579 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:13.669075    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59246922-40fa-40ad-a84b-7604b077ef92-istio-token\" (UniqueName: \"kubernetes.io/projected/59246922-40fa-40ad-a84b-7604b077ef92-istio-token\") pod \"59246922-40fa-40ad-a84b-7604b077ef92\" (UID: \"59246922-40fa-40ad-a84b-7604b077ef92\") "
Oct 30 21:04:13.669579 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:13.669110    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59246922-40fa-40ad-a84b-7604b077ef92-credential-socket" pod "59246922-40fa-40ad-a84b-7604b077ef92" (UID: "59246922-40fa-40ad-a84b-7604b077ef92"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:04:13.669579 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:13.669221    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59246922-40fa-40ad-a84b-7604b077ef92-workload-certs" pod "59246922-40fa-40ad-a84b-7604b077ef92" (UID: "59246922-40fa-40ad-a84b-7604b077ef92"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:04:13.669579 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:13.669361    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59246922-40fa-40ad-a84b-7604b077ef92-istiod-ca-cert" pod "59246922-40fa-40ad-a84b-7604b077ef92" (UID: "59246922-40fa-40ad-a84b-7604b077ef92"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:04:13.670133 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:13.670111    5713 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59246922-40fa-40ad-a84b-7604b077ef92-workload-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:13.670617 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:13.670138    5713 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59246922-40fa-40ad-a84b-7604b077ef92-credential-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:13.670617 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:13.670155    5713 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59246922-40fa-40ad-a84b-7604b077ef92-workload-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:13.670617 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:13.670168    5713 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59246922-40fa-40ad-a84b-7604b077ef92-istiod-ca-cert\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:13.670617 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:13.670181    5713 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59246922-40fa-40ad-a84b-7604b077ef92-istio-data\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:13.671582 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:13.671554    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59246922-40fa-40ad-a84b-7604b077ef92-istio-podinfo" pod "59246922-40fa-40ad-a84b-7604b077ef92" (UID: "59246922-40fa-40ad-a84b-7604b077ef92"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:04:13.671899 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:13.671874    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59246922-40fa-40ad-a84b-7604b077ef92-istio-token" pod "59246922-40fa-40ad-a84b-7604b077ef92" (UID: "59246922-40fa-40ad-a84b-7604b077ef92"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:04:13.672002 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:13.671903    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59246922-40fa-40ad-a84b-7604b077ef92-kube-api-access-j8bgd" pod "59246922-40fa-40ad-a84b-7604b077ef92" (UID: "59246922-40fa-40ad-a84b-7604b077ef92"). InnerVolumeSpecName "kube-api-access-j8bgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:04:13.672228 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:13.672210    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59246922-40fa-40ad-a84b-7604b077ef92-istio-envoy" pod "59246922-40fa-40ad-a84b-7604b077ef92" (UID: "59246922-40fa-40ad-a84b-7604b077ef92"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:04:13.704643 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:13.704622    5713 scope.go:122] "RemoveContainer" containerID="cb1db293e5e6f79d85e87828737f890a585cef853602139c637a94e501426ffa"
Oct 30 21:04:13.715657 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:13.715636    5713 scope.go:122] "RemoveContainer" containerID="3a2fcbaa407c72cdbd80eeb60a83c8b9e3a0a20f2ff6a90294f077bf5720a6c9"
Oct 30 21:04:13.715948 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:04:13.715932    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a2fcbaa407c72cdbd80eeb60a83c8b9e3a0a20f2ff6a90294f077bf5720a6c9\": container with ID starting with 3a2fcbaa407c72cdbd80eeb60a83c8b9e3a0a20f2ff6a90294f077bf5720a6c9 not found: ID does not exist" containerID="3a2fcbaa407c72cdbd80eeb60a83c8b9e3a0a20f2ff6a90294f077bf5720a6c9"
Oct 30 21:04:13.715987 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:13.715956    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a2fcbaa407c72cdbd80eeb60a83c8b9e3a0a20f2ff6a90294f077bf5720a6c9"} err="failed to get container status \"3a2fcbaa407c72cdbd80eeb60a83c8b9e3a0a20f2ff6a90294f077bf5720a6c9\": rpc error: code = NotFound desc = could not find container \"3a2fcbaa407c72cdbd80eeb60a83c8b9e3a0a20f2ff6a90294f077bf5720a6c9\": container with ID starting with 3a2fcbaa407c72cdbd80eeb60a83c8b9e3a0a20f2ff6a90294f077bf5720a6c9 not found: ID does not exist"
Oct 30 21:04:13.715987 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:13.715980    5713 scope.go:122] "RemoveContainer" containerID="cb1db293e5e6f79d85e87828737f890a585cef853602139c637a94e501426ffa"
Oct 30 21:04:13.716210 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:04:13.716192    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb1db293e5e6f79d85e87828737f890a585cef853602139c637a94e501426ffa\": container with ID starting with cb1db293e5e6f79d85e87828737f890a585cef853602139c637a94e501426ffa not found: ID does not exist" containerID="cb1db293e5e6f79d85e87828737f890a585cef853602139c637a94e501426ffa"
Oct 30 21:04:13.716246 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:13.716213    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb1db293e5e6f79d85e87828737f890a585cef853602139c637a94e501426ffa"} err="failed to get container status \"cb1db293e5e6f79d85e87828737f890a585cef853602139c637a94e501426ffa\": rpc error: code = NotFound desc = could not find container \"cb1db293e5e6f79d85e87828737f890a585cef853602139c637a94e501426ffa\": container with ID starting with cb1db293e5e6f79d85e87828737f890a585cef853602139c637a94e501426ffa not found: ID does not exist"
Oct 30 21:04:13.722459 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:13.722408    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jg6f5" podStartSLOduration=4.253917682 podStartE2EDuration="8.722393028s" totalImagesPullingTime="4.468475346s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:04:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:04:11.865532677 +0000 UTC m=+1534.930309086" watchObservedRunningTime="2026-10-30 21:04:13.722393028 +0000 UTC m=+1536.787169427"
Oct 30 21:04:13.772713 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:13.772675    5713 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59246922-40fa-40ad-a84b-7604b077ef92-istio-podinfo\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:13.772713 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:13.772714    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8bgd\" (UniqueName: \"kubernetes.io/projected/59246922-40fa-40ad-a84b-7604b077ef92-kube-api-access-j8bgd\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:13.772943 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:13.772729    5713 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59246922-40fa-40ad-a84b-7604b077ef92-istio-token\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:13.772943 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:13.772746    5713 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59246922-40fa-40ad-a84b-7604b077ef92-istio-envoy\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:04:14.099864 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:14.099815    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj"]
Oct 30 21:04:14.149224 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:14.149193    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-bc88b34e-openshift-default-7967f66c-jqpzj"]
Oct 30 21:04:15.064316 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:15.064281    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59246922-40fa-40ad-a84b-7604b077ef92" path="/var/lib/kubelet/pods/59246922-40fa-40ad-a84b-7604b077ef92/volumes"
Oct 30 21:04:15.659225 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:15.659189    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jg6f5"
Oct 30 21:04:23.047061 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:23.047029    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e61ebc76-d7p4p"]
Oct 30 21:04:23.096602 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:23.096572    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e61ebc76-d7p4p"]
Oct 30 21:04:24.460889 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:04:24.460850    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:04:25.064834 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:25.064783    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdc5287e-e62c-41b0-9ffe-c9f9d27f0cdd" path="/var/lib/kubelet/pods/fdc5287e-e62c-41b0-9ffe-c9f9d27f0cdd/volumes"
Oct 30 21:04:36.060253 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:36.060223    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9424c8f8-jqnc2"]
Oct 30 21:04:36.108136 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:36.108097    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9424c8f8-jqnc2"]
Oct 30 21:04:37.066014 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:37.065984    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bba59f9-e753-4e47-8c53-3e59fc35a780" path="/var/lib/kubelet/pods/4bba59f9-e753-4e47-8c53-3e59fc35a780/volumes"
Oct 30 21:04:37.687335 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:37.687302    5713 scope.go:122] "RemoveContainer" containerID="9bf01375c663cfbd6cd8ad936488452bcc18b49e6e426eaada02b733ca2e2c7f"
Oct 30 21:04:37.698767 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:37.698749    5713 scope.go:122] "RemoveContainer" containerID="f422395d38ac91efcb119f8113c34046dfd647edce573fa71089afa63a298c63"
Oct 30 21:04:37.710507 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:37.710487    5713 scope.go:122] "RemoveContainer" containerID="12b13e7f0ad225d372ba1b7dd88d224c95a1382b608683a1615b7f7aaeb05d8e"
Oct 30 21:04:38.049354 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:38.049283    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ac02efe-kkzwt"]
Oct 30 21:04:38.098057 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:38.098028    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ac02efe-kkzwt"]
Oct 30 21:04:39.066019 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:39.065984    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5346bed6-4f88-420e-a76f-b98fc50d1c2c" path="/var/lib/kubelet/pods/5346bed6-4f88-420e-a76f-b98fc50d1c2c/volumes"
Oct 30 21:04:41.098669 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:41.098640    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c76f5bd-z4wm4"]
Oct 30 21:04:41.104702 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:41.104669    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c76f5bd-z4wm4"]
Oct 30 21:04:42.049257 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:42.049227    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ed62713-2ff75"]
Oct 30 21:04:42.098537 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:42.098509    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ed62713-2ff75"]
Oct 30 21:04:43.065400 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:43.065366    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a541157-cbb6-442a-98cf-02dd5345cb8e" path="/var/lib/kubelet/pods/5a541157-cbb6-442a-98cf-02dd5345cb8e/volumes"
Oct 30 21:04:43.066056 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:43.066035    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da4c03af-ef11-46ae-bc29-e8e607cfc505" path="/var/lib/kubelet/pods/da4c03af-ef11-46ae-bc29-e8e607cfc505/volumes"
Oct 30 21:04:44.257780 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:44.257747    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-19db3d5f-7c8fc84467-lblmt"]
Oct 30 21:04:44.258435 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:44.258420    5713 cpu_manager.go:414] "containerMap: removing container" podUID="59246922-40fa-40ad-a84b-7604b077ef92" containerName="istio-proxy"
Oct 30 21:04:44.258435 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:44.258435    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59246922-40fa-40ad-a84b-7604b077ef92" containerName="istio-proxy"
Oct 30 21:04:44.258589 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:44.258579    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="59246922-40fa-40ad-a84b-7604b077ef92" containerName="istio-proxy"
Oct 30 21:04:44.267619 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:44.267589    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-19db3d5f-7c8fc84467-lblmt"
Oct 30 21:04:44.326042 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:44.325956    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-19db3d5f-7c8fc84467-lblmt"]
Oct 30 21:04:44.371818 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:44.369997    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2pkq\" (UniqueName: \"kubernetes.io/projected/ed8f7bb0-daa5-45b6-bee7-2689677ea6a5-kube-api-access-r2pkq\") pod \"payload-pre-processing-e2e-mt-19db3d5f-7c8fc84467-lblmt\" (UID: \"ed8f7bb0-daa5-45b6-bee7-2689677ea6a5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-19db3d5f-7c8fc84467-lblmt"
Oct 30 21:04:44.371818 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:44.370276    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed8f7bb0-daa5-45b6-bee7-2689677ea6a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-19db3d5f-7c8fc84467-lblmt\" (UID: \"ed8f7bb0-daa5-45b6-bee7-2689677ea6a5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-19db3d5f-7c8fc84467-lblmt"
Oct 30 21:04:44.375480 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:44.373560    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-19db3d5f-dockercfg-kjvj5\""
Oct 30 21:04:44.375480 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:44.373644    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-19db3d5f\""
Oct 30 21:04:44.472221 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:44.472169    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed8f7bb0-daa5-45b6-bee7-2689677ea6a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-19db3d5f-7c8fc84467-lblmt\" (UID: \"ed8f7bb0-daa5-45b6-bee7-2689677ea6a5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-19db3d5f-7c8fc84467-lblmt"
Oct 30 21:04:44.472819 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:44.472757    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2pkq\" (UniqueName: \"kubernetes.io/projected/ed8f7bb0-daa5-45b6-bee7-2689677ea6a5-kube-api-access-r2pkq\") pod \"payload-pre-processing-e2e-mt-19db3d5f-7c8fc84467-lblmt\" (UID: \"ed8f7bb0-daa5-45b6-bee7-2689677ea6a5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-19db3d5f-7c8fc84467-lblmt"
Oct 30 21:04:44.473418 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:44.473395    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed8f7bb0-daa5-45b6-bee7-2689677ea6a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-19db3d5f-7c8fc84467-lblmt\" (UID: \"ed8f7bb0-daa5-45b6-bee7-2689677ea6a5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-19db3d5f-7c8fc84467-lblmt"
Oct 30 21:04:44.530278 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:44.530210    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2pkq\" (UniqueName: \"kubernetes.io/projected/ed8f7bb0-daa5-45b6-bee7-2689677ea6a5-kube-api-access-r2pkq\") pod \"payload-pre-processing-e2e-mt-19db3d5f-7c8fc84467-lblmt\" (UID: \"ed8f7bb0-daa5-45b6-bee7-2689677ea6a5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-19db3d5f-7c8fc84467-lblmt"
Oct 30 21:04:44.584175 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:44.584145    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-19db3d5f-7c8fc84467-lblmt"
Oct 30 21:04:44.980709 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:04:44.980670    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded8f7bb0_daa5_45b6_bee7_2689677ea6a5.slice/crio-3924a0d6e31b3b528445b349a7e0526750193a4c54385a1b7198f7437d5fdd35 WatchSource:0}: Error finding container 3924a0d6e31b3b528445b349a7e0526750193a4c54385a1b7198f7437d5fdd35: Status 404 returned error can't find the container with id 3924a0d6e31b3b528445b349a7e0526750193a4c54385a1b7198f7437d5fdd35
Oct 30 21:04:44.980997 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:44.980973    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-19db3d5f-7c8fc84467-lblmt"]
Oct 30 21:04:45.067958 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:45.067928    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-19db3d5f-7c8fc84467-lblmt" event={"ID":"ed8f7bb0-daa5-45b6-bee7-2689677ea6a5","Type":"ContainerStarted","Data":"3924a0d6e31b3b528445b349a7e0526750193a4c54385a1b7198f7437d5fdd35"}
Oct 30 21:04:46.072558 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:46.072520    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-19db3d5f-7c8fc84467-lblmt" event={"ID":"ed8f7bb0-daa5-45b6-bee7-2689677ea6a5","Type":"ContainerStarted","Data":"bd2c0893c37df605aa71e805f482e09d9460ab784c9ee2386f81ac75f6ee5523"}
Oct 30 21:04:46.073042 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:46.072622    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-19db3d5f-7c8fc84467-lblmt"
Oct 30 21:04:46.190159 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:46.190108    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-19db3d5f-7c8fc84467-lblmt" podStartSLOduration=2.190089883 podStartE2EDuration="2.190089883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:04:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:04:46.189179991 +0000 UTC m=+1569.253956470" watchObservedRunningTime="2026-10-30 21:04:46.190089883 +0000 UTC m=+1569.254866281"
Oct 30 21:04:50.095925 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:50.095859    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef5fd58f-j4dmr"]
Oct 30 21:04:50.111933 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:50.111903    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef5fd58f-j4dmr"]
Oct 30 21:04:51.112966 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:51.112340    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90aaa4a2-48e3-4baa-bdc9-27b7c308ec89" path="/var/lib/kubelet/pods/90aaa4a2-48e3-4baa-bdc9-27b7c308ec89/volumes"
Oct 30 21:04:54.051065 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:54.051031    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-824fa8f4-2jt6r"]
Oct 30 21:04:54.099765 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:54.099735    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-824fa8f4-2jt6r"]
Oct 30 21:04:54.465151 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:04:54.465116    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:04:55.077378 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:55.077345    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2948b895-a1c8-426c-ba07-751bff16e075" path="/var/lib/kubelet/pods/2948b895-a1c8-426c-ba07-751bff16e075/volumes"
Oct 30 21:04:56.713669 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:56.713637    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z"]
Oct 30 21:04:56.714389 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:56.714372    5713 cpu_manager.go:414] "containerMap: removing container" podUID="59246922-40fa-40ad-a84b-7604b077ef92" containerName="istio-proxy"
Oct 30 21:04:56.714435 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:56.714390    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59246922-40fa-40ad-a84b-7604b077ef92" containerName="istio-proxy"
Oct 30 21:04:56.714536 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:56.714527    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="59246922-40fa-40ad-a84b-7604b077ef92" containerName="istio-proxy"
Oct 30 21:04:56.718088 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:56.718068    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z"
Oct 30 21:04:56.763493 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:56.763460    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd"]
Oct 30 21:04:56.767816 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:56.767779    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd"
Oct 30 21:04:56.770295 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:56.770275    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-bf9c0727-openshift-default-dockercfg-d4rns\""
Oct 30 21:04:56.797423 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:56.797394    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7784e9af-82d7-4072-ac6b-f2ae0e82e0b2-credential-socket\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z\" (UID: \"7784e9af-82d7-4072-ac6b-f2ae0e82e0b2\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z"
Oct 30 21:04:56.797530 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:56.797444    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7784e9af-82d7-4072-ac6b-f2ae0e82e0b2-istio-data\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z\" (UID: \"7784e9af-82d7-4072-ac6b-f2ae0e82e0b2\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z"
Oct 30 21:04:56.797593 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:56.797581    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7784e9af-82d7-4072-ac6b-f2ae0e82e0b2-workload-certs\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z\" (UID: \"7784e9af-82d7-4072-ac6b-f2ae0e82e0b2\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z"
Oct 30 21:04:56.797631 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:56.797608    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7784e9af-82d7-4072-ac6b-f2ae0e82e0b2-istiod-ca-cert\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z\" (UID: \"7784e9af-82d7-4072-ac6b-f2ae0e82e0b2\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z"
Oct 30 21:04:56.797668 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:56.797634    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7784e9af-82d7-4072-ac6b-f2ae0e82e0b2-workload-socket\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z\" (UID: \"7784e9af-82d7-4072-ac6b-f2ae0e82e0b2\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z"
Oct 30 21:04:56.797810 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:56.797767    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dh9c\" (UniqueName: \"kubernetes.io/projected/7784e9af-82d7-4072-ac6b-f2ae0e82e0b2-kube-api-access-9dh9c\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z\" (UID: \"7784e9af-82d7-4072-ac6b-f2ae0e82e0b2\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z"
Oct 30 21:04:56.797853 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:56.797832    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7784e9af-82d7-4072-ac6b-f2ae0e82e0b2-istio-podinfo\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z\" (UID: \"7784e9af-82d7-4072-ac6b-f2ae0e82e0b2\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z"
Oct 30 21:04:56.798009 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:56.797977    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7784e9af-82d7-4072-ac6b-f2ae0e82e0b2-istio-token\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z\" (UID: \"7784e9af-82d7-4072-ac6b-f2ae0e82e0b2\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z"
Oct 30 21:04:56.798126 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:56.798045    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7784e9af-82d7-4072-ac6b-f2ae0e82e0b2-istio-envoy\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z\" (UID: \"7784e9af-82d7-4072-ac6b-f2ae0e82e0b2\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z"
Oct 30 21:04:56.818242 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:56.818212    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z"]
Oct 30 21:04:56.869691 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:56.869649    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd"]
Oct 30 21:04:56.899216 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:56.899185    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7784e9af-82d7-4072-ac6b-f2ae0e82e0b2-credential-socket\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z\" (UID: \"7784e9af-82d7-4072-ac6b-f2ae0e82e0b2\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z"
Oct 30 21:04:56.899413 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:56.899250    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47b70e28-cf76-4cc2-a5d4-0396fb3e985c-istio-podinfo\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd\" (UID: \"47b70e28-cf76-4cc2-a5d4-0396fb3e985c\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd"
Oct 30 21:04:56.899413 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:56.899283    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7784e9af-82d7-4072-ac6b-f2ae0e82e0b2-istio-data\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z\" (UID: \"7784e9af-82d7-4072-ac6b-f2ae0e82e0b2\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z"
Oct 30 21:04:56.899413 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:56.899366    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47b70e28-cf76-4cc2-a5d4-0396fb3e985c-workload-certs\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd\" (UID: \"47b70e28-cf76-4cc2-a5d4-0396fb3e985c\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd"
Oct 30 21:04:56.899587 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:56.899520    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrv28\" (UniqueName: \"kubernetes.io/projected/47b70e28-cf76-4cc2-a5d4-0396fb3e985c-kube-api-access-wrv28\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd\" (UID: \"47b70e28-cf76-4cc2-a5d4-0396fb3e985c\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd"
Oct 30 21:04:56.899587 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:56.899559    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7784e9af-82d7-4072-ac6b-f2ae0e82e0b2-workload-certs\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z\" (UID: \"7784e9af-82d7-4072-ac6b-f2ae0e82e0b2\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z"
Oct 30 21:04:56.899700 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:56.899590    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7784e9af-82d7-4072-ac6b-f2ae0e82e0b2-istiod-ca-cert\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z\" (UID: \"7784e9af-82d7-4072-ac6b-f2ae0e82e0b2\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z"
Oct 30 21:04:56.899700 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:56.899625    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7784e9af-82d7-4072-ac6b-f2ae0e82e0b2-workload-socket\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z\" (UID: \"7784e9af-82d7-4072-ac6b-f2ae0e82e0b2\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z"
Oct 30 21:04:56.899700 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:56.899673    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7784e9af-82d7-4072-ac6b-f2ae0e82e0b2-credential-socket\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z\" (UID: \"7784e9af-82d7-4072-ac6b-f2ae0e82e0b2\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z"
Oct 30 21:04:56.899905 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:56.899711    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7784e9af-82d7-4072-ac6b-f2ae0e82e0b2-istio-data\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z\" (UID: \"7784e9af-82d7-4072-ac6b-f2ae0e82e0b2\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z"
Oct 30 21:04:56.899905 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:56.899713    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dh9c\" (UniqueName: \"kubernetes.io/projected/7784e9af-82d7-4072-ac6b-f2ae0e82e0b2-kube-api-access-9dh9c\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z\" (UID: \"7784e9af-82d7-4072-ac6b-f2ae0e82e0b2\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z"
Oct 30 21:04:56.899905 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:56.899776    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7784e9af-82d7-4072-ac6b-f2ae0e82e0b2-istio-podinfo\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z\" (UID: \"7784e9af-82d7-4072-ac6b-f2ae0e82e0b2\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z"
Oct 30 21:04:56.900046 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:56.899906    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47b70e28-cf76-4cc2-a5d4-0396fb3e985c-istiod-ca-cert\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd\" (UID: \"47b70e28-cf76-4cc2-a5d4-0396fb3e985c\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd"
Oct 30 21:04:56.900046 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:56.899972    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47b70e28-cf76-4cc2-a5d4-0396fb3e985c-istio-envoy\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd\" (UID: \"47b70e28-cf76-4cc2-a5d4-0396fb3e985c\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd"
Oct 30 21:04:56.900046 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:56.900022    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7784e9af-82d7-4072-ac6b-f2ae0e82e0b2-workload-certs\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z\" (UID: \"7784e9af-82d7-4072-ac6b-f2ae0e82e0b2\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z"
Oct 30 21:04:56.900213 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:56.900051    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47b70e28-cf76-4cc2-a5d4-0396fb3e985c-istio-token\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd\" (UID: \"47b70e28-cf76-4cc2-a5d4-0396fb3e985c\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd"
Oct 30 21:04:56.900213 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:56.900127    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7784e9af-82d7-4072-ac6b-f2ae0e82e0b2-istio-token\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z\" (UID: \"7784e9af-82d7-4072-ac6b-f2ae0e82e0b2\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z"
Oct 30 21:04:56.900213 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:56.900160    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47b70e28-cf76-4cc2-a5d4-0396fb3e985c-credential-socket\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd\" (UID: \"47b70e28-cf76-4cc2-a5d4-0396fb3e985c\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd"
Oct 30 21:04:56.900213 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:56.900200    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47b70e28-cf76-4cc2-a5d4-0396fb3e985c-workload-socket\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd\" (UID: \"47b70e28-cf76-4cc2-a5d4-0396fb3e985c\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd"
Oct 30 21:04:56.900377 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:56.900291    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47b70e28-cf76-4cc2-a5d4-0396fb3e985c-istio-data\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd\" (UID: \"47b70e28-cf76-4cc2-a5d4-0396fb3e985c\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd"
Oct 30 21:04:56.900377 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:56.900351    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7784e9af-82d7-4072-ac6b-f2ae0e82e0b2-istio-envoy\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z\" (UID: \"7784e9af-82d7-4072-ac6b-f2ae0e82e0b2\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z"
Oct 30 21:04:56.900758 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:56.900720    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7784e9af-82d7-4072-ac6b-f2ae0e82e0b2-workload-socket\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z\" (UID: \"7784e9af-82d7-4072-ac6b-f2ae0e82e0b2\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z"
Oct 30 21:04:56.901065 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:56.901041    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7784e9af-82d7-4072-ac6b-f2ae0e82e0b2-istiod-ca-cert\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z\" (UID: \"7784e9af-82d7-4072-ac6b-f2ae0e82e0b2\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z"
Oct 30 21:04:56.902722 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:56.902694    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7784e9af-82d7-4072-ac6b-f2ae0e82e0b2-istio-podinfo\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z\" (UID: \"7784e9af-82d7-4072-ac6b-f2ae0e82e0b2\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z"
Oct 30 21:04:56.903061 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:56.903039    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7784e9af-82d7-4072-ac6b-f2ae0e82e0b2-istio-envoy\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z\" (UID: \"7784e9af-82d7-4072-ac6b-f2ae0e82e0b2\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z"
Oct 30 21:04:56.957664 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:56.957633    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7784e9af-82d7-4072-ac6b-f2ae0e82e0b2-istio-token\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z\" (UID: \"7784e9af-82d7-4072-ac6b-f2ae0e82e0b2\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z"
Oct 30 21:04:56.958420 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:56.958402    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dh9c\" (UniqueName: \"kubernetes.io/projected/7784e9af-82d7-4072-ac6b-f2ae0e82e0b2-kube-api-access-9dh9c\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z\" (UID: \"7784e9af-82d7-4072-ac6b-f2ae0e82e0b2\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z"
Oct 30 21:04:57.001722 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:57.001652    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47b70e28-cf76-4cc2-a5d4-0396fb3e985c-istiod-ca-cert\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd\" (UID: \"47b70e28-cf76-4cc2-a5d4-0396fb3e985c\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd"
Oct 30 21:04:57.001722 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:57.001702    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47b70e28-cf76-4cc2-a5d4-0396fb3e985c-istio-envoy\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd\" (UID: \"47b70e28-cf76-4cc2-a5d4-0396fb3e985c\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd"
Oct 30 21:04:57.001954 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:57.001734    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47b70e28-cf76-4cc2-a5d4-0396fb3e985c-istio-token\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd\" (UID: \"47b70e28-cf76-4cc2-a5d4-0396fb3e985c\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd"
Oct 30 21:04:57.001954 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:57.001767    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47b70e28-cf76-4cc2-a5d4-0396fb3e985c-credential-socket\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd\" (UID: \"47b70e28-cf76-4cc2-a5d4-0396fb3e985c\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd"
Oct 30 21:04:57.001954 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:57.001817    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47b70e28-cf76-4cc2-a5d4-0396fb3e985c-workload-socket\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd\" (UID: \"47b70e28-cf76-4cc2-a5d4-0396fb3e985c\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd"
Oct 30 21:04:57.001954 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:57.001852    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47b70e28-cf76-4cc2-a5d4-0396fb3e985c-istio-data\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd\" (UID: \"47b70e28-cf76-4cc2-a5d4-0396fb3e985c\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd"
Oct 30 21:04:57.001954 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:57.001942    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47b70e28-cf76-4cc2-a5d4-0396fb3e985c-istio-podinfo\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd\" (UID: \"47b70e28-cf76-4cc2-a5d4-0396fb3e985c\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd"
Oct 30 21:04:57.002234 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:57.002019    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47b70e28-cf76-4cc2-a5d4-0396fb3e985c-workload-certs\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd\" (UID: \"47b70e28-cf76-4cc2-a5d4-0396fb3e985c\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd"
Oct 30 21:04:57.002234 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:57.002064    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrv28\" (UniqueName: \"kubernetes.io/projected/47b70e28-cf76-4cc2-a5d4-0396fb3e985c-kube-api-access-wrv28\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd\" (UID: \"47b70e28-cf76-4cc2-a5d4-0396fb3e985c\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd"
Oct 30 21:04:57.002349 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:57.002290    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47b70e28-cf76-4cc2-a5d4-0396fb3e985c-credential-socket\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd\" (UID: \"47b70e28-cf76-4cc2-a5d4-0396fb3e985c\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd"
Oct 30 21:04:57.002349 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:57.002297    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47b70e28-cf76-4cc2-a5d4-0396fb3e985c-istio-data\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd\" (UID: \"47b70e28-cf76-4cc2-a5d4-0396fb3e985c\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd"
Oct 30 21:04:57.002463 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:57.002426    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47b70e28-cf76-4cc2-a5d4-0396fb3e985c-istiod-ca-cert\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd\" (UID: \"47b70e28-cf76-4cc2-a5d4-0396fb3e985c\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd"
Oct 30 21:04:57.002567 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:57.002545    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47b70e28-cf76-4cc2-a5d4-0396fb3e985c-workload-socket\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd\" (UID: \"47b70e28-cf76-4cc2-a5d4-0396fb3e985c\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd"
Oct 30 21:04:57.002622 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:57.002609    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47b70e28-cf76-4cc2-a5d4-0396fb3e985c-workload-certs\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd\" (UID: \"47b70e28-cf76-4cc2-a5d4-0396fb3e985c\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd"
Oct 30 21:04:57.004618 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:57.004597    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47b70e28-cf76-4cc2-a5d4-0396fb3e985c-istio-envoy\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd\" (UID: \"47b70e28-cf76-4cc2-a5d4-0396fb3e985c\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd"
Oct 30 21:04:57.004739 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:57.004672    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47b70e28-cf76-4cc2-a5d4-0396fb3e985c-istio-podinfo\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd\" (UID: \"47b70e28-cf76-4cc2-a5d4-0396fb3e985c\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd"
Oct 30 21:04:57.032042 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:57.032012    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z"
Oct 30 21:04:57.058438 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:57.058402    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrv28\" (UniqueName: \"kubernetes.io/projected/47b70e28-cf76-4cc2-a5d4-0396fb3e985c-kube-api-access-wrv28\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd\" (UID: \"47b70e28-cf76-4cc2-a5d4-0396fb3e985c\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd"
Oct 30 21:04:57.058561 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:57.058534    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47b70e28-cf76-4cc2-a5d4-0396fb3e985c-istio-token\") pod \"e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd\" (UID: \"47b70e28-cf76-4cc2-a5d4-0396fb3e985c\") " pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd"
Oct 30 21:04:57.082315 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:57.082284    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd"
Oct 30 21:04:57.085954 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:57.085928    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-19db3d5f-7c8fc84467-lblmt"
Oct 30 21:04:57.474399 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:04:57.474369    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7784e9af_82d7_4072_ac6b_f2ae0e82e0b2.slice/crio-a81090a030fdace8fca9ebdf1cce412bb6e12a65d5ebf0957dae77bd9113438a WatchSource:0}: Error finding container a81090a030fdace8fca9ebdf1cce412bb6e12a65d5ebf0957dae77bd9113438a: Status 404 returned error can't find the container with id a81090a030fdace8fca9ebdf1cce412bb6e12a65d5ebf0957dae77bd9113438a
Oct 30 21:04:57.474643 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:57.474623    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z"]
Oct 30 21:04:57.477045 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:57.477018    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:04:57.477128 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:57.477077    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:04:57.477128 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:57.477109    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:04:57.527055 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:57.527029    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd"]
Oct 30 21:04:57.530297 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:04:57.530271    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47b70e28_cf76_4cc2_a5d4_0396fb3e985c.slice/crio-3485a8770fa2768e5b8f53eec85e73b00d0e49e0f3dbe3835411a75f802a6128 WatchSource:0}: Error finding container 3485a8770fa2768e5b8f53eec85e73b00d0e49e0f3dbe3835411a75f802a6128: Status 404 returned error can't find the container with id 3485a8770fa2768e5b8f53eec85e73b00d0e49e0f3dbe3835411a75f802a6128
Oct 30 21:04:57.533338 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:57.533307    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:04:57.533437 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:57.533373    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:04:57.533437 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:57.533401    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:04:58.277070 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:58.277038    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd" event={"ID":"47b70e28-cf76-4cc2-a5d4-0396fb3e985c","Type":"ContainerStarted","Data":"9928059868f25092856d94350ca3468f176e6d0df55de7c1ddfb4631c4e43d90"}
Oct 30 21:04:58.277511 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:58.277091    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd" event={"ID":"47b70e28-cf76-4cc2-a5d4-0396fb3e985c","Type":"ContainerStarted","Data":"3485a8770fa2768e5b8f53eec85e73b00d0e49e0f3dbe3835411a75f802a6128"}
Oct 30 21:04:58.278904 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:58.278821    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z" event={"ID":"7784e9af-82d7-4072-ac6b-f2ae0e82e0b2","Type":"ContainerStarted","Data":"7472a41fc790c302d57461fc209939a50e62ed47a16a93ca4507563a1fb88b11"}
Oct 30 21:04:58.278904 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:58.278873    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z" event={"ID":"7784e9af-82d7-4072-ac6b-f2ae0e82e0b2","Type":"ContainerStarted","Data":"a81090a030fdace8fca9ebdf1cce412bb6e12a65d5ebf0957dae77bd9113438a"}
Oct 30 21:04:58.625879 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:58.625764    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd" podStartSLOduration=2.625746012 podStartE2EDuration="2.625746012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:04:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:04:58.506097222 +0000 UTC m=+1581.570873619" watchObservedRunningTime="2026-10-30 21:04:58.625746012 +0000 UTC m=+1581.690522422"
Oct 30 21:04:58.626319 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:58.626298    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z" podStartSLOduration=2.626289059 podStartE2EDuration="2.626289059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:04:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:04:58.622823394 +0000 UTC m=+1581.687599792" watchObservedRunningTime="2026-10-30 21:04:58.626289059 +0000 UTC m=+1581.691065538"
Oct 30 21:04:59.033507 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:59.033476    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z"
Oct 30 21:04:59.048186 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:59.048147    5713 patch_prober.go:29] interesting pod/e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Oct 30 21:04:59.048459 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:59.048431    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z" podUID="7784e9af-82d7-4072-ac6b-f2ae0e82e0b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Oct 30 21:04:59.089643 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:59.089549    5713 patch_prober.go:29] interesting pod/e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Oct 30 21:04:59.089643 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:59.089606    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd" podUID="47b70e28-cf76-4cc2-a5d4-0396fb3e985c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Oct 30 21:04:59.097492 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:04:59.097464    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd"
Oct 30 21:05:00.032508 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:00.032474    5713 patch_prober.go:29] interesting pod/e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Oct 30 21:05:00.033022 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:00.032531    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z" podUID="7784e9af-82d7-4072-ac6b-f2ae0e82e0b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Oct 30 21:05:00.085143 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:00.085109    5713 patch_prober.go:29] interesting pod/e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Oct 30 21:05:00.085425 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:00.085394    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd" podUID="47b70e28-cf76-4cc2-a5d4-0396fb3e985c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Oct 30 21:05:01.032954 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:01.032921    5713 patch_prober.go:29] interesting pod/e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Oct 30 21:05:01.033474 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:01.032980    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z" podUID="7784e9af-82d7-4072-ac6b-f2ae0e82e0b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Oct 30 21:05:01.090177 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:01.090144    5713 patch_prober.go:29] interesting pod/e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Oct 30 21:05:01.090354 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:01.090266    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd" podUID="47b70e28-cf76-4cc2-a5d4-0396fb3e985c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Oct 30 21:05:02.032425 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:02.032386    5713 patch_prober.go:29] interesting pod/e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Oct 30 21:05:02.032720 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:02.032687    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z" podUID="7784e9af-82d7-4072-ac6b-f2ae0e82e0b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Oct 30 21:05:02.082905 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:02.082868    5713 patch_prober.go:29] interesting pod/e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Oct 30 21:05:02.083481 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:02.083452    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd" podUID="47b70e28-cf76-4cc2-a5d4-0396fb3e985c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Oct 30 21:05:03.033043 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:03.033017    5713 patch_prober.go:29] interesting pod/e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Oct 30 21:05:03.033271 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:03.033247    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z" podUID="7784e9af-82d7-4072-ac6b-f2ae0e82e0b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Oct 30 21:05:03.087865 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:03.087827    5713 patch_prober.go:29] interesting pod/e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Oct 30 21:05:03.088511 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:03.088471    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd" podUID="47b70e28-cf76-4cc2-a5d4-0396fb3e985c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Oct 30 21:05:03.101888 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:03.101860    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj"]
Oct 30 21:05:03.102067 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:03.102041    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj" podUID="abfc50b7-cb57-4beb-bed1-70204a890a8a" containerName="istio-proxy" containerID="cri-o://41fbd6aa0515284ade2bdb53b910c4d2662ab55942efce7633fb565c93cfb232" gracePeriod=30
Oct 30 21:05:04.062856 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:04.062825    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z"
Oct 30 21:05:04.063383 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:04.063365    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z"
Oct 30 21:05:04.064142 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:04.064123    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z"
Oct 30 21:05:04.088268 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:04.088239    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd"
Oct 30 21:05:04.088695 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:04.088554    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd"
Oct 30 21:05:04.089244 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:04.089227    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd"
Oct 30 21:05:08.378322 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:08.378257    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj"
Oct 30 21:05:08.380552 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:08.380523    5713 generic.go:356] "Generic (PLEG): container finished" podID="abfc50b7-cb57-4beb-bed1-70204a890a8a" containerID="41fbd6aa0515284ade2bdb53b910c4d2662ab55942efce7633fb565c93cfb232" exitCode=0
Oct 30 21:05:08.380679 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:08.380565    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj"
Oct 30 21:05:08.380679 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:08.380598    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj" event={"ID":"abfc50b7-cb57-4beb-bed1-70204a890a8a","Type":"ContainerDied","Data":"41fbd6aa0515284ade2bdb53b910c4d2662ab55942efce7633fb565c93cfb232"}
Oct 30 21:05:08.380679 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:08.380649    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj" event={"ID":"abfc50b7-cb57-4beb-bed1-70204a890a8a","Type":"ContainerDied","Data":"4b95131a0a417dc9b14c2cd182e1b2d25a488090752affeced873d1ee5e2ff12"}
Oct 30 21:05:08.380679 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:08.380670    5713 scope.go:122] "RemoveContainer" containerID="41fbd6aa0515284ade2bdb53b910c4d2662ab55942efce7633fb565c93cfb232"
Oct 30 21:05:08.394616 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:08.394596    5713 scope.go:122] "RemoveContainer" containerID="41fbd6aa0515284ade2bdb53b910c4d2662ab55942efce7633fb565c93cfb232"
Oct 30 21:05:08.394947 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:05:08.394924    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41fbd6aa0515284ade2bdb53b910c4d2662ab55942efce7633fb565c93cfb232\": container with ID starting with 41fbd6aa0515284ade2bdb53b910c4d2662ab55942efce7633fb565c93cfb232 not found: ID does not exist" containerID="41fbd6aa0515284ade2bdb53b910c4d2662ab55942efce7633fb565c93cfb232"
Oct 30 21:05:08.395019 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:08.394958    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41fbd6aa0515284ade2bdb53b910c4d2662ab55942efce7633fb565c93cfb232"} err="failed to get container status \"41fbd6aa0515284ade2bdb53b910c4d2662ab55942efce7633fb565c93cfb232\": rpc error: code = NotFound desc = could not find container \"41fbd6aa0515284ade2bdb53b910c4d2662ab55942efce7633fb565c93cfb232\": container with ID starting with 41fbd6aa0515284ade2bdb53b910c4d2662ab55942efce7633fb565c93cfb232 not found: ID does not exist"
Oct 30 21:05:08.514633 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:08.514603    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abfc50b7-cb57-4beb-bed1-70204a890a8a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abfc50b7-cb57-4beb-bed1-70204a890a8a-istio-envoy\") pod \"abfc50b7-cb57-4beb-bed1-70204a890a8a\" (UID: \"abfc50b7-cb57-4beb-bed1-70204a890a8a\") "
Oct 30 21:05:08.514866 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:08.514673    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abfc50b7-cb57-4beb-bed1-70204a890a8a-kube-api-access-ks45n\" (UniqueName: \"kubernetes.io/projected/abfc50b7-cb57-4beb-bed1-70204a890a8a-kube-api-access-ks45n\") pod \"abfc50b7-cb57-4beb-bed1-70204a890a8a\" (UID: \"abfc50b7-cb57-4beb-bed1-70204a890a8a\") "
Oct 30 21:05:08.514866 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:08.514694    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abfc50b7-cb57-4beb-bed1-70204a890a8a-istio-token\" (UniqueName: \"kubernetes.io/projected/abfc50b7-cb57-4beb-bed1-70204a890a8a-istio-token\") pod \"abfc50b7-cb57-4beb-bed1-70204a890a8a\" (UID: \"abfc50b7-cb57-4beb-bed1-70204a890a8a\") "
Oct 30 21:05:08.514866 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:08.514766    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/abfc50b7-cb57-4beb-bed1-70204a890a8a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abfc50b7-cb57-4beb-bed1-70204a890a8a-istio-podinfo\") pod \"abfc50b7-cb57-4beb-bed1-70204a890a8a\" (UID: \"abfc50b7-cb57-4beb-bed1-70204a890a8a\") "
Oct 30 21:05:08.515064 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:08.514877    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abfc50b7-cb57-4beb-bed1-70204a890a8a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abfc50b7-cb57-4beb-bed1-70204a890a8a-istiod-ca-cert\") pod \"abfc50b7-cb57-4beb-bed1-70204a890a8a\" (UID: \"abfc50b7-cb57-4beb-bed1-70204a890a8a\") "
Oct 30 21:05:08.515064 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:08.514901    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abfc50b7-cb57-4beb-bed1-70204a890a8a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abfc50b7-cb57-4beb-bed1-70204a890a8a-workload-certs\") pod \"abfc50b7-cb57-4beb-bed1-70204a890a8a\" (UID: \"abfc50b7-cb57-4beb-bed1-70204a890a8a\") "
Oct 30 21:05:08.515064 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:08.515019    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abfc50b7-cb57-4beb-bed1-70204a890a8a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abfc50b7-cb57-4beb-bed1-70204a890a8a-credential-socket\") pod \"abfc50b7-cb57-4beb-bed1-70204a890a8a\" (UID: \"abfc50b7-cb57-4beb-bed1-70204a890a8a\") "
Oct 30 21:05:08.515229 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:08.515171    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abfc50b7-cb57-4beb-bed1-70204a890a8a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abfc50b7-cb57-4beb-bed1-70204a890a8a-istio-data\") pod \"abfc50b7-cb57-4beb-bed1-70204a890a8a\" (UID: \"abfc50b7-cb57-4beb-bed1-70204a890a8a\") "
Oct 30 21:05:08.515229 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:08.515203    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abfc50b7-cb57-4beb-bed1-70204a890a8a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abfc50b7-cb57-4beb-bed1-70204a890a8a-workload-socket\") pod \"abfc50b7-cb57-4beb-bed1-70204a890a8a\" (UID: \"abfc50b7-cb57-4beb-bed1-70204a890a8a\") "
Oct 30 21:05:08.515336 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:08.515278    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abfc50b7-cb57-4beb-bed1-70204a890a8a-istiod-ca-cert" pod "abfc50b7-cb57-4beb-bed1-70204a890a8a" (UID: "abfc50b7-cb57-4beb-bed1-70204a890a8a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:05:08.515963 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:08.515506    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abfc50b7-cb57-4beb-bed1-70204a890a8a-credential-socket" pod "abfc50b7-cb57-4beb-bed1-70204a890a8a" (UID: "abfc50b7-cb57-4beb-bed1-70204a890a8a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:05:08.515963 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:08.515672    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abfc50b7-cb57-4beb-bed1-70204a890a8a-workload-certs" pod "abfc50b7-cb57-4beb-bed1-70204a890a8a" (UID: "abfc50b7-cb57-4beb-bed1-70204a890a8a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:05:08.515963 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:08.515867    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abfc50b7-cb57-4beb-bed1-70204a890a8a-workload-socket" pod "abfc50b7-cb57-4beb-bed1-70204a890a8a" (UID: "abfc50b7-cb57-4beb-bed1-70204a890a8a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:05:08.516224 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:08.516003    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abfc50b7-cb57-4beb-bed1-70204a890a8a-istio-data" pod "abfc50b7-cb57-4beb-bed1-70204a890a8a" (UID: "abfc50b7-cb57-4beb-bed1-70204a890a8a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:05:08.516224 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:08.516108    5713 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abfc50b7-cb57-4beb-bed1-70204a890a8a-workload-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:08.516224 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:08.516144    5713 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abfc50b7-cb57-4beb-bed1-70204a890a8a-credential-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:08.516224 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:08.516159    5713 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abfc50b7-cb57-4beb-bed1-70204a890a8a-workload-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:08.516224 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:08.516173    5713 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abfc50b7-cb57-4beb-bed1-70204a890a8a-istiod-ca-cert\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:08.517312 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:08.517284    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abfc50b7-cb57-4beb-bed1-70204a890a8a-kube-api-access-ks45n" pod "abfc50b7-cb57-4beb-bed1-70204a890a8a" (UID: "abfc50b7-cb57-4beb-bed1-70204a890a8a"). InnerVolumeSpecName "kube-api-access-ks45n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:05:08.517429 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:08.517315    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abfc50b7-cb57-4beb-bed1-70204a890a8a-istio-envoy" pod "abfc50b7-cb57-4beb-bed1-70204a890a8a" (UID: "abfc50b7-cb57-4beb-bed1-70204a890a8a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:05:08.517651 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:08.517620    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abfc50b7-cb57-4beb-bed1-70204a890a8a-istio-token" pod "abfc50b7-cb57-4beb-bed1-70204a890a8a" (UID: "abfc50b7-cb57-4beb-bed1-70204a890a8a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:05:08.517651 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:08.517646    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/abfc50b7-cb57-4beb-bed1-70204a890a8a-istio-podinfo" pod "abfc50b7-cb57-4beb-bed1-70204a890a8a" (UID: "abfc50b7-cb57-4beb-bed1-70204a890a8a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:05:08.618521 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:08.618480    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks45n\" (UniqueName: \"kubernetes.io/projected/abfc50b7-cb57-4beb-bed1-70204a890a8a-kube-api-access-ks45n\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:08.618521 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:08.618524    5713 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abfc50b7-cb57-4beb-bed1-70204a890a8a-istio-token\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:08.618777 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:08.618541    5713 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abfc50b7-cb57-4beb-bed1-70204a890a8a-istio-podinfo\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:08.618777 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:08.618557    5713 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abfc50b7-cb57-4beb-bed1-70204a890a8a-istio-data\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:08.618777 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:08.618571    5713 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abfc50b7-cb57-4beb-bed1-70204a890a8a-istio-envoy\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:08.808265 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:08.808234    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj"]
Oct 30 21:05:08.812967 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:08.812942    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-pxtbj"]
Oct 30 21:05:09.066002 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:09.065921    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abfc50b7-cb57-4beb-bed1-70204a890a8a" path="/var/lib/kubelet/pods/abfc50b7-cb57-4beb-bed1-70204a890a8a/volumes"
Oct 30 21:05:20.047308 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:20.047276    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f22a82ec-fb4ps"]
Oct 30 21:05:20.096472 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:20.096440    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f22a82ec-fb4ps"]
Oct 30 21:05:21.065521 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:21.065486    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdd58401-d3df-4fa1-a8bd-68c8abccc563" path="/var/lib/kubelet/pods/cdd58401-d3df-4fa1-a8bd-68c8abccc563/volumes"
Oct 30 21:05:24.460378 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:05:24.460343    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:05:25.047553 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:25.047520    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba72103f-l4l8h"]
Oct 30 21:05:25.097441 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:25.097414    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba72103f-l4l8h"]
Oct 30 21:05:27.065215 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:27.065183    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0482372-b713-458e-aae5-412c81b49544" path="/var/lib/kubelet/pods/c0482372-b713-458e-aae5-412c81b49544/volumes"
Oct 30 21:05:32.049940 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:32.049907    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14e949b0-6rch5"]
Oct 30 21:05:32.099169 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:32.099135    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14e949b0-6rch5"]
Oct 30 21:05:32.572463 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:32.572426    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2cf21005-f9k2m"]
Oct 30 21:05:32.573187 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:32.573172    5713 cpu_manager.go:414] "containerMap: removing container" podUID="abfc50b7-cb57-4beb-bed1-70204a890a8a" containerName="istio-proxy"
Oct 30 21:05:32.573248 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:32.573188    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abfc50b7-cb57-4beb-bed1-70204a890a8a" containerName="istio-proxy"
Oct 30 21:05:32.573305 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:32.573294    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="abfc50b7-cb57-4beb-bed1-70204a890a8a" containerName="istio-proxy"
Oct 30 21:05:32.577379 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:32.577362    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2cf21005-f9k2m"
Oct 30 21:05:32.632513 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:32.632480    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2cf21005-f9k2m"]
Oct 30 21:05:32.633907 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:32.633876    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc434250-12e2-4d99-b6d9-f013d995507b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2cf21005-f9k2m\" (UID: \"dc434250-12e2-4d99-b6d9-f013d995507b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2cf21005-f9k2m"
Oct 30 21:05:32.634036 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:32.633999    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg8gs\" (UniqueName: \"kubernetes.io/projected/dc434250-12e2-4d99-b6d9-f013d995507b-kube-api-access-jg8gs\") pod \"maas-api-revoke-keys-e2e-ait-2cf21005-f9k2m\" (UID: \"dc434250-12e2-4d99-b6d9-f013d995507b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2cf21005-f9k2m"
Oct 30 21:05:32.634264 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:32.634246    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc434250-12e2-4d99-b6d9-f013d995507b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2cf21005-f9k2m\" (UID: \"dc434250-12e2-4d99-b6d9-f013d995507b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2cf21005-f9k2m"
Oct 30 21:05:32.676860 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:32.676831    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:05:32.677042 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:32.676893    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:05:32.677281 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:32.677265    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:05:32.677374 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:32.677353    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:05:32.735518 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:32.735463    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc434250-12e2-4d99-b6d9-f013d995507b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2cf21005-f9k2m\" (UID: \"dc434250-12e2-4d99-b6d9-f013d995507b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2cf21005-f9k2m"
Oct 30 21:05:32.735736 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:32.735582    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc434250-12e2-4d99-b6d9-f013d995507b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2cf21005-f9k2m\" (UID: \"dc434250-12e2-4d99-b6d9-f013d995507b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2cf21005-f9k2m"
Oct 30 21:05:32.735736 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:32.735674    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg8gs\" (UniqueName: \"kubernetes.io/projected/dc434250-12e2-4d99-b6d9-f013d995507b-kube-api-access-jg8gs\") pod \"maas-api-revoke-keys-e2e-ait-2cf21005-f9k2m\" (UID: \"dc434250-12e2-4d99-b6d9-f013d995507b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2cf21005-f9k2m"
Oct 30 21:05:32.736237 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:32.736214    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc434250-12e2-4d99-b6d9-f013d995507b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2cf21005-f9k2m\" (UID: \"dc434250-12e2-4d99-b6d9-f013d995507b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2cf21005-f9k2m"
Oct 30 21:05:32.738084 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:32.738065    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc434250-12e2-4d99-b6d9-f013d995507b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2cf21005-f9k2m\" (UID: \"dc434250-12e2-4d99-b6d9-f013d995507b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2cf21005-f9k2m"
Oct 30 21:05:32.794320 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:32.794282    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg8gs\" (UniqueName: \"kubernetes.io/projected/dc434250-12e2-4d99-b6d9-f013d995507b-kube-api-access-jg8gs\") pod \"maas-api-revoke-keys-e2e-ait-2cf21005-f9k2m\" (UID: \"dc434250-12e2-4d99-b6d9-f013d995507b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2cf21005-f9k2m"
Oct 30 21:05:32.892483 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:32.892393    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2cf21005-f9k2m"
Oct 30 21:05:33.065636 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:33.065600    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c5f608f-fd3f-4180-af2f-e9b162996c67" path="/var/lib/kubelet/pods/4c5f608f-fd3f-4180-af2f-e9b162996c67/volumes"
Oct 30 21:05:33.280988 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:05:33.280951    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc434250_12e2_4d99_b6d9_f013d995507b.slice/crio-e88be44e26f206450f066a3b6bdb9cba2fa9e3d262a897c778c362f04b142a14 WatchSource:0}: Error finding container e88be44e26f206450f066a3b6bdb9cba2fa9e3d262a897c778c362f04b142a14: Status 404 returned error can't find the container with id e88be44e26f206450f066a3b6bdb9cba2fa9e3d262a897c778c362f04b142a14
Oct 30 21:05:33.281141 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:33.281115    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2cf21005-f9k2m"]
Oct 30 21:05:33.601340 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:33.601288    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2cf21005-f9k2m" event={"ID":"dc434250-12e2-4d99-b6d9-f013d995507b","Type":"ContainerStarted","Data":"7cc5a6e50d2524ac9ca47aa75cde48fe8031aea6a1c398b8a70592914995a270"}
Oct 30 21:05:33.601540 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:33.601356    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2cf21005-f9k2m" event={"ID":"dc434250-12e2-4d99-b6d9-f013d995507b","Type":"ContainerStarted","Data":"e88be44e26f206450f066a3b6bdb9cba2fa9e3d262a897c778c362f04b142a14"}
Oct 30 21:05:33.717744 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:33.717689    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2cf21005-f9k2m" podStartSLOduration=1.717672549 podStartE2EDuration="1.717672549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:05:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:05:33.715768483 +0000 UTC m=+1616.780544894" watchObservedRunningTime="2026-10-30 21:05:33.717672549 +0000 UTC m=+1616.782448947"
Oct 30 21:05:34.116549 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:34.116518    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk"]
Oct 30 21:05:34.117015 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:34.116761    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk" podUID="9a73cfa4-d614-4c39-b324-731fca0564c4" containerName="istio-proxy" containerID="cri-o://c1ec5df9fca78a32bc25d54802c6a63c1a397a64889906056dfba49017d9ad30" gracePeriod=30
Oct 30 21:05:34.613506 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:34.613469    5713 generic.go:356] "Generic (PLEG): container finished" podID="dc434250-12e2-4d99-b6d9-f013d995507b" containerID="7cc5a6e50d2524ac9ca47aa75cde48fe8031aea6a1c398b8a70592914995a270" exitCode=0
Oct 30 21:05:34.613694 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:34.613558    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2cf21005-f9k2m" event={"ID":"dc434250-12e2-4d99-b6d9-f013d995507b","Type":"ContainerDied","Data":"7cc5a6e50d2524ac9ca47aa75cde48fe8031aea6a1c398b8a70592914995a270"}
Oct 30 21:05:35.799066 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:35.799042    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2cf21005-f9k2m"
Oct 30 21:05:35.891679 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:35.891601    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc434250-12e2-4d99-b6d9-f013d995507b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc434250-12e2-4d99-b6d9-f013d995507b-maas-api-xks-ca\") pod \"dc434250-12e2-4d99-b6d9-f013d995507b\" (UID: \"dc434250-12e2-4d99-b6d9-f013d995507b\") "
Oct 30 21:05:35.891869 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:35.891855    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc434250-12e2-4d99-b6d9-f013d995507b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc434250-12e2-4d99-b6d9-f013d995507b-maas-api-openshift-ca\") pod \"dc434250-12e2-4d99-b6d9-f013d995507b\" (UID: \"dc434250-12e2-4d99-b6d9-f013d995507b\") "
Oct 30 21:05:35.891946 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:35.891934    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc434250-12e2-4d99-b6d9-f013d995507b-kube-api-access-jg8gs\" (UniqueName: \"kubernetes.io/projected/dc434250-12e2-4d99-b6d9-f013d995507b-kube-api-access-jg8gs\") pod \"dc434250-12e2-4d99-b6d9-f013d995507b\" (UID: \"dc434250-12e2-4d99-b6d9-f013d995507b\") "
Oct 30 21:05:35.892184 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:35.892156    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc434250-12e2-4d99-b6d9-f013d995507b-maas-api-openshift-ca" pod "dc434250-12e2-4d99-b6d9-f013d995507b" (UID: "dc434250-12e2-4d99-b6d9-f013d995507b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:05:35.892477 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:35.892463    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc434250-12e2-4d99-b6d9-f013d995507b-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:35.893814 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:35.893777    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc434250-12e2-4d99-b6d9-f013d995507b-maas-api-xks-ca" pod "dc434250-12e2-4d99-b6d9-f013d995507b" (UID: "dc434250-12e2-4d99-b6d9-f013d995507b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:05:35.894082 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:35.894062    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc434250-12e2-4d99-b6d9-f013d995507b-kube-api-access-jg8gs" pod "dc434250-12e2-4d99-b6d9-f013d995507b" (UID: "dc434250-12e2-4d99-b6d9-f013d995507b"). InnerVolumeSpecName "kube-api-access-jg8gs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:05:35.993925 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:35.993890    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jg8gs\" (UniqueName: \"kubernetes.io/projected/dc434250-12e2-4d99-b6d9-f013d995507b-kube-api-access-jg8gs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:35.993925 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:35.993920    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc434250-12e2-4d99-b6d9-f013d995507b-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:36.638316 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:36.638287    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2cf21005-f9k2m"
Oct 30 21:05:36.638510 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:36.638318    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2cf21005-f9k2m" event={"ID":"dc434250-12e2-4d99-b6d9-f013d995507b","Type":"ContainerDied","Data":"e88be44e26f206450f066a3b6bdb9cba2fa9e3d262a897c778c362f04b142a14"}
Oct 30 21:05:36.638510 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:36.638375    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e88be44e26f206450f066a3b6bdb9cba2fa9e3d262a897c778c362f04b142a14"
Oct 30 21:05:37.070263 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:37.070223    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-378d8807-h944m"]
Oct 30 21:05:37.079890 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:37.079856    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-378d8807-h944m"]
Oct 30 21:05:37.085017 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:37.084987    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c80d8c0a-97b7d"]
Oct 30 21:05:37.094503 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:37.094474    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c80d8c0a-97b7d"]
Oct 30 21:05:37.750045 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:37.749341    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf9c0727-v9llb"]
Oct 30 21:05:37.750450 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:37.750421    5713 cpu_manager.go:414] "containerMap: removing container" podUID="dc434250-12e2-4d99-b6d9-f013d995507b" containerName="revoke-keys"
Oct 30 21:05:37.750638 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:37.750620    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc434250-12e2-4d99-b6d9-f013d995507b" containerName="revoke-keys"
Oct 30 21:05:37.751095 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:37.751071    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc434250-12e2-4d99-b6d9-f013d995507b" containerName="revoke-keys"
Oct 30 21:05:37.756440 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:37.756418    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf9c0727-v9llb"
Oct 30 21:05:37.775594 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:37.775567    5713 scope.go:122] "RemoveContainer" containerID="6f2db1d55aa80062081ace611bd2d24c1d85946ec87078eb57615843ef9d34c8"
Oct 30 21:05:37.799855 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:37.799829    5713 scope.go:122] "RemoveContainer" containerID="57936174c0bbdc452f0ba8250b4ba5501322f3fe448b4365427a7a2720b7c72f"
Oct 30 21:05:37.812862 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:37.812828    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf9c0727-v9llb"]
Oct 30 21:05:37.813405 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:37.813383    5713 scope.go:122] "RemoveContainer" containerID="6045a0c58a6b856e0ec0931bde8a2dcad3f0e191f85d4fc4830426f026b07ddb"
Oct 30 21:05:37.822338 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:37.822310    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ced570ba-f5f3-4c25-bc1d-0544046ed9dc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-bf9c0727-v9llb\" (UID: \"ced570ba-f5f3-4c25-bc1d-0544046ed9dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf9c0727-v9llb"
Oct 30 21:05:37.822474 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:37.822355    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ced570ba-f5f3-4c25-bc1d-0544046ed9dc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-bf9c0727-v9llb\" (UID: \"ced570ba-f5f3-4c25-bc1d-0544046ed9dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf9c0727-v9llb"
Oct 30 21:05:37.822576 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:37.822560    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj2bv\" (UniqueName: \"kubernetes.io/projected/ced570ba-f5f3-4c25-bc1d-0544046ed9dc-kube-api-access-kj2bv\") pod \"maas-api-revoke-keys-e2e-derive-bf9c0727-v9llb\" (UID: \"ced570ba-f5f3-4c25-bc1d-0544046ed9dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf9c0727-v9llb"
Oct 30 21:05:37.832281 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:37.832257    5713 scope.go:122] "RemoveContainer" containerID="c24e3bb2ee11c3d413ab1f10968dc4032925964860c1bc5e91fff88fb1abff5a"
Oct 30 21:05:37.846032 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:37.846010    5713 scope.go:122] "RemoveContainer" containerID="29d2eef40f757d550bce5da59435377a159730d5c0e008b928238bc3ec97d1cc"
Oct 30 21:05:37.856038 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:37.856014    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:05:37.856167 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:37.856075    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:05:37.856167 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:37.856099    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:05:37.856167 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:37.856099    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:05:37.860642 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:37.860622    5713 scope.go:122] "RemoveContainer" containerID="12d2ee9d8baa72033da22dcff21970f476b48440d1e1cc69236d2471ba567fc5"
Oct 30 21:05:37.924292 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:37.924262    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj2bv\" (UniqueName: \"kubernetes.io/projected/ced570ba-f5f3-4c25-bc1d-0544046ed9dc-kube-api-access-kj2bv\") pod \"maas-api-revoke-keys-e2e-derive-bf9c0727-v9llb\" (UID: \"ced570ba-f5f3-4c25-bc1d-0544046ed9dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf9c0727-v9llb"
Oct 30 21:05:37.924517 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:37.924497    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ced570ba-f5f3-4c25-bc1d-0544046ed9dc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-bf9c0727-v9llb\" (UID: \"ced570ba-f5f3-4c25-bc1d-0544046ed9dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf9c0727-v9llb"
Oct 30 21:05:37.924672 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:37.924533    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ced570ba-f5f3-4c25-bc1d-0544046ed9dc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-bf9c0727-v9llb\" (UID: \"ced570ba-f5f3-4c25-bc1d-0544046ed9dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf9c0727-v9llb"
Oct 30 21:05:37.925594 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:37.925574    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ced570ba-f5f3-4c25-bc1d-0544046ed9dc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-bf9c0727-v9llb\" (UID: \"ced570ba-f5f3-4c25-bc1d-0544046ed9dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf9c0727-v9llb"
Oct 30 21:05:37.927256 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:37.927234    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ced570ba-f5f3-4c25-bc1d-0544046ed9dc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-bf9c0727-v9llb\" (UID: \"ced570ba-f5f3-4c25-bc1d-0544046ed9dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf9c0727-v9llb"
Oct 30 21:05:37.981456 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:37.981425    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj2bv\" (UniqueName: \"kubernetes.io/projected/ced570ba-f5f3-4c25-bc1d-0544046ed9dc-kube-api-access-kj2bv\") pod \"maas-api-revoke-keys-e2e-derive-bf9c0727-v9llb\" (UID: \"ced570ba-f5f3-4c25-bc1d-0544046ed9dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf9c0727-v9llb"
Oct 30 21:05:38.072219 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:38.072141    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf9c0727-v9llb"
Oct 30 21:05:38.478009 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:05:38.477973    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podced570ba_f5f3_4c25_bc1d_0544046ed9dc.slice/crio-46e2920bbf4aa1ec5aac7a27fe9be68b3e54b06826e7e8c730bf4c8a38ab1029 WatchSource:0}: Error finding container 46e2920bbf4aa1ec5aac7a27fe9be68b3e54b06826e7e8c730bf4c8a38ab1029: Status 404 returned error can't find the container with id 46e2920bbf4aa1ec5aac7a27fe9be68b3e54b06826e7e8c730bf4c8a38ab1029
Oct 30 21:05:38.478142 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:38.478051    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf9c0727-v9llb"]
Oct 30 21:05:38.658313 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:38.658283    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf9c0727-v9llb" event={"ID":"ced570ba-f5f3-4c25-bc1d-0544046ed9dc","Type":"ContainerStarted","Data":"46e2920bbf4aa1ec5aac7a27fe9be68b3e54b06826e7e8c730bf4c8a38ab1029"}
Oct 30 21:05:39.066624 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:39.066542    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1d682ca-81f6-4509-8440-243c337a4117" path="/var/lib/kubelet/pods/a1d682ca-81f6-4509-8440-243c337a4117/volumes"
Oct 30 21:05:39.067177 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:39.067155    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e92b3948-3eca-4062-ae3a-f2163d552b54" path="/var/lib/kubelet/pods/e92b3948-3eca-4062-ae3a-f2163d552b54/volumes"
Oct 30 21:05:39.464471 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:39.464448    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk"
Oct 30 21:05:39.580596 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:39.580562    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a73cfa4-d614-4c39-b324-731fca0564c4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a73cfa4-d614-4c39-b324-731fca0564c4-workload-certs\") pod \"9a73cfa4-d614-4c39-b324-731fca0564c4\" (UID: \"9a73cfa4-d614-4c39-b324-731fca0564c4\") "
Oct 30 21:05:39.580763 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:39.580618    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a73cfa4-d614-4c39-b324-731fca0564c4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a73cfa4-d614-4c39-b324-731fca0564c4-istiod-ca-cert\") pod \"9a73cfa4-d614-4c39-b324-731fca0564c4\" (UID: \"9a73cfa4-d614-4c39-b324-731fca0564c4\") "
Oct 30 21:05:39.580763 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:39.580681    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a73cfa4-d614-4c39-b324-731fca0564c4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a73cfa4-d614-4c39-b324-731fca0564c4-istio-envoy\") pod \"9a73cfa4-d614-4c39-b324-731fca0564c4\" (UID: \"9a73cfa4-d614-4c39-b324-731fca0564c4\") "
Oct 30 21:05:39.580763 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:39.580732    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a73cfa4-d614-4c39-b324-731fca0564c4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a73cfa4-d614-4c39-b324-731fca0564c4-istio-podinfo\") pod \"9a73cfa4-d614-4c39-b324-731fca0564c4\" (UID: \"9a73cfa4-d614-4c39-b324-731fca0564c4\") "
Oct 30 21:05:39.580763 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:39.580759    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a73cfa4-d614-4c39-b324-731fca0564c4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a73cfa4-d614-4c39-b324-731fca0564c4-workload-socket\") pod \"9a73cfa4-d614-4c39-b324-731fca0564c4\" (UID: \"9a73cfa4-d614-4c39-b324-731fca0564c4\") "
Oct 30 21:05:39.580993 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:39.580843    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a73cfa4-d614-4c39-b324-731fca0564c4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a73cfa4-d614-4c39-b324-731fca0564c4-istio-data\") pod \"9a73cfa4-d614-4c39-b324-731fca0564c4\" (UID: \"9a73cfa4-d614-4c39-b324-731fca0564c4\") "
Oct 30 21:05:39.580993 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:39.580892    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a73cfa4-d614-4c39-b324-731fca0564c4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a73cfa4-d614-4c39-b324-731fca0564c4-credential-socket\") pod \"9a73cfa4-d614-4c39-b324-731fca0564c4\" (UID: \"9a73cfa4-d614-4c39-b324-731fca0564c4\") "
Oct 30 21:05:39.580993 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:39.580902    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a73cfa4-d614-4c39-b324-731fca0564c4-workload-certs" pod "9a73cfa4-d614-4c39-b324-731fca0564c4" (UID: "9a73cfa4-d614-4c39-b324-731fca0564c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:05:39.581156 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:39.581044    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a73cfa4-d614-4c39-b324-731fca0564c4-istiod-ca-cert" pod "9a73cfa4-d614-4c39-b324-731fca0564c4" (UID: "9a73cfa4-d614-4c39-b324-731fca0564c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:05:39.581156 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:39.581087    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a73cfa4-d614-4c39-b324-731fca0564c4-kube-api-access-6snnr\" (UniqueName: \"kubernetes.io/projected/9a73cfa4-d614-4c39-b324-731fca0564c4-kube-api-access-6snnr\") pod \"9a73cfa4-d614-4c39-b324-731fca0564c4\" (UID: \"9a73cfa4-d614-4c39-b324-731fca0564c4\") "
Oct 30 21:05:39.581271 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:39.581143    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a73cfa4-d614-4c39-b324-731fca0564c4-workload-socket" pod "9a73cfa4-d614-4c39-b324-731fca0564c4" (UID: "9a73cfa4-d614-4c39-b324-731fca0564c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:05:39.581271 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:39.581167    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a73cfa4-d614-4c39-b324-731fca0564c4-istio-token\" (UniqueName: \"kubernetes.io/projected/9a73cfa4-d614-4c39-b324-731fca0564c4-istio-token\") pod \"9a73cfa4-d614-4c39-b324-731fca0564c4\" (UID: \"9a73cfa4-d614-4c39-b324-731fca0564c4\") "
Oct 30 21:05:39.581271 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:39.581205    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a73cfa4-d614-4c39-b324-731fca0564c4-istio-data" pod "9a73cfa4-d614-4c39-b324-731fca0564c4" (UID: "9a73cfa4-d614-4c39-b324-731fca0564c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:05:39.581271 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:39.581212    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a73cfa4-d614-4c39-b324-731fca0564c4-credential-socket" pod "9a73cfa4-d614-4c39-b324-731fca0564c4" (UID: "9a73cfa4-d614-4c39-b324-731fca0564c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:05:39.582183 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:39.582157    5713 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a73cfa4-d614-4c39-b324-731fca0564c4-workload-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:39.582295 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:39.582186    5713 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a73cfa4-d614-4c39-b324-731fca0564c4-istiod-ca-cert\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:39.582295 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:39.582201    5713 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a73cfa4-d614-4c39-b324-731fca0564c4-workload-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:39.582295 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:39.582218    5713 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a73cfa4-d614-4c39-b324-731fca0564c4-istio-data\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:39.582295 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:39.582235    5713 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a73cfa4-d614-4c39-b324-731fca0564c4-credential-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:39.583304 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:39.583278    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a73cfa4-d614-4c39-b324-731fca0564c4-istio-envoy" pod "9a73cfa4-d614-4c39-b324-731fca0564c4" (UID: "9a73cfa4-d614-4c39-b324-731fca0564c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:05:39.583304 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:39.583287    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a73cfa4-d614-4c39-b324-731fca0564c4-istio-podinfo" pod "9a73cfa4-d614-4c39-b324-731fca0564c4" (UID: "9a73cfa4-d614-4c39-b324-731fca0564c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:05:39.583451 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:39.583352    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a73cfa4-d614-4c39-b324-731fca0564c4-kube-api-access-6snnr" pod "9a73cfa4-d614-4c39-b324-731fca0564c4" (UID: "9a73cfa4-d614-4c39-b324-731fca0564c4"). InnerVolumeSpecName "kube-api-access-6snnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:05:39.583520 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:39.583453    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a73cfa4-d614-4c39-b324-731fca0564c4-istio-token" pod "9a73cfa4-d614-4c39-b324-731fca0564c4" (UID: "9a73cfa4-d614-4c39-b324-731fca0564c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:05:39.670889 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:39.670763    5713 generic.go:356] "Generic (PLEG): container finished" podID="ced570ba-f5f3-4c25-bc1d-0544046ed9dc" containerID="1dcae5b4d62b3b70ef7e3b26b089786d1f2fd0aba75fc3da23173c4ffaff3b20" exitCode=0
Oct 30 21:05:39.670889 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:39.670852    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf9c0727-v9llb" event={"ID":"ced570ba-f5f3-4c25-bc1d-0544046ed9dc","Type":"ContainerDied","Data":"1dcae5b4d62b3b70ef7e3b26b089786d1f2fd0aba75fc3da23173c4ffaff3b20"}
Oct 30 21:05:39.672673 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:39.672646    5713 generic.go:356] "Generic (PLEG): container finished" podID="9a73cfa4-d614-4c39-b324-731fca0564c4" containerID="c1ec5df9fca78a32bc25d54802c6a63c1a397a64889906056dfba49017d9ad30" exitCode=0
Oct 30 21:05:39.672859 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:39.672693    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk"
Oct 30 21:05:39.672859 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:39.672727    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk" event={"ID":"9a73cfa4-d614-4c39-b324-731fca0564c4","Type":"ContainerDied","Data":"c1ec5df9fca78a32bc25d54802c6a63c1a397a64889906056dfba49017d9ad30"}
Oct 30 21:05:39.672859 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:39.672809    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk" event={"ID":"9a73cfa4-d614-4c39-b324-731fca0564c4","Type":"ContainerDied","Data":"7a1036f5812038a461d162457f0be1a35c1fa7d5219ea6130ab8395109b3ab1c"}
Oct 30 21:05:39.672859 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:39.672850    5713 scope.go:122] "RemoveContainer" containerID="c1ec5df9fca78a32bc25d54802c6a63c1a397a64889906056dfba49017d9ad30"
Oct 30 21:05:39.683557 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:39.683515    5713 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a73cfa4-d614-4c39-b324-731fca0564c4-istio-podinfo\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:39.683557 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:39.683538    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6snnr\" (UniqueName: \"kubernetes.io/projected/9a73cfa4-d614-4c39-b324-731fca0564c4-kube-api-access-6snnr\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:39.683557 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:39.683554    5713 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a73cfa4-d614-4c39-b324-731fca0564c4-istio-token\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:39.683864 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:39.683567    5713 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a73cfa4-d614-4c39-b324-731fca0564c4-istio-envoy\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:39.691721 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:39.691694    5713 scope.go:122] "RemoveContainer" containerID="c1ec5df9fca78a32bc25d54802c6a63c1a397a64889906056dfba49017d9ad30"
Oct 30 21:05:39.692456 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:05:39.692256    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1ec5df9fca78a32bc25d54802c6a63c1a397a64889906056dfba49017d9ad30\": container with ID starting with c1ec5df9fca78a32bc25d54802c6a63c1a397a64889906056dfba49017d9ad30 not found: ID does not exist" containerID="c1ec5df9fca78a32bc25d54802c6a63c1a397a64889906056dfba49017d9ad30"
Oct 30 21:05:39.692456 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:39.692299    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1ec5df9fca78a32bc25d54802c6a63c1a397a64889906056dfba49017d9ad30"} err="failed to get container status \"c1ec5df9fca78a32bc25d54802c6a63c1a397a64889906056dfba49017d9ad30\": rpc error: code = NotFound desc = could not find container \"c1ec5df9fca78a32bc25d54802c6a63c1a397a64889906056dfba49017d9ad30\": container with ID starting with c1ec5df9fca78a32bc25d54802c6a63c1a397a64889906056dfba49017d9ad30 not found: ID does not exist"
Oct 30 21:05:39.910258 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:39.910216    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk"]
Oct 30 21:05:39.916655 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:39.916628    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-wd8jk"]
Oct 30 21:05:40.937723 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:40.937659    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf9c0727-v9llb"
Oct 30 21:05:41.004930 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:41.004902    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ced570ba-f5f3-4c25-bc1d-0544046ed9dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ced570ba-f5f3-4c25-bc1d-0544046ed9dc-maas-api-xks-ca\") pod \"ced570ba-f5f3-4c25-bc1d-0544046ed9dc\" (UID: \"ced570ba-f5f3-4c25-bc1d-0544046ed9dc\") "
Oct 30 21:05:41.005097 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:41.005033    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ced570ba-f5f3-4c25-bc1d-0544046ed9dc-kube-api-access-kj2bv\" (UniqueName: \"kubernetes.io/projected/ced570ba-f5f3-4c25-bc1d-0544046ed9dc-kube-api-access-kj2bv\") pod \"ced570ba-f5f3-4c25-bc1d-0544046ed9dc\" (UID: \"ced570ba-f5f3-4c25-bc1d-0544046ed9dc\") "
Oct 30 21:05:41.005139 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:41.005095    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ced570ba-f5f3-4c25-bc1d-0544046ed9dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ced570ba-f5f3-4c25-bc1d-0544046ed9dc-maas-api-openshift-ca\") pod \"ced570ba-f5f3-4c25-bc1d-0544046ed9dc\" (UID: \"ced570ba-f5f3-4c25-bc1d-0544046ed9dc\") "
Oct 30 21:05:41.005507 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:41.005479    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ced570ba-f5f3-4c25-bc1d-0544046ed9dc-maas-api-openshift-ca" pod "ced570ba-f5f3-4c25-bc1d-0544046ed9dc" (UID: "ced570ba-f5f3-4c25-bc1d-0544046ed9dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:05:41.006036 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:41.006015    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ced570ba-f5f3-4c25-bc1d-0544046ed9dc-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:41.007328 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:41.007302    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ced570ba-f5f3-4c25-bc1d-0544046ed9dc-kube-api-access-kj2bv" pod "ced570ba-f5f3-4c25-bc1d-0544046ed9dc" (UID: "ced570ba-f5f3-4c25-bc1d-0544046ed9dc"). InnerVolumeSpecName "kube-api-access-kj2bv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:05:41.007328 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:41.007302    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ced570ba-f5f3-4c25-bc1d-0544046ed9dc-maas-api-xks-ca" pod "ced570ba-f5f3-4c25-bc1d-0544046ed9dc" (UID: "ced570ba-f5f3-4c25-bc1d-0544046ed9dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:05:41.066646 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:41.066066    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a73cfa4-d614-4c39-b324-731fca0564c4" path="/var/lib/kubelet/pods/9a73cfa4-d614-4c39-b324-731fca0564c4/volumes"
Oct 30 21:05:41.107249 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:41.107221    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ced570ba-f5f3-4c25-bc1d-0544046ed9dc-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:41.107249 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:41.107248    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kj2bv\" (UniqueName: \"kubernetes.io/projected/ced570ba-f5f3-4c25-bc1d-0544046ed9dc-kube-api-access-kj2bv\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:41.693028 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:41.692999    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf9c0727-v9llb"
Oct 30 21:05:41.693210 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:41.693022    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf9c0727-v9llb" event={"ID":"ced570ba-f5f3-4c25-bc1d-0544046ed9dc","Type":"ContainerDied","Data":"46e2920bbf4aa1ec5aac7a27fe9be68b3e54b06826e7e8c730bf4c8a38ab1029"}
Oct 30 21:05:41.693210 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:41.693098    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46e2920bbf4aa1ec5aac7a27fe9be68b3e54b06826e7e8c730bf4c8a38ab1029"
Oct 30 21:05:45.048721 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:45.048687    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4614c1a-q6hld"]
Oct 30 21:05:45.098430 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:45.098391    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4614c1a-q6hld"]
Oct 30 21:05:47.066167 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:47.066092    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f2bc097-ac83-4106-91aa-a50abf5fc9c6" path="/var/lib/kubelet/pods/7f2bc097-ac83-4106-91aa-a50abf5fc9c6/volumes"
Oct 30 21:05:48.457032 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:48.457002    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-bf9c0727-8d8698f4d-hjmdn"]
Oct 30 21:05:49.049185 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:49.049153    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf186c09-hcr7j"]
Oct 30 21:05:49.054317 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:49.054289    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf186c09-hcr7j"]
Oct 30 21:05:49.068996 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:49.068953    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96e75be8-c20d-4f62-9956-5ee10ea210ac" path="/var/lib/kubelet/pods/96e75be8-c20d-4f62-9956-5ee10ea210ac/volumes"
Oct 30 21:05:52.048831 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:52.048765    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6d5ec58-5k6ft"]
Oct 30 21:05:52.098268 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:52.098240    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6d5ec58-5k6ft"]
Oct 30 21:05:53.065966 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:53.065932    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98feba90-5e8f-443e-8c3e-0cfb79875caa" path="/var/lib/kubelet/pods/98feba90-5e8f-443e-8c3e-0cfb79875caa/volumes"
Oct 30 21:05:53.457776 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:53.457732    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-bf9c0727-8d8698f4d-hjmdn" podUID="f5478d17-b619-43ac-8f68-2675932b4abe" containerName="payload-processing" containerID="cri-o://905415ba78668542403721985f468689cdc2ab6d86bcf9767bafe371996f364b" gracePeriod=25
Oct 30 21:05:53.755701 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:53.755677    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-bf9c0727-8d8698f4d-hjmdn"
Oct 30 21:05:53.758839 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:53.758772    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5478d17-b619-43ac-8f68-2675932b4abe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5478d17-b619-43ac-8f68-2675932b4abe-plugins-config-volume\") pod \"f5478d17-b619-43ac-8f68-2675932b4abe\" (UID: \"f5478d17-b619-43ac-8f68-2675932b4abe\") "
Oct 30 21:05:53.759036 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:53.759020    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5478d17-b619-43ac-8f68-2675932b4abe-kube-api-access-fjlj8\" (UniqueName: \"kubernetes.io/projected/f5478d17-b619-43ac-8f68-2675932b4abe-kube-api-access-fjlj8\") pod \"f5478d17-b619-43ac-8f68-2675932b4abe\" (UID: \"f5478d17-b619-43ac-8f68-2675932b4abe\") "
Oct 30 21:05:53.759101 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:53.759080    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5478d17-b619-43ac-8f68-2675932b4abe-plugins-config-volume" pod "f5478d17-b619-43ac-8f68-2675932b4abe" (UID: "f5478d17-b619-43ac-8f68-2675932b4abe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:05:53.759655 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:53.759638    5713 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5478d17-b619-43ac-8f68-2675932b4abe-plugins-config-volume\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:53.761281 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:53.761259    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5478d17-b619-43ac-8f68-2675932b4abe-kube-api-access-fjlj8" pod "f5478d17-b619-43ac-8f68-2675932b4abe" (UID: "f5478d17-b619-43ac-8f68-2675932b4abe"). InnerVolumeSpecName "kube-api-access-fjlj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:05:53.804151 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:53.804109    5713 generic.go:356] "Generic (PLEG): container finished" podID="f5478d17-b619-43ac-8f68-2675932b4abe" containerID="905415ba78668542403721985f468689cdc2ab6d86bcf9767bafe371996f364b" exitCode=0
Oct 30 21:05:53.804283 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:53.804168    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-bf9c0727-8d8698f4d-hjmdn"
Oct 30 21:05:53.804283 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:53.804189    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-bf9c0727-8d8698f4d-hjmdn" event={"ID":"f5478d17-b619-43ac-8f68-2675932b4abe","Type":"ContainerDied","Data":"905415ba78668542403721985f468689cdc2ab6d86bcf9767bafe371996f364b"}
Oct 30 21:05:53.804283 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:53.804245    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-bf9c0727-8d8698f4d-hjmdn" event={"ID":"f5478d17-b619-43ac-8f68-2675932b4abe","Type":"ContainerDied","Data":"c347c5721e9c4b1fba7c6e8ca8306b359036ffc1b714bf8af80d6fc5469981c0"}
Oct 30 21:05:53.804283 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:53.804267    5713 scope.go:122] "RemoveContainer" containerID="905415ba78668542403721985f468689cdc2ab6d86bcf9767bafe371996f364b"
Oct 30 21:05:53.816049 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:53.816030    5713 scope.go:122] "RemoveContainer" containerID="905415ba78668542403721985f468689cdc2ab6d86bcf9767bafe371996f364b"
Oct 30 21:05:53.816346 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:05:53.816322    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"905415ba78668542403721985f468689cdc2ab6d86bcf9767bafe371996f364b\": container with ID starting with 905415ba78668542403721985f468689cdc2ab6d86bcf9767bafe371996f364b not found: ID does not exist" containerID="905415ba78668542403721985f468689cdc2ab6d86bcf9767bafe371996f364b"
Oct 30 21:05:53.816448 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:53.816351    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"905415ba78668542403721985f468689cdc2ab6d86bcf9767bafe371996f364b"} err="failed to get container status \"905415ba78668542403721985f468689cdc2ab6d86bcf9767bafe371996f364b\": rpc error: code = NotFound desc = could not find container \"905415ba78668542403721985f468689cdc2ab6d86bcf9767bafe371996f364b\": container with ID starting with 905415ba78668542403721985f468689cdc2ab6d86bcf9767bafe371996f364b not found: ID does not exist"
Oct 30 21:05:53.860610 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:53.860584    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjlj8\" (UniqueName: \"kubernetes.io/projected/f5478d17-b619-43ac-8f68-2675932b4abe-kube-api-access-fjlj8\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:05:53.988598 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:53.988514    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-bf9c0727-8d8698f4d-hjmdn"]
Oct 30 21:05:53.993041 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:53.993008    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-bf9c0727-8d8698f4d-hjmdn"]
Oct 30 21:05:54.460089 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:05:54.460050    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:05:55.046212 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:55.046185    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb69c39a-6ctbl"]
Oct 30 21:05:55.064850 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:55.064821    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5478d17-b619-43ac-8f68-2675932b4abe" path="/var/lib/kubelet/pods/f5478d17-b619-43ac-8f68-2675932b4abe/volumes"
Oct 30 21:05:55.094559 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:55.094532    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb69c39a-6ctbl"]
Oct 30 21:05:55.886428 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:55.886387    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2cf21005-548d5fccfb-kzfwl"]
Oct 30 21:05:56.047584 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:56.047548    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2cf21005-55df98b7dc-h466s"]
Oct 30 21:05:57.068610 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:57.068574    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ddf3248-69bc-458b-b12e-344bc0cb7ef8" path="/var/lib/kubelet/pods/0ddf3248-69bc-458b-b12e-344bc0cb7ef8/volumes"
Oct 30 21:05:57.083814 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:57.083449    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12aab4d8-ntbqw"]
Oct 30 21:05:57.097017 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:57.096983    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12aab4d8-ntbqw"]
Oct 30 21:05:59.065238 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:05:59.065192    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b827544-d79c-42dc-aeb5-12610c1c4674" path="/var/lib/kubelet/pods/5b827544-d79c-42dc-aeb5-12610c1c4674/volumes"
Oct 30 21:06:00.887704 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:00.887647    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-2cf21005-548d5fccfb-kzfwl" podUID="427927da-96f1-464c-9a58-1116d61bee68" containerName="payload-processing" containerID="cri-o://cd3ef5c457b8ec624cad2c4be51f2531bd02bf77f1cdff4439ef9b03453e0cb2" gracePeriod=25
Oct 30 21:06:01.048024 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:01.047984    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-2cf21005-55df98b7dc-h466s" podUID="bdc39409-a6df-4bed-8017-a6f502bb7b44" containerName="payload-pre-processing" containerID="cri-o://a179c70d401dc1bbd0a6d76798061d5d26c509f32d643f4e83035290ac0af0ed" gracePeriod=25
Oct 30 21:06:01.181464 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:01.181437    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2cf21005-548d5fccfb-kzfwl"
Oct 30 21:06:01.281506 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:01.281463    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/427927da-96f1-464c-9a58-1116d61bee68-kube-api-access-fd6k4\" (UniqueName: \"kubernetes.io/projected/427927da-96f1-464c-9a58-1116d61bee68-kube-api-access-fd6k4\") pod \"427927da-96f1-464c-9a58-1116d61bee68\" (UID: \"427927da-96f1-464c-9a58-1116d61bee68\") "
Oct 30 21:06:01.281693 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:01.281619    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/427927da-96f1-464c-9a58-1116d61bee68-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/427927da-96f1-464c-9a58-1116d61bee68-plugins-config-volume\") pod \"427927da-96f1-464c-9a58-1116d61bee68\" (UID: \"427927da-96f1-464c-9a58-1116d61bee68\") "
Oct 30 21:06:01.282878 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:01.282854    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/427927da-96f1-464c-9a58-1116d61bee68-plugins-config-volume" pod "427927da-96f1-464c-9a58-1116d61bee68" (UID: "427927da-96f1-464c-9a58-1116d61bee68"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:06:01.295021 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:01.294976    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/427927da-96f1-464c-9a58-1116d61bee68-kube-api-access-fd6k4" pod "427927da-96f1-464c-9a58-1116d61bee68" (UID: "427927da-96f1-464c-9a58-1116d61bee68"). InnerVolumeSpecName "kube-api-access-fd6k4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:06:01.322473 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:01.322445    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2cf21005-55df98b7dc-h466s"
Oct 30 21:06:01.383218 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:01.383189    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdc39409-a6df-4bed-8017-a6f502bb7b44-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdc39409-a6df-4bed-8017-a6f502bb7b44-plugins-config-volume\") pod \"bdc39409-a6df-4bed-8017-a6f502bb7b44\" (UID: \"bdc39409-a6df-4bed-8017-a6f502bb7b44\") "
Oct 30 21:06:01.383391 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:01.383362    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdc39409-a6df-4bed-8017-a6f502bb7b44-kube-api-access-65hck\" (UniqueName: \"kubernetes.io/projected/bdc39409-a6df-4bed-8017-a6f502bb7b44-kube-api-access-65hck\") pod \"bdc39409-a6df-4bed-8017-a6f502bb7b44\" (UID: \"bdc39409-a6df-4bed-8017-a6f502bb7b44\") "
Oct 30 21:06:01.383647 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:01.383610    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdc39409-a6df-4bed-8017-a6f502bb7b44-plugins-config-volume" pod "bdc39409-a6df-4bed-8017-a6f502bb7b44" (UID: "bdc39409-a6df-4bed-8017-a6f502bb7b44"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:06:01.384089 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:01.384070    5713 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/427927da-96f1-464c-9a58-1116d61bee68-plugins-config-volume\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:01.384166 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:01.384090    5713 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdc39409-a6df-4bed-8017-a6f502bb7b44-plugins-config-volume\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:01.384166 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:01.384101    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fd6k4\" (UniqueName: \"kubernetes.io/projected/427927da-96f1-464c-9a58-1116d61bee68-kube-api-access-fd6k4\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:01.385555 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:01.385532    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdc39409-a6df-4bed-8017-a6f502bb7b44-kube-api-access-65hck" pod "bdc39409-a6df-4bed-8017-a6f502bb7b44" (UID: "bdc39409-a6df-4bed-8017-a6f502bb7b44"). InnerVolumeSpecName "kube-api-access-65hck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:06:01.485657 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:01.485626    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65hck\" (UniqueName: \"kubernetes.io/projected/bdc39409-a6df-4bed-8017-a6f502bb7b44-kube-api-access-65hck\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:01.875494 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:01.875404    5713 generic.go:356] "Generic (PLEG): container finished" podID="bdc39409-a6df-4bed-8017-a6f502bb7b44" containerID="a179c70d401dc1bbd0a6d76798061d5d26c509f32d643f4e83035290ac0af0ed" exitCode=0
Oct 30 21:06:01.875494 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:01.875466    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2cf21005-55df98b7dc-h466s"
Oct 30 21:06:01.875740 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:01.875485    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2cf21005-55df98b7dc-h466s" event={"ID":"bdc39409-a6df-4bed-8017-a6f502bb7b44","Type":"ContainerDied","Data":"a179c70d401dc1bbd0a6d76798061d5d26c509f32d643f4e83035290ac0af0ed"}
Oct 30 21:06:01.875740 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:01.875571    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2cf21005-55df98b7dc-h466s" event={"ID":"bdc39409-a6df-4bed-8017-a6f502bb7b44","Type":"ContainerDied","Data":"1d97e9d169e39adf96109033c17843b04f64e356d3712b36ab0c9b2672955663"}
Oct 30 21:06:01.875740 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:01.875602    5713 scope.go:122] "RemoveContainer" containerID="a179c70d401dc1bbd0a6d76798061d5d26c509f32d643f4e83035290ac0af0ed"
Oct 30 21:06:01.877037 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:01.877012    5713 generic.go:356] "Generic (PLEG): container finished" podID="427927da-96f1-464c-9a58-1116d61bee68" containerID="cd3ef5c457b8ec624cad2c4be51f2531bd02bf77f1cdff4439ef9b03453e0cb2" exitCode=0
Oct 30 21:06:01.877145 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:01.877072    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2cf21005-548d5fccfb-kzfwl"
Oct 30 21:06:01.877145 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:01.877098    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2cf21005-548d5fccfb-kzfwl" event={"ID":"427927da-96f1-464c-9a58-1116d61bee68","Type":"ContainerDied","Data":"cd3ef5c457b8ec624cad2c4be51f2531bd02bf77f1cdff4439ef9b03453e0cb2"}
Oct 30 21:06:01.877264 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:01.877142    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2cf21005-548d5fccfb-kzfwl" event={"ID":"427927da-96f1-464c-9a58-1116d61bee68","Type":"ContainerDied","Data":"cb3edbd7c4de3c9192509e0d1a200d479a058681f1d058ff4d4ac79e177c58ee"}
Oct 30 21:06:01.889580 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:01.889455    5713 scope.go:122] "RemoveContainer" containerID="a179c70d401dc1bbd0a6d76798061d5d26c509f32d643f4e83035290ac0af0ed"
Oct 30 21:06:01.889810 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:06:01.889754    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a179c70d401dc1bbd0a6d76798061d5d26c509f32d643f4e83035290ac0af0ed\": container with ID starting with a179c70d401dc1bbd0a6d76798061d5d26c509f32d643f4e83035290ac0af0ed not found: ID does not exist" containerID="a179c70d401dc1bbd0a6d76798061d5d26c509f32d643f4e83035290ac0af0ed"
Oct 30 21:06:01.889862 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:01.889778    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a179c70d401dc1bbd0a6d76798061d5d26c509f32d643f4e83035290ac0af0ed"} err="failed to get container status \"a179c70d401dc1bbd0a6d76798061d5d26c509f32d643f4e83035290ac0af0ed\": rpc error: code = NotFound desc = could not find container \"a179c70d401dc1bbd0a6d76798061d5d26c509f32d643f4e83035290ac0af0ed\": container with ID starting with a179c70d401dc1bbd0a6d76798061d5d26c509f32d643f4e83035290ac0af0ed not found: ID does not exist"
Oct 30 21:06:01.889862 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:01.889830    5713 scope.go:122] "RemoveContainer" containerID="cd3ef5c457b8ec624cad2c4be51f2531bd02bf77f1cdff4439ef9b03453e0cb2"
Oct 30 21:06:01.902097 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:01.902079    5713 scope.go:122] "RemoveContainer" containerID="cd3ef5c457b8ec624cad2c4be51f2531bd02bf77f1cdff4439ef9b03453e0cb2"
Oct 30 21:06:01.902401 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:06:01.902380    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd3ef5c457b8ec624cad2c4be51f2531bd02bf77f1cdff4439ef9b03453e0cb2\": container with ID starting with cd3ef5c457b8ec624cad2c4be51f2531bd02bf77f1cdff4439ef9b03453e0cb2 not found: ID does not exist" containerID="cd3ef5c457b8ec624cad2c4be51f2531bd02bf77f1cdff4439ef9b03453e0cb2"
Oct 30 21:06:01.902522 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:01.902404    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd3ef5c457b8ec624cad2c4be51f2531bd02bf77f1cdff4439ef9b03453e0cb2"} err="failed to get container status \"cd3ef5c457b8ec624cad2c4be51f2531bd02bf77f1cdff4439ef9b03453e0cb2\": rpc error: code = NotFound desc = could not find container \"cd3ef5c457b8ec624cad2c4be51f2531bd02bf77f1cdff4439ef9b03453e0cb2\": container with ID starting with cd3ef5c457b8ec624cad2c4be51f2531bd02bf77f1cdff4439ef9b03453e0cb2 not found: ID does not exist"
Oct 30 21:06:02.042312 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:02.042275    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2cf21005-548d5fccfb-kzfwl"]
Oct 30 21:06:02.046427 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:02.046399    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2cf21005-548d5fccfb-kzfwl"]
Oct 30 21:06:02.092766 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:02.092731    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6946eef6-f5z25"]
Oct 30 21:06:02.096761 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:02.096736    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6946eef6-f5z25"]
Oct 30 21:06:02.154026 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:02.153942    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2cf21005-55df98b7dc-h466s"]
Oct 30 21:06:02.157818 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:02.157782    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2cf21005-55df98b7dc-h466s"]
Oct 30 21:06:03.065551 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:03.065520    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="262866a6-b173-44f4-bebb-b129d4372f61" path="/var/lib/kubelet/pods/262866a6-b173-44f4-bebb-b129d4372f61/volumes"
Oct 30 21:06:03.066057 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:03.066040    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="427927da-96f1-464c-9a58-1116d61bee68" path="/var/lib/kubelet/pods/427927da-96f1-464c-9a58-1116d61bee68/volumes"
Oct 30 21:06:03.066474 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:03.066456    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdc39409-a6df-4bed-8017-a6f502bb7b44" path="/var/lib/kubelet/pods/bdc39409-a6df-4bed-8017-a6f502bb7b44/volumes"
Oct 30 21:06:04.354951 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:04.354912    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8"]
Oct 30 21:06:04.355471 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:04.355273    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8" podUID="03ef9f7c-175a-4012-a021-ec011e37928e" containerName="istio-proxy" containerID="cri-o://669acbc53462750b4744dab1559955a86540a29cb8282e3f2f3b70ec1f5b2798" gracePeriod=30
Oct 30 21:06:04.395501 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:04.395463    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq"]
Oct 30 21:06:04.396082 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:04.396043    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq" podUID="8b28543a-e85d-4593-93c9-44f0d5b1016a" containerName="istio-proxy" containerID="cri-o://51fd22ecb1c16e833fc4935f631f0d9a9b91dff21878c140abbae1cb3dab8f64" gracePeriod=30
Oct 30 21:06:09.553153 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.553121    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8"
Oct 30 21:06:09.670328 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.670293    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03ef9f7c-175a-4012-a021-ec011e37928e-kube-api-access-j9rtk\" (UniqueName: \"kubernetes.io/projected/03ef9f7c-175a-4012-a021-ec011e37928e-kube-api-access-j9rtk\") pod \"03ef9f7c-175a-4012-a021-ec011e37928e\" (UID: \"03ef9f7c-175a-4012-a021-ec011e37928e\") "
Oct 30 21:06:09.670434 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.670349    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03ef9f7c-175a-4012-a021-ec011e37928e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03ef9f7c-175a-4012-a021-ec011e37928e-workload-certs\") pod \"03ef9f7c-175a-4012-a021-ec011e37928e\" (UID: \"03ef9f7c-175a-4012-a021-ec011e37928e\") "
Oct 30 21:06:09.670483 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.670455    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03ef9f7c-175a-4012-a021-ec011e37928e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ef9f7c-175a-4012-a021-ec011e37928e-workload-socket\") pod \"03ef9f7c-175a-4012-a021-ec011e37928e\" (UID: \"03ef9f7c-175a-4012-a021-ec011e37928e\") "
Oct 30 21:06:09.670526 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.670496    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03ef9f7c-175a-4012-a021-ec011e37928e-istio-token\" (UniqueName: \"kubernetes.io/projected/03ef9f7c-175a-4012-a021-ec011e37928e-istio-token\") pod \"03ef9f7c-175a-4012-a021-ec011e37928e\" (UID: \"03ef9f7c-175a-4012-a021-ec011e37928e\") "
Oct 30 21:06:09.670587 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.670524    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03ef9f7c-175a-4012-a021-ec011e37928e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03ef9f7c-175a-4012-a021-ec011e37928e-istio-podinfo\") pod \"03ef9f7c-175a-4012-a021-ec011e37928e\" (UID: \"03ef9f7c-175a-4012-a021-ec011e37928e\") "
Oct 30 21:06:09.670681 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.670659    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03ef9f7c-175a-4012-a021-ec011e37928e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ef9f7c-175a-4012-a021-ec011e37928e-credential-socket\") pod \"03ef9f7c-175a-4012-a021-ec011e37928e\" (UID: \"03ef9f7c-175a-4012-a021-ec011e37928e\") "
Oct 30 21:06:09.670832 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.670678    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03ef9f7c-175a-4012-a021-ec011e37928e-workload-certs" pod "03ef9f7c-175a-4012-a021-ec011e37928e" (UID: "03ef9f7c-175a-4012-a021-ec011e37928e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:09.670832 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.670701    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03ef9f7c-175a-4012-a021-ec011e37928e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03ef9f7c-175a-4012-a021-ec011e37928e-istiod-ca-cert\") pod \"03ef9f7c-175a-4012-a021-ec011e37928e\" (UID: \"03ef9f7c-175a-4012-a021-ec011e37928e\") "
Oct 30 21:06:09.670832 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.670781    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03ef9f7c-175a-4012-a021-ec011e37928e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03ef9f7c-175a-4012-a021-ec011e37928e-istio-data\") pod \"03ef9f7c-175a-4012-a021-ec011e37928e\" (UID: \"03ef9f7c-175a-4012-a021-ec011e37928e\") "
Oct 30 21:06:09.671019 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.670844    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03ef9f7c-175a-4012-a021-ec011e37928e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03ef9f7c-175a-4012-a021-ec011e37928e-istio-envoy\") pod \"03ef9f7c-175a-4012-a021-ec011e37928e\" (UID: \"03ef9f7c-175a-4012-a021-ec011e37928e\") "
Oct 30 21:06:09.671019 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.670882    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03ef9f7c-175a-4012-a021-ec011e37928e-workload-socket" pod "03ef9f7c-175a-4012-a021-ec011e37928e" (UID: "03ef9f7c-175a-4012-a021-ec011e37928e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:09.671019 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.670969    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03ef9f7c-175a-4012-a021-ec011e37928e-credential-socket" pod "03ef9f7c-175a-4012-a021-ec011e37928e" (UID: "03ef9f7c-175a-4012-a021-ec011e37928e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:09.671166 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.671097    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03ef9f7c-175a-4012-a021-ec011e37928e-istiod-ca-cert" pod "03ef9f7c-175a-4012-a021-ec011e37928e" (UID: "03ef9f7c-175a-4012-a021-ec011e37928e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:06:09.671358 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.671334    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03ef9f7c-175a-4012-a021-ec011e37928e-istio-data" pod "03ef9f7c-175a-4012-a021-ec011e37928e" (UID: "03ef9f7c-175a-4012-a021-ec011e37928e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:09.672216 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.671626    5713 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ef9f7c-175a-4012-a021-ec011e37928e-workload-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:09.672216 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.671652    5713 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ef9f7c-175a-4012-a021-ec011e37928e-credential-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:09.672216 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.671668    5713 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03ef9f7c-175a-4012-a021-ec011e37928e-istiod-ca-cert\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:09.672216 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.671679    5713 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03ef9f7c-175a-4012-a021-ec011e37928e-istio-data\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:09.672216 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.671689    5713 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03ef9f7c-175a-4012-a021-ec011e37928e-workload-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:09.673227 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.673187    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03ef9f7c-175a-4012-a021-ec011e37928e-istio-podinfo" pod "03ef9f7c-175a-4012-a021-ec011e37928e" (UID: "03ef9f7c-175a-4012-a021-ec011e37928e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:06:09.673540 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.673521    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03ef9f7c-175a-4012-a021-ec011e37928e-istio-envoy" pod "03ef9f7c-175a-4012-a021-ec011e37928e" (UID: "03ef9f7c-175a-4012-a021-ec011e37928e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:09.673603 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.673539    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03ef9f7c-175a-4012-a021-ec011e37928e-istio-token" pod "03ef9f7c-175a-4012-a021-ec011e37928e" (UID: "03ef9f7c-175a-4012-a021-ec011e37928e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:06:09.673603 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.673565    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03ef9f7c-175a-4012-a021-ec011e37928e-kube-api-access-j9rtk" pod "03ef9f7c-175a-4012-a021-ec011e37928e" (UID: "03ef9f7c-175a-4012-a021-ec011e37928e"). InnerVolumeSpecName "kube-api-access-j9rtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:06:09.676169 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.676148    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq"
Oct 30 21:06:09.772770 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.772741    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b28543a-e85d-4593-93c9-44f0d5b1016a-istio-token\" (UniqueName: \"kubernetes.io/projected/8b28543a-e85d-4593-93c9-44f0d5b1016a-istio-token\") pod \"8b28543a-e85d-4593-93c9-44f0d5b1016a\" (UID: \"8b28543a-e85d-4593-93c9-44f0d5b1016a\") "
Oct 30 21:06:09.772983 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.772835    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b28543a-e85d-4593-93c9-44f0d5b1016a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b28543a-e85d-4593-93c9-44f0d5b1016a-workload-socket\") pod \"8b28543a-e85d-4593-93c9-44f0d5b1016a\" (UID: \"8b28543a-e85d-4593-93c9-44f0d5b1016a\") "
Oct 30 21:06:09.772983 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.772867    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b28543a-e85d-4593-93c9-44f0d5b1016a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b28543a-e85d-4593-93c9-44f0d5b1016a-istio-envoy\") pod \"8b28543a-e85d-4593-93c9-44f0d5b1016a\" (UID: \"8b28543a-e85d-4593-93c9-44f0d5b1016a\") "
Oct 30 21:06:09.773100 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.773069    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b28543a-e85d-4593-93c9-44f0d5b1016a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b28543a-e85d-4593-93c9-44f0d5b1016a-credential-socket\") pod \"8b28543a-e85d-4593-93c9-44f0d5b1016a\" (UID: \"8b28543a-e85d-4593-93c9-44f0d5b1016a\") "
Oct 30 21:06:09.773941 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.773209    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b28543a-e85d-4593-93c9-44f0d5b1016a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b28543a-e85d-4593-93c9-44f0d5b1016a-workload-certs\") pod \"8b28543a-e85d-4593-93c9-44f0d5b1016a\" (UID: \"8b28543a-e85d-4593-93c9-44f0d5b1016a\") "
Oct 30 21:06:09.773941 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.773326    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b28543a-e85d-4593-93c9-44f0d5b1016a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b28543a-e85d-4593-93c9-44f0d5b1016a-istiod-ca-cert\") pod \"8b28543a-e85d-4593-93c9-44f0d5b1016a\" (UID: \"8b28543a-e85d-4593-93c9-44f0d5b1016a\") "
Oct 30 21:06:09.773941 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.773371    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b28543a-e85d-4593-93c9-44f0d5b1016a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b28543a-e85d-4593-93c9-44f0d5b1016a-istio-podinfo\") pod \"8b28543a-e85d-4593-93c9-44f0d5b1016a\" (UID: \"8b28543a-e85d-4593-93c9-44f0d5b1016a\") "
Oct 30 21:06:09.773941 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.773436    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b28543a-e85d-4593-93c9-44f0d5b1016a-kube-api-access-xsr8f\" (UniqueName: \"kubernetes.io/projected/8b28543a-e85d-4593-93c9-44f0d5b1016a-kube-api-access-xsr8f\") pod \"8b28543a-e85d-4593-93c9-44f0d5b1016a\" (UID: \"8b28543a-e85d-4593-93c9-44f0d5b1016a\") "
Oct 30 21:06:09.773941 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.773466    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b28543a-e85d-4593-93c9-44f0d5b1016a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b28543a-e85d-4593-93c9-44f0d5b1016a-istio-data\") pod \"8b28543a-e85d-4593-93c9-44f0d5b1016a\" (UID: \"8b28543a-e85d-4593-93c9-44f0d5b1016a\") "
Oct 30 21:06:09.773941 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.773584    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b28543a-e85d-4593-93c9-44f0d5b1016a-credential-socket" pod "8b28543a-e85d-4593-93c9-44f0d5b1016a" (UID: "8b28543a-e85d-4593-93c9-44f0d5b1016a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:09.774316 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.774141    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b28543a-e85d-4593-93c9-44f0d5b1016a-istiod-ca-cert" pod "8b28543a-e85d-4593-93c9-44f0d5b1016a" (UID: "8b28543a-e85d-4593-93c9-44f0d5b1016a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:06:09.774316 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.774224    5713 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03ef9f7c-175a-4012-a021-ec011e37928e-istio-token\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:09.774316 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.774248    5713 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b28543a-e85d-4593-93c9-44f0d5b1016a-credential-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:09.774316 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.774263    5713 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03ef9f7c-175a-4012-a021-ec011e37928e-istio-podinfo\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:09.774316 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.774279    5713 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03ef9f7c-175a-4012-a021-ec011e37928e-istio-envoy\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:09.774316 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.774293    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9rtk\" (UniqueName: \"kubernetes.io/projected/03ef9f7c-175a-4012-a021-ec011e37928e-kube-api-access-j9rtk\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:09.774635 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.774383    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b28543a-e85d-4593-93c9-44f0d5b1016a-workload-certs" pod "8b28543a-e85d-4593-93c9-44f0d5b1016a" (UID: "8b28543a-e85d-4593-93c9-44f0d5b1016a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:09.774844 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.774750    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b28543a-e85d-4593-93c9-44f0d5b1016a-workload-socket" pod "8b28543a-e85d-4593-93c9-44f0d5b1016a" (UID: "8b28543a-e85d-4593-93c9-44f0d5b1016a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:09.775107 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.774891    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b28543a-e85d-4593-93c9-44f0d5b1016a-istio-data" pod "8b28543a-e85d-4593-93c9-44f0d5b1016a" (UID: "8b28543a-e85d-4593-93c9-44f0d5b1016a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:09.777231 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.777196    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b28543a-e85d-4593-93c9-44f0d5b1016a-istio-podinfo" pod "8b28543a-e85d-4593-93c9-44f0d5b1016a" (UID: "8b28543a-e85d-4593-93c9-44f0d5b1016a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:06:09.777474 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.777447    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b28543a-e85d-4593-93c9-44f0d5b1016a-kube-api-access-xsr8f" pod "8b28543a-e85d-4593-93c9-44f0d5b1016a" (UID: "8b28543a-e85d-4593-93c9-44f0d5b1016a"). InnerVolumeSpecName "kube-api-access-xsr8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:06:09.777684 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.777663    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b28543a-e85d-4593-93c9-44f0d5b1016a-istio-envoy" pod "8b28543a-e85d-4593-93c9-44f0d5b1016a" (UID: "8b28543a-e85d-4593-93c9-44f0d5b1016a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:09.779019 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.779001    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b28543a-e85d-4593-93c9-44f0d5b1016a-istio-token" pod "8b28543a-e85d-4593-93c9-44f0d5b1016a" (UID: "8b28543a-e85d-4593-93c9-44f0d5b1016a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:06:09.875774 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.875741    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsr8f\" (UniqueName: \"kubernetes.io/projected/8b28543a-e85d-4593-93c9-44f0d5b1016a-kube-api-access-xsr8f\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:09.875774 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.875769    5713 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b28543a-e85d-4593-93c9-44f0d5b1016a-istio-data\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:09.875774 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.875778    5713 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b28543a-e85d-4593-93c9-44f0d5b1016a-istio-token\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:09.875774 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.875801    5713 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b28543a-e85d-4593-93c9-44f0d5b1016a-workload-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:09.875774 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.875809    5713 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b28543a-e85d-4593-93c9-44f0d5b1016a-istio-envoy\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:09.875774 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.875818    5713 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b28543a-e85d-4593-93c9-44f0d5b1016a-workload-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:09.875774 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.875828    5713 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b28543a-e85d-4593-93c9-44f0d5b1016a-istiod-ca-cert\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:09.875774 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.875838    5713 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b28543a-e85d-4593-93c9-44f0d5b1016a-istio-podinfo\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:09.961498 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.961456    5713 generic.go:356] "Generic (PLEG): container finished" podID="03ef9f7c-175a-4012-a021-ec011e37928e" containerID="669acbc53462750b4744dab1559955a86540a29cb8282e3f2f3b70ec1f5b2798" exitCode=0
Oct 30 21:06:09.961720 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.961515    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8"
Oct 30 21:06:09.961720 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.961553    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8" event={"ID":"03ef9f7c-175a-4012-a021-ec011e37928e","Type":"ContainerDied","Data":"669acbc53462750b4744dab1559955a86540a29cb8282e3f2f3b70ec1f5b2798"}
Oct 30 21:06:09.961720 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.961662    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8" event={"ID":"03ef9f7c-175a-4012-a021-ec011e37928e","Type":"ContainerDied","Data":"202d8bfcd23e19f1db504b4333fb6ec8dc586082820ae5f4f7d6a006e9d8fc25"}
Oct 30 21:06:09.961720 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.961697    5713 scope.go:122] "RemoveContainer" containerID="669acbc53462750b4744dab1559955a86540a29cb8282e3f2f3b70ec1f5b2798"
Oct 30 21:06:09.963718 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.963694    5713 generic.go:356] "Generic (PLEG): container finished" podID="8b28543a-e85d-4593-93c9-44f0d5b1016a" containerID="51fd22ecb1c16e833fc4935f631f0d9a9b91dff21878c140abbae1cb3dab8f64" exitCode=0
Oct 30 21:06:09.963862 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.963834    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq" event={"ID":"8b28543a-e85d-4593-93c9-44f0d5b1016a","Type":"ContainerDied","Data":"51fd22ecb1c16e833fc4935f631f0d9a9b91dff21878c140abbae1cb3dab8f64"}
Oct 30 21:06:09.963862 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.963856    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq"
Oct 30 21:06:09.963962 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.963898    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq" event={"ID":"8b28543a-e85d-4593-93c9-44f0d5b1016a","Type":"ContainerDied","Data":"a5e468285eca9b68954aca6322fd741ea78a70c3a2642b6658d2093bd060c90a"}
Oct 30 21:06:09.975992 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.975973    5713 scope.go:122] "RemoveContainer" containerID="669acbc53462750b4744dab1559955a86540a29cb8282e3f2f3b70ec1f5b2798"
Oct 30 21:06:09.976376 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:06:09.976352    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"669acbc53462750b4744dab1559955a86540a29cb8282e3f2f3b70ec1f5b2798\": container with ID starting with 669acbc53462750b4744dab1559955a86540a29cb8282e3f2f3b70ec1f5b2798 not found: ID does not exist" containerID="669acbc53462750b4744dab1559955a86540a29cb8282e3f2f3b70ec1f5b2798"
Oct 30 21:06:09.976484 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.976390    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"669acbc53462750b4744dab1559955a86540a29cb8282e3f2f3b70ec1f5b2798"} err="failed to get container status \"669acbc53462750b4744dab1559955a86540a29cb8282e3f2f3b70ec1f5b2798\": rpc error: code = NotFound desc = could not find container \"669acbc53462750b4744dab1559955a86540a29cb8282e3f2f3b70ec1f5b2798\": container with ID starting with 669acbc53462750b4744dab1559955a86540a29cb8282e3f2f3b70ec1f5b2798 not found: ID does not exist"
Oct 30 21:06:09.976484 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.976415    5713 scope.go:122] "RemoveContainer" containerID="51fd22ecb1c16e833fc4935f631f0d9a9b91dff21878c140abbae1cb3dab8f64"
Oct 30 21:06:09.987348 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.987329    5713 scope.go:122] "RemoveContainer" containerID="51fd22ecb1c16e833fc4935f631f0d9a9b91dff21878c140abbae1cb3dab8f64"
Oct 30 21:06:09.987642 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:06:09.987620    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51fd22ecb1c16e833fc4935f631f0d9a9b91dff21878c140abbae1cb3dab8f64\": container with ID starting with 51fd22ecb1c16e833fc4935f631f0d9a9b91dff21878c140abbae1cb3dab8f64 not found: ID does not exist" containerID="51fd22ecb1c16e833fc4935f631f0d9a9b91dff21878c140abbae1cb3dab8f64"
Oct 30 21:06:09.987744 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:09.987652    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51fd22ecb1c16e833fc4935f631f0d9a9b91dff21878c140abbae1cb3dab8f64"} err="failed to get container status \"51fd22ecb1c16e833fc4935f631f0d9a9b91dff21878c140abbae1cb3dab8f64\": rpc error: code = NotFound desc = could not find container \"51fd22ecb1c16e833fc4935f631f0d9a9b91dff21878c140abbae1cb3dab8f64\": container with ID starting with 51fd22ecb1c16e833fc4935f631f0d9a9b91dff21878c140abbae1cb3dab8f64 not found: ID does not exist"
Oct 30 21:06:10.130615 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:10.130584    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8"]
Oct 30 21:06:10.134292 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:10.134267    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-6mlr8"]
Oct 30 21:06:10.244447 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:10.244362    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq"]
Oct 30 21:06:10.248072 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:10.248051    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-nlwqq"]
Oct 30 21:06:11.066720 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:11.066687    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03ef9f7c-175a-4012-a021-ec011e37928e" path="/var/lib/kubelet/pods/03ef9f7c-175a-4012-a021-ec011e37928e/volumes"
Oct 30 21:06:11.067230 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:11.067212    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b28543a-e85d-4593-93c9-44f0d5b1016a" path="/var/lib/kubelet/pods/8b28543a-e85d-4593-93c9-44f0d5b1016a/volumes"
Oct 30 21:06:13.047705 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:13.047673    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e134f233-v8lrp"]
Oct 30 21:06:13.097367 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:13.097338    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e134f233-v8lrp"]
Oct 30 21:06:14.046996 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:14.046959    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ce9871f-vgscg"]
Oct 30 21:06:14.096622 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:14.096591    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ce9871f-vgscg"]
Oct 30 21:06:15.064951 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:15.064922    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="827ff375-c35d-4a58-8d30-16ad01bb0e08" path="/var/lib/kubelet/pods/827ff375-c35d-4a58-8d30-16ad01bb0e08/volumes"
Oct 30 21:06:15.065407 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:15.065393    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be5f6063-9e6b-4805-a07f-ee96f0d437f8" path="/var/lib/kubelet/pods/be5f6063-9e6b-4805-a07f-ee96f0d437f8/volumes"
Oct 30 21:06:24.461442 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:06:24.461405    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:06:34.850826 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:34.850771    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2cf21005-f9k2m"]
Oct 30 21:06:34.901885 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:34.901845    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2cf21005-f9k2m"]
Oct 30 21:06:35.083177 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:35.083144    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc434250-12e2-4d99-b6d9-f013d995507b" path="/var/lib/kubelet/pods/dc434250-12e2-4d99-b6d9-f013d995507b/volumes"
Oct 30 21:06:37.422987 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:37.422948    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf9c0727-v9llb"]
Oct 30 21:06:37.479911 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:37.476639    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bf9c0727-v9llb"]
Oct 30 21:06:37.525905 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:37.525854    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z"]
Oct 30 21:06:37.526423 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:37.526391    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z" podUID="7784e9af-82d7-4072-ac6b-f2ae0e82e0b2" containerName="istio-proxy" containerID="cri-o://7472a41fc790c302d57461fc209939a50e62ed47a16a93ca4507563a1fb88b11" gracePeriod=30
Oct 30 21:06:37.535231 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:37.535183    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd"]
Oct 30 21:06:37.535843 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:37.535723    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd" podUID="47b70e28-cf76-4cc2-a5d4-0396fb3e985c" containerName="istio-proxy" containerID="cri-o://9928059868f25092856d94350ca3468f176e6d0df55de7c1ddfb4631c4e43d90" gracePeriod=30
Oct 30 21:06:37.825602 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:37.825564    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-88da25-5689b65574-4grv2"]
Oct 30 21:06:37.826227 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:37.826212    5713 cpu_manager.go:414] "containerMap: removing container" podUID="03ef9f7c-175a-4012-a021-ec011e37928e" containerName="istio-proxy"
Oct 30 21:06:37.826282 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:37.826230    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03ef9f7c-175a-4012-a021-ec011e37928e" containerName="istio-proxy"
Oct 30 21:06:37.826282 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:37.826252    5713 cpu_manager.go:414] "containerMap: removing container" podUID="427927da-96f1-464c-9a58-1116d61bee68" containerName="payload-processing"
Oct 30 21:06:37.826282 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:37.826257    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="427927da-96f1-464c-9a58-1116d61bee68" containerName="payload-processing"
Oct 30 21:06:37.826282 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:37.826264    5713 cpu_manager.go:414] "containerMap: removing container" podUID="ced570ba-f5f3-4c25-bc1d-0544046ed9dc" containerName="revoke-keys"
Oct 30 21:06:37.826282 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:37.826269    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ced570ba-f5f3-4c25-bc1d-0544046ed9dc" containerName="revoke-keys"
Oct 30 21:06:37.826282 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:37.826280    5713 cpu_manager.go:414] "containerMap: removing container" podUID="8b28543a-e85d-4593-93c9-44f0d5b1016a" containerName="istio-proxy"
Oct 30 21:06:37.826282 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:37.826284    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b28543a-e85d-4593-93c9-44f0d5b1016a" containerName="istio-proxy"
Oct 30 21:06:37.826490 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:37.826295    5713 cpu_manager.go:414] "containerMap: removing container" podUID="bdc39409-a6df-4bed-8017-a6f502bb7b44" containerName="payload-pre-processing"
Oct 30 21:06:37.826490 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:37.826300    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdc39409-a6df-4bed-8017-a6f502bb7b44" containerName="payload-pre-processing"
Oct 30 21:06:37.826490 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:37.826312    5713 cpu_manager.go:414] "containerMap: removing container" podUID="9a73cfa4-d614-4c39-b324-731fca0564c4" containerName="istio-proxy"
Oct 30 21:06:37.826490 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:37.826316    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a73cfa4-d614-4c39-b324-731fca0564c4" containerName="istio-proxy"
Oct 30 21:06:37.826490 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:37.826331    5713 cpu_manager.go:414] "containerMap: removing container" podUID="f5478d17-b619-43ac-8f68-2675932b4abe" containerName="payload-processing"
Oct 30 21:06:37.826490 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:37.826335    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5478d17-b619-43ac-8f68-2675932b4abe" containerName="payload-processing"
Oct 30 21:06:37.826490 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:37.826424    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdc39409-a6df-4bed-8017-a6f502bb7b44" containerName="payload-pre-processing"
Oct 30 21:06:37.826490 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:37.826435    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a73cfa4-d614-4c39-b324-731fca0564c4" containerName="istio-proxy"
Oct 30 21:06:37.826490 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:37.826442    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5478d17-b619-43ac-8f68-2675932b4abe" containerName="payload-processing"
Oct 30 21:06:37.826490 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:37.826452    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="03ef9f7c-175a-4012-a021-ec011e37928e" containerName="istio-proxy"
Oct 30 21:06:37.826490 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:37.826459    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="427927da-96f1-464c-9a58-1116d61bee68" containerName="payload-processing"
Oct 30 21:06:37.826490 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:37.826465    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="ced570ba-f5f3-4c25-bc1d-0544046ed9dc" containerName="revoke-keys"
Oct 30 21:06:37.826490 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:37.826471    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b28543a-e85d-4593-93c9-44f0d5b1016a" containerName="istio-proxy"
Oct 30 21:06:37.830168 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:37.830151    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-88da25-5689b65574-4grv2"
Oct 30 21:06:37.907816 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:37.898995    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-88da25-5689b65574-4grv2"]
Oct 30 21:06:37.907816 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:37.902944    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-88da25-dockercfg-khgc8\""
Oct 30 21:06:37.907816 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:37.904469    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-88da25\""
Oct 30 21:06:37.983811 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:37.980888    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b53c958-a28c-44d6-bf7b-76b9891f67e5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-88da25-5689b65574-4grv2\" (UID: \"2b53c958-a28c-44d6-bf7b-76b9891f67e5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-88da25-5689b65574-4grv2"
Oct 30 21:06:37.983811 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:37.981098    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbqt5\" (UniqueName: \"kubernetes.io/projected/2b53c958-a28c-44d6-bf7b-76b9891f67e5-kube-api-access-rbqt5\") pod \"payload-processing-e2e-shared-b-w4-88da25-5689b65574-4grv2\" (UID: \"2b53c958-a28c-44d6-bf7b-76b9891f67e5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-88da25-5689b65574-4grv2"
Oct 30 21:06:37.983811 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:37.983425    5713 scope.go:122] "RemoveContainer" containerID="41a24ca00d9e8ef9950f6133112d43da5d7127d33712d95444167d4e0c4e17de"
Oct 30 21:06:38.004496 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:38.004334    5713 scope.go:122] "RemoveContainer" containerID="99bad37d32417706ad98f34bdd2bf0e71af26d86da4b38608b00d3988fbaf48a"
Oct 30 21:06:38.021665 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:38.021584    5713 scope.go:122] "RemoveContainer" containerID="4c1464ea69aee92331245ff41b7a3e4efc3b5be5d7afc28e580a22a2a58344a6"
Oct 30 21:06:38.068273 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:38.063650    5713 scope.go:122] "RemoveContainer" containerID="cb3d45c5f447e45ad4b8beead70d6c86aa78662ee299d59fee99170a6eac06d0"
Oct 30 21:06:38.091857 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:38.089460    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b53c958-a28c-44d6-bf7b-76b9891f67e5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-88da25-5689b65574-4grv2\" (UID: \"2b53c958-a28c-44d6-bf7b-76b9891f67e5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-88da25-5689b65574-4grv2"
Oct 30 21:06:38.091857 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:38.089656    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbqt5\" (UniqueName: \"kubernetes.io/projected/2b53c958-a28c-44d6-bf7b-76b9891f67e5-kube-api-access-rbqt5\") pod \"payload-processing-e2e-shared-b-w4-88da25-5689b65574-4grv2\" (UID: \"2b53c958-a28c-44d6-bf7b-76b9891f67e5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-88da25-5689b65574-4grv2"
Oct 30 21:06:38.091857 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:38.091512    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b53c958-a28c-44d6-bf7b-76b9891f67e5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-88da25-5689b65574-4grv2\" (UID: \"2b53c958-a28c-44d6-bf7b-76b9891f67e5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-88da25-5689b65574-4grv2"
Oct 30 21:06:38.092568 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:38.092545    5713 scope.go:122] "RemoveContainer" containerID="2105fc4b9d195aa05730e6eaf81c681e9e23c019099d26c8d6a7852be37b3015"
Oct 30 21:06:38.122902 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:38.122723    5713 scope.go:122] "RemoveContainer" containerID="520343b3a94b83fa11e96f6cd390b95e23a29c03d51dc9f0e6d6bb0b2b2fdb4d"
Oct 30 21:06:38.135805 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:38.135759    5713 scope.go:122] "RemoveContainer" containerID="9fa5eda54fb8661dfa6eb5d769e21917a3f66d79d5560116e25258d74e6deef7"
Oct 30 21:06:38.146487 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:38.146466    5713 scope.go:122] "RemoveContainer" containerID="67d051a3eece47998d8af66bbbc9b3e01756838dd951cd314c2eea46932f6525"
Oct 30 21:06:38.148388 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:38.148363    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbqt5\" (UniqueName: \"kubernetes.io/projected/2b53c958-a28c-44d6-bf7b-76b9891f67e5-kube-api-access-rbqt5\") pod \"payload-processing-e2e-shared-b-w4-88da25-5689b65574-4grv2\" (UID: \"2b53c958-a28c-44d6-bf7b-76b9891f67e5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-88da25-5689b65574-4grv2"
Oct 30 21:06:38.161503 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:38.161480    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-88da25-5689b65574-4grv2"
Oct 30 21:06:38.565742 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:38.565708    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-88da25-5689b65574-4grv2"]
Oct 30 21:06:38.567183 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:06:38.567151    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b53c958_a28c_44d6_bf7b_76b9891f67e5.slice/crio-be86d7437c78698de40d965cee0d2651f10be2ab1b76f9f0114838777f249cfd WatchSource:0}: Error finding container be86d7437c78698de40d965cee0d2651f10be2ab1b76f9f0114838777f249cfd: Status 404 returned error can't find the container with id be86d7437c78698de40d965cee0d2651f10be2ab1b76f9f0114838777f249cfd
Oct 30 21:06:39.071641 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:39.071607    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ced570ba-f5f3-4c25-bc1d-0544046ed9dc" path="/var/lib/kubelet/pods/ced570ba-f5f3-4c25-bc1d-0544046ed9dc/volumes"
Oct 30 21:06:39.296533 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:39.296498    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-88da25-5689b65574-4grv2" event={"ID":"2b53c958-a28c-44d6-bf7b-76b9891f67e5","Type":"ContainerStarted","Data":"330e193080a93bbed8e503754669b4abb9783e65b94fe3e27c38f650059ae247"}
Oct 30 21:06:39.296739 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:39.296552    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-88da25-5689b65574-4grv2" event={"ID":"2b53c958-a28c-44d6-bf7b-76b9891f67e5","Type":"ContainerStarted","Data":"be86d7437c78698de40d965cee0d2651f10be2ab1b76f9f0114838777f249cfd"}
Oct 30 21:06:39.296860 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:39.296830    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-88da25-5689b65574-4grv2"
Oct 30 21:06:39.422402 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:39.422303    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-88da25-5689b65574-4grv2" podStartSLOduration=2.422289724 podStartE2EDuration="2.422289724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:06:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:06:39.41961094 +0000 UTC m=+1682.484387338" watchObservedRunningTime="2026-10-30 21:06:39.422289724 +0000 UTC m=+1682.487066126"
Oct 30 21:06:42.888405 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:42.888369    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z"
Oct 30 21:06:42.894100 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:42.894060    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd"
Oct 30 21:06:42.977396 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:42.977357    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47b70e28-cf76-4cc2-a5d4-0396fb3e985c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47b70e28-cf76-4cc2-a5d4-0396fb3e985c-istiod-ca-cert\") pod \"47b70e28-cf76-4cc2-a5d4-0396fb3e985c\" (UID: \"47b70e28-cf76-4cc2-a5d4-0396fb3e985c\") "
Oct 30 21:06:42.977640 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:42.977497    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7784e9af-82d7-4072-ac6b-f2ae0e82e0b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7784e9af-82d7-4072-ac6b-f2ae0e82e0b2-workload-certs\") pod \"7784e9af-82d7-4072-ac6b-f2ae0e82e0b2\" (UID: \"7784e9af-82d7-4072-ac6b-f2ae0e82e0b2\") "
Oct 30 21:06:42.977640 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:42.977542    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7784e9af-82d7-4072-ac6b-f2ae0e82e0b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7784e9af-82d7-4072-ac6b-f2ae0e82e0b2-credential-socket\") pod \"7784e9af-82d7-4072-ac6b-f2ae0e82e0b2\" (UID: \"7784e9af-82d7-4072-ac6b-f2ae0e82e0b2\") "
Oct 30 21:06:42.977640 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:42.977590    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47b70e28-cf76-4cc2-a5d4-0396fb3e985c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47b70e28-cf76-4cc2-a5d4-0396fb3e985c-credential-socket\") pod \"47b70e28-cf76-4cc2-a5d4-0396fb3e985c\" (UID: \"47b70e28-cf76-4cc2-a5d4-0396fb3e985c\") "
Oct 30 21:06:42.978013 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:42.977641    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47b70e28-cf76-4cc2-a5d4-0396fb3e985c-kube-api-access-wrv28\" (UniqueName: \"kubernetes.io/projected/47b70e28-cf76-4cc2-a5d4-0396fb3e985c-kube-api-access-wrv28\") pod \"47b70e28-cf76-4cc2-a5d4-0396fb3e985c\" (UID: \"47b70e28-cf76-4cc2-a5d4-0396fb3e985c\") "
Oct 30 21:06:42.978013 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:42.977712    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47b70e28-cf76-4cc2-a5d4-0396fb3e985c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47b70e28-cf76-4cc2-a5d4-0396fb3e985c-workload-certs\") pod \"47b70e28-cf76-4cc2-a5d4-0396fb3e985c\" (UID: \"47b70e28-cf76-4cc2-a5d4-0396fb3e985c\") "
Oct 30 21:06:42.978013 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:42.977756    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7784e9af-82d7-4072-ac6b-f2ae0e82e0b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7784e9af-82d7-4072-ac6b-f2ae0e82e0b2-istio-data\") pod \"7784e9af-82d7-4072-ac6b-f2ae0e82e0b2\" (UID: \"7784e9af-82d7-4072-ac6b-f2ae0e82e0b2\") "
Oct 30 21:06:42.978013 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:42.977834    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47b70e28-cf76-4cc2-a5d4-0396fb3e985c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47b70e28-cf76-4cc2-a5d4-0396fb3e985c-istio-envoy\") pod \"47b70e28-cf76-4cc2-a5d4-0396fb3e985c\" (UID: \"47b70e28-cf76-4cc2-a5d4-0396fb3e985c\") "
Oct 30 21:06:42.979348 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:42.977904    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47b70e28-cf76-4cc2-a5d4-0396fb3e985c-istiod-ca-cert" pod "47b70e28-cf76-4cc2-a5d4-0396fb3e985c" (UID: "47b70e28-cf76-4cc2-a5d4-0396fb3e985c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:06:42.979446 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:42.979322    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7784e9af-82d7-4072-ac6b-f2ae0e82e0b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7784e9af-82d7-4072-ac6b-f2ae0e82e0b2-workload-socket\") pod \"7784e9af-82d7-4072-ac6b-f2ae0e82e0b2\" (UID: \"7784e9af-82d7-4072-ac6b-f2ae0e82e0b2\") "
Oct 30 21:06:42.979446 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:42.977993    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7784e9af-82d7-4072-ac6b-f2ae0e82e0b2-workload-certs" pod "7784e9af-82d7-4072-ac6b-f2ae0e82e0b2" (UID: "7784e9af-82d7-4072-ac6b-f2ae0e82e0b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:42.979446 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:42.978375    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47b70e28-cf76-4cc2-a5d4-0396fb3e985c-workload-certs" pod "47b70e28-cf76-4cc2-a5d4-0396fb3e985c" (UID: "47b70e28-cf76-4cc2-a5d4-0396fb3e985c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:42.979446 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:42.978672    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7784e9af-82d7-4072-ac6b-f2ae0e82e0b2-credential-socket" pod "7784e9af-82d7-4072-ac6b-f2ae0e82e0b2" (UID: "7784e9af-82d7-4072-ac6b-f2ae0e82e0b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:42.979446 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:42.978866    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47b70e28-cf76-4cc2-a5d4-0396fb3e985c-credential-socket" pod "47b70e28-cf76-4cc2-a5d4-0396fb3e985c" (UID: "47b70e28-cf76-4cc2-a5d4-0396fb3e985c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:42.979446 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:42.979018    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7784e9af-82d7-4072-ac6b-f2ae0e82e0b2-istio-data" pod "7784e9af-82d7-4072-ac6b-f2ae0e82e0b2" (UID: "7784e9af-82d7-4072-ac6b-f2ae0e82e0b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:42.979834 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:42.979450    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7784e9af-82d7-4072-ac6b-f2ae0e82e0b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7784e9af-82d7-4072-ac6b-f2ae0e82e0b2-istio-podinfo\") pod \"7784e9af-82d7-4072-ac6b-f2ae0e82e0b2\" (UID: \"7784e9af-82d7-4072-ac6b-f2ae0e82e0b2\") "
Oct 30 21:06:42.979834 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:42.979509    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47b70e28-cf76-4cc2-a5d4-0396fb3e985c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47b70e28-cf76-4cc2-a5d4-0396fb3e985c-istio-data\") pod \"47b70e28-cf76-4cc2-a5d4-0396fb3e985c\" (UID: \"47b70e28-cf76-4cc2-a5d4-0396fb3e985c\") "
Oct 30 21:06:42.979834 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:42.979512    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7784e9af-82d7-4072-ac6b-f2ae0e82e0b2-workload-socket" pod "7784e9af-82d7-4072-ac6b-f2ae0e82e0b2" (UID: "7784e9af-82d7-4072-ac6b-f2ae0e82e0b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:42.979834 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:42.979626    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47b70e28-cf76-4cc2-a5d4-0396fb3e985c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47b70e28-cf76-4cc2-a5d4-0396fb3e985c-workload-socket\") pod \"47b70e28-cf76-4cc2-a5d4-0396fb3e985c\" (UID: \"47b70e28-cf76-4cc2-a5d4-0396fb3e985c\") "
Oct 30 21:06:42.979834 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:42.979743    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7784e9af-82d7-4072-ac6b-f2ae0e82e0b2-kube-api-access-9dh9c\" (UniqueName: \"kubernetes.io/projected/7784e9af-82d7-4072-ac6b-f2ae0e82e0b2-kube-api-access-9dh9c\") pod \"7784e9af-82d7-4072-ac6b-f2ae0e82e0b2\" (UID: \"7784e9af-82d7-4072-ac6b-f2ae0e82e0b2\") "
Oct 30 21:06:42.979834 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:42.979756    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47b70e28-cf76-4cc2-a5d4-0396fb3e985c-istio-data" pod "47b70e28-cf76-4cc2-a5d4-0396fb3e985c" (UID: "47b70e28-cf76-4cc2-a5d4-0396fb3e985c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:42.980175 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:42.979881    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47b70e28-cf76-4cc2-a5d4-0396fb3e985c-istio-token\" (UniqueName: \"kubernetes.io/projected/47b70e28-cf76-4cc2-a5d4-0396fb3e985c-istio-token\") pod \"47b70e28-cf76-4cc2-a5d4-0396fb3e985c\" (UID: \"47b70e28-cf76-4cc2-a5d4-0396fb3e985c\") "
Oct 30 21:06:42.980175 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:42.979975    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7784e9af-82d7-4072-ac6b-f2ae0e82e0b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7784e9af-82d7-4072-ac6b-f2ae0e82e0b2-istio-envoy\") pod \"7784e9af-82d7-4072-ac6b-f2ae0e82e0b2\" (UID: \"7784e9af-82d7-4072-ac6b-f2ae0e82e0b2\") "
Oct 30 21:06:42.980175 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:42.980099    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7784e9af-82d7-4072-ac6b-f2ae0e82e0b2-istio-token\" (UniqueName: \"kubernetes.io/projected/7784e9af-82d7-4072-ac6b-f2ae0e82e0b2-istio-token\") pod \"7784e9af-82d7-4072-ac6b-f2ae0e82e0b2\" (UID: \"7784e9af-82d7-4072-ac6b-f2ae0e82e0b2\") "
Oct 30 21:06:42.980175 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:42.980151    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/47b70e28-cf76-4cc2-a5d4-0396fb3e985c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47b70e28-cf76-4cc2-a5d4-0396fb3e985c-istio-podinfo\") pod \"47b70e28-cf76-4cc2-a5d4-0396fb3e985c\" (UID: \"47b70e28-cf76-4cc2-a5d4-0396fb3e985c\") "
Oct 30 21:06:42.980384 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:42.980199    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7784e9af-82d7-4072-ac6b-f2ae0e82e0b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7784e9af-82d7-4072-ac6b-f2ae0e82e0b2-istiod-ca-cert\") pod \"7784e9af-82d7-4072-ac6b-f2ae0e82e0b2\" (UID: \"7784e9af-82d7-4072-ac6b-f2ae0e82e0b2\") "
Oct 30 21:06:42.980493 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:42.980469    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47b70e28-cf76-4cc2-a5d4-0396fb3e985c-workload-socket" pod "47b70e28-cf76-4cc2-a5d4-0396fb3e985c" (UID: "47b70e28-cf76-4cc2-a5d4-0396fb3e985c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:42.981621 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:42.981135    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7784e9af-82d7-4072-ac6b-f2ae0e82e0b2-istiod-ca-cert" pod "7784e9af-82d7-4072-ac6b-f2ae0e82e0b2" (UID: "7784e9af-82d7-4072-ac6b-f2ae0e82e0b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:06:42.981621 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:42.981427    5713 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47b70e28-cf76-4cc2-a5d4-0396fb3e985c-credential-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:42.981621 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:42.981442    5713 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47b70e28-cf76-4cc2-a5d4-0396fb3e985c-workload-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:42.981621 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:42.981453    5713 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7784e9af-82d7-4072-ac6b-f2ae0e82e0b2-istio-data\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:42.981621 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:42.981468    5713 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7784e9af-82d7-4072-ac6b-f2ae0e82e0b2-workload-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:42.981621 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:42.981478    5713 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47b70e28-cf76-4cc2-a5d4-0396fb3e985c-istio-data\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:42.981621 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:42.981491    5713 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47b70e28-cf76-4cc2-a5d4-0396fb3e985c-workload-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:42.981621 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:42.981502    5713 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7784e9af-82d7-4072-ac6b-f2ae0e82e0b2-istiod-ca-cert\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:42.981621 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:42.981517    5713 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47b70e28-cf76-4cc2-a5d4-0396fb3e985c-istiod-ca-cert\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:42.981621 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:42.981526    5713 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7784e9af-82d7-4072-ac6b-f2ae0e82e0b2-workload-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:42.981621 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:42.981534    5713 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7784e9af-82d7-4072-ac6b-f2ae0e82e0b2-credential-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:42.987352 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:42.986925    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47b70e28-cf76-4cc2-a5d4-0396fb3e985c-istio-envoy" pod "47b70e28-cf76-4cc2-a5d4-0396fb3e985c" (UID: "47b70e28-cf76-4cc2-a5d4-0396fb3e985c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:42.992472 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:42.992050    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7784e9af-82d7-4072-ac6b-f2ae0e82e0b2-istio-envoy" pod "7784e9af-82d7-4072-ac6b-f2ae0e82e0b2" (UID: "7784e9af-82d7-4072-ac6b-f2ae0e82e0b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:42.992472 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:42.992309    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7784e9af-82d7-4072-ac6b-f2ae0e82e0b2-kube-api-access-9dh9c" pod "7784e9af-82d7-4072-ac6b-f2ae0e82e0b2" (UID: "7784e9af-82d7-4072-ac6b-f2ae0e82e0b2"). InnerVolumeSpecName "kube-api-access-9dh9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:06:42.992472 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:42.992372    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7784e9af-82d7-4072-ac6b-f2ae0e82e0b2-istio-podinfo" pod "7784e9af-82d7-4072-ac6b-f2ae0e82e0b2" (UID: "7784e9af-82d7-4072-ac6b-f2ae0e82e0b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:06:42.992472 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:42.992426    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47b70e28-cf76-4cc2-a5d4-0396fb3e985c-istio-podinfo" pod "47b70e28-cf76-4cc2-a5d4-0396fb3e985c" (UID: "47b70e28-cf76-4cc2-a5d4-0396fb3e985c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:06:42.994024 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:42.993987    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47b70e28-cf76-4cc2-a5d4-0396fb3e985c-kube-api-access-wrv28" pod "47b70e28-cf76-4cc2-a5d4-0396fb3e985c" (UID: "47b70e28-cf76-4cc2-a5d4-0396fb3e985c"). InnerVolumeSpecName "kube-api-access-wrv28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:06:42.995214 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:42.995185    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7784e9af-82d7-4072-ac6b-f2ae0e82e0b2-istio-token" pod "7784e9af-82d7-4072-ac6b-f2ae0e82e0b2" (UID: "7784e9af-82d7-4072-ac6b-f2ae0e82e0b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:06:42.995577 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:42.995548    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47b70e28-cf76-4cc2-a5d4-0396fb3e985c-istio-token" pod "47b70e28-cf76-4cc2-a5d4-0396fb3e985c" (UID: "47b70e28-cf76-4cc2-a5d4-0396fb3e985c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:06:43.109005 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:43.095050    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrv28\" (UniqueName: \"kubernetes.io/projected/47b70e28-cf76-4cc2-a5d4-0396fb3e985c-kube-api-access-wrv28\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:43.218191 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:43.218147    5713 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47b70e28-cf76-4cc2-a5d4-0396fb3e985c-istio-envoy\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:43.218453 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:43.218436    5713 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7784e9af-82d7-4072-ac6b-f2ae0e82e0b2-istio-podinfo\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:43.218606 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:43.218569    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dh9c\" (UniqueName: \"kubernetes.io/projected/7784e9af-82d7-4072-ac6b-f2ae0e82e0b2-kube-api-access-9dh9c\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:43.218768 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:43.218744    5713 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47b70e28-cf76-4cc2-a5d4-0396fb3e985c-istio-token\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:43.218951 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:43.218917    5713 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7784e9af-82d7-4072-ac6b-f2ae0e82e0b2-istio-envoy\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:43.219175 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:43.219140    5713 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7784e9af-82d7-4072-ac6b-f2ae0e82e0b2-istio-token\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:43.219295 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:43.219280    5713 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47b70e28-cf76-4cc2-a5d4-0396fb3e985c-istio-podinfo\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:43.347813 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:43.347460    5713 generic.go:356] "Generic (PLEG): container finished" podID="47b70e28-cf76-4cc2-a5d4-0396fb3e985c" containerID="9928059868f25092856d94350ca3468f176e6d0df55de7c1ddfb4631c4e43d90" exitCode=0
Oct 30 21:06:43.356959 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:43.347537    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd" event={"ID":"47b70e28-cf76-4cc2-a5d4-0396fb3e985c","Type":"ContainerDied","Data":"9928059868f25092856d94350ca3468f176e6d0df55de7c1ddfb4631c4e43d90"}
Oct 30 21:06:43.356959 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:43.348053    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd" event={"ID":"47b70e28-cf76-4cc2-a5d4-0396fb3e985c","Type":"ContainerDied","Data":"3485a8770fa2768e5b8f53eec85e73b00d0e49e0f3dbe3835411a75f802a6128"}
Oct 30 21:06:43.356959 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:43.348089    5713 scope.go:122] "RemoveContainer" containerID="9928059868f25092856d94350ca3468f176e6d0df55de7c1ddfb4631c4e43d90"
Oct 30 21:06:43.356959 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:43.348340    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd"
Oct 30 21:06:43.356959 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:43.349607    5713 generic.go:356] "Generic (PLEG): container finished" podID="7784e9af-82d7-4072-ac6b-f2ae0e82e0b2" containerID="7472a41fc790c302d57461fc209939a50e62ed47a16a93ca4507563a1fb88b11" exitCode=0
Oct 30 21:06:43.356959 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:43.349707    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z"
Oct 30 21:06:43.356959 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:43.349705    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z" event={"ID":"7784e9af-82d7-4072-ac6b-f2ae0e82e0b2","Type":"ContainerDied","Data":"7472a41fc790c302d57461fc209939a50e62ed47a16a93ca4507563a1fb88b11"}
Oct 30 21:06:43.356959 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:43.350314    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z" event={"ID":"7784e9af-82d7-4072-ac6b-f2ae0e82e0b2","Type":"ContainerDied","Data":"a81090a030fdace8fca9ebdf1cce412bb6e12a65d5ebf0957dae77bd9113438a"}
Oct 30 21:06:43.368107 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:43.368081    5713 scope.go:122] "RemoveContainer" containerID="9928059868f25092856d94350ca3468f176e6d0df55de7c1ddfb4631c4e43d90"
Oct 30 21:06:43.368554 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:06:43.368531    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9928059868f25092856d94350ca3468f176e6d0df55de7c1ddfb4631c4e43d90\": container with ID starting with 9928059868f25092856d94350ca3468f176e6d0df55de7c1ddfb4631c4e43d90 not found: ID does not exist" containerID="9928059868f25092856d94350ca3468f176e6d0df55de7c1ddfb4631c4e43d90"
Oct 30 21:06:43.368625 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:43.368564    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9928059868f25092856d94350ca3468f176e6d0df55de7c1ddfb4631c4e43d90"} err="failed to get container status \"9928059868f25092856d94350ca3468f176e6d0df55de7c1ddfb4631c4e43d90\": rpc error: code = NotFound desc = could not find container \"9928059868f25092856d94350ca3468f176e6d0df55de7c1ddfb4631c4e43d90\": container with ID starting with 9928059868f25092856d94350ca3468f176e6d0df55de7c1ddfb4631c4e43d90 not found: ID does not exist"
Oct 30 21:06:43.368625 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:43.368585    5713 scope.go:122] "RemoveContainer" containerID="7472a41fc790c302d57461fc209939a50e62ed47a16a93ca4507563a1fb88b11"
Oct 30 21:06:43.383367 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:43.383343    5713 scope.go:122] "RemoveContainer" containerID="7472a41fc790c302d57461fc209939a50e62ed47a16a93ca4507563a1fb88b11"
Oct 30 21:06:43.383739 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:06:43.383709    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7472a41fc790c302d57461fc209939a50e62ed47a16a93ca4507563a1fb88b11\": container with ID starting with 7472a41fc790c302d57461fc209939a50e62ed47a16a93ca4507563a1fb88b11 not found: ID does not exist" containerID="7472a41fc790c302d57461fc209939a50e62ed47a16a93ca4507563a1fb88b11"
Oct 30 21:06:43.383923 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:43.383753    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7472a41fc790c302d57461fc209939a50e62ed47a16a93ca4507563a1fb88b11"} err="failed to get container status \"7472a41fc790c302d57461fc209939a50e62ed47a16a93ca4507563a1fb88b11\": rpc error: code = NotFound desc = could not find container \"7472a41fc790c302d57461fc209939a50e62ed47a16a93ca4507563a1fb88b11\": container with ID starting with 7472a41fc790c302d57461fc209939a50e62ed47a16a93ca4507563a1fb88b11 not found: ID does not exist"
Oct 30 21:06:43.563520 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:43.563481    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd"]
Oct 30 21:06:43.567055 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:43.567033    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-qbdxd"]
Oct 30 21:06:43.676082 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:43.676005    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z"]
Oct 30 21:06:43.679512 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:43.679493    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-bf9c0727-openshift-default-7fb479dd89-p8w4z"]
Oct 30 21:06:45.064916 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:45.064885    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47b70e28-cf76-4cc2-a5d4-0396fb3e985c" path="/var/lib/kubelet/pods/47b70e28-cf76-4cc2-a5d4-0396fb3e985c/volumes"
Oct 30 21:06:45.065433 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:45.065417    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7784e9af-82d7-4072-ac6b-f2ae0e82e0b2" path="/var/lib/kubelet/pods/7784e9af-82d7-4072-ac6b-f2ae0e82e0b2/volumes"
Oct 30 21:06:47.838367 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:47.838322    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-388e5b76-4tqxt"]
Oct 30 21:06:47.839204 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:47.839183    5713 cpu_manager.go:414] "containerMap: removing container" podUID="7784e9af-82d7-4072-ac6b-f2ae0e82e0b2" containerName="istio-proxy"
Oct 30 21:06:47.839325 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:47.839205    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7784e9af-82d7-4072-ac6b-f2ae0e82e0b2" containerName="istio-proxy"
Oct 30 21:06:47.839325 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:47.839230    5713 cpu_manager.go:414] "containerMap: removing container" podUID="47b70e28-cf76-4cc2-a5d4-0396fb3e985c" containerName="istio-proxy"
Oct 30 21:06:47.839325 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:47.839237    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47b70e28-cf76-4cc2-a5d4-0396fb3e985c" containerName="istio-proxy"
Oct 30 21:06:47.839476 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:47.839437    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="7784e9af-82d7-4072-ac6b-f2ae0e82e0b2" containerName="istio-proxy"
Oct 30 21:06:47.839476 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:47.839452    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="47b70e28-cf76-4cc2-a5d4-0396fb3e985c" containerName="istio-proxy"
Oct 30 21:06:47.844563 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:47.844540    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-388e5b76-4tqxt"
Oct 30 21:06:47.899807 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:47.899757    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-388e5b76-4tqxt"]
Oct 30 21:06:47.926434 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:47.926402    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjrfm\" (UniqueName: \"kubernetes.io/projected/2cbf9725-307f-4b4e-8119-dbd41f83be2a-kube-api-access-cjrfm\") pod \"maas-api-revoke-sub-388e5b76-4tqxt\" (UID: \"2cbf9725-307f-4b4e-8119-dbd41f83be2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-388e5b76-4tqxt"
Oct 30 21:06:47.926622 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:47.926457    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cbf9725-307f-4b4e-8119-dbd41f83be2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-388e5b76-4tqxt\" (UID: \"2cbf9725-307f-4b4e-8119-dbd41f83be2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-388e5b76-4tqxt"
Oct 30 21:06:47.926777 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:47.926747    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cbf9725-307f-4b4e-8119-dbd41f83be2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-388e5b76-4tqxt\" (UID: \"2cbf9725-307f-4b4e-8119-dbd41f83be2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-388e5b76-4tqxt"
Oct 30 21:06:47.945079 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:47.945047    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:06:47.945079 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:47.945071    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:06:47.946740 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:47.946718    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:06:47.947012 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:47.946997    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:06:48.029393 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:48.029359    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cbf9725-307f-4b4e-8119-dbd41f83be2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-388e5b76-4tqxt\" (UID: \"2cbf9725-307f-4b4e-8119-dbd41f83be2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-388e5b76-4tqxt"
Oct 30 21:06:48.029582 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:48.029542    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjrfm\" (UniqueName: \"kubernetes.io/projected/2cbf9725-307f-4b4e-8119-dbd41f83be2a-kube-api-access-cjrfm\") pod \"maas-api-revoke-sub-388e5b76-4tqxt\" (UID: \"2cbf9725-307f-4b4e-8119-dbd41f83be2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-388e5b76-4tqxt"
Oct 30 21:06:48.029658 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:48.029601    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cbf9725-307f-4b4e-8119-dbd41f83be2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-388e5b76-4tqxt\" (UID: \"2cbf9725-307f-4b4e-8119-dbd41f83be2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-388e5b76-4tqxt"
Oct 30 21:06:48.030176 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:48.030145    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cbf9725-307f-4b4e-8119-dbd41f83be2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-388e5b76-4tqxt\" (UID: \"2cbf9725-307f-4b4e-8119-dbd41f83be2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-388e5b76-4tqxt"
Oct 30 21:06:48.032539 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:48.032516    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cbf9725-307f-4b4e-8119-dbd41f83be2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-388e5b76-4tqxt\" (UID: \"2cbf9725-307f-4b4e-8119-dbd41f83be2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-388e5b76-4tqxt"
Oct 30 21:06:48.087999 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:48.087975    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjrfm\" (UniqueName: \"kubernetes.io/projected/2cbf9725-307f-4b4e-8119-dbd41f83be2a-kube-api-access-cjrfm\") pod \"maas-api-revoke-sub-388e5b76-4tqxt\" (UID: \"2cbf9725-307f-4b4e-8119-dbd41f83be2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-388e5b76-4tqxt"
Oct 30 21:06:48.160464 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:48.160386    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-388e5b76-4tqxt"
Oct 30 21:06:48.566363 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:48.566331    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-388e5b76-4tqxt"]
Oct 30 21:06:48.566487 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:06:48.566412    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cbf9725_307f_4b4e_8119_dbd41f83be2a.slice/crio-1594cf7ec1410bafbdfbc455deea36ec2baebf546ecd14134e150beeec7df7f8 WatchSource:0}: Error finding container 1594cf7ec1410bafbdfbc455deea36ec2baebf546ecd14134e150beeec7df7f8: Status 404 returned error can't find the container with id 1594cf7ec1410bafbdfbc455deea36ec2baebf546ecd14134e150beeec7df7f8
Oct 30 21:06:49.421483 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:49.421453    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-388e5b76-4tqxt" event={"ID":"2cbf9725-307f-4b4e-8119-dbd41f83be2a","Type":"ContainerStarted","Data":"44959020f09c0a00a00ef1a891c7de7e2c79e96055404ff92a99ec10c9f987f5"}
Oct 30 21:06:49.421937 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:49.421506    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-388e5b76-4tqxt" event={"ID":"2cbf9725-307f-4b4e-8119-dbd41f83be2a","Type":"ContainerStarted","Data":"1594cf7ec1410bafbdfbc455deea36ec2baebf546ecd14134e150beeec7df7f8"}
Oct 30 21:06:49.809861 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:49.809829    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff"]
Oct 30 21:06:49.810070 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:49.810044    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff" podUID="c58cfb06-4eaa-4e30-8382-ab66354eec5a" containerName="istio-proxy" containerID="cri-o://78cb10ce4647f90759b455fd84f99cb23c65d7290f042a1f206bca954246fb5a" gracePeriod=30
Oct 30 21:06:49.860735 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:49.859481    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9"]
Oct 30 21:06:49.860735 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:49.859838    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9" podUID="0a88fd37-ae88-4075-8203-024e856be6ee" containerName="istio-proxy" containerID="cri-o://4bfc0ffcf59e5706d3ec5d7854427e5e6b5ea158791048ed7fb2487dea0f7656" gracePeriod=30
Oct 30 21:06:50.309627 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:50.309600    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-88da25-5689b65574-4grv2"
Oct 30 21:06:50.432894 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:50.432857    5713 generic.go:356] "Generic (PLEG): container finished" podID="2cbf9725-307f-4b4e-8119-dbd41f83be2a" containerID="44959020f09c0a00a00ef1a891c7de7e2c79e96055404ff92a99ec10c9f987f5" exitCode=0
Oct 30 21:06:50.433309 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:50.432946    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-388e5b76-4tqxt" event={"ID":"2cbf9725-307f-4b4e-8119-dbd41f83be2a","Type":"ContainerDied","Data":"44959020f09c0a00a00ef1a891c7de7e2c79e96055404ff92a99ec10c9f987f5"}
Oct 30 21:06:50.624254 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:50.624224    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-388e5b76-4tqxt"
Oct 30 21:06:50.668370 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:50.668339    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cbf9725-307f-4b4e-8119-dbd41f83be2a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cbf9725-307f-4b4e-8119-dbd41f83be2a-maas-api-openshift-ca\") pod \"2cbf9725-307f-4b4e-8119-dbd41f83be2a\" (UID: \"2cbf9725-307f-4b4e-8119-dbd41f83be2a\") "
Oct 30 21:06:50.668545 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:50.668527    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cbf9725-307f-4b4e-8119-dbd41f83be2a-kube-api-access-cjrfm\" (UniqueName: \"kubernetes.io/projected/2cbf9725-307f-4b4e-8119-dbd41f83be2a-kube-api-access-cjrfm\") pod \"2cbf9725-307f-4b4e-8119-dbd41f83be2a\" (UID: \"2cbf9725-307f-4b4e-8119-dbd41f83be2a\") "
Oct 30 21:06:50.668683 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:50.668666    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cbf9725-307f-4b4e-8119-dbd41f83be2a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cbf9725-307f-4b4e-8119-dbd41f83be2a-maas-api-xks-ca\") pod \"2cbf9725-307f-4b4e-8119-dbd41f83be2a\" (UID: \"2cbf9725-307f-4b4e-8119-dbd41f83be2a\") "
Oct 30 21:06:50.668757 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:50.668737    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cbf9725-307f-4b4e-8119-dbd41f83be2a-maas-api-openshift-ca" pod "2cbf9725-307f-4b4e-8119-dbd41f83be2a" (UID: "2cbf9725-307f-4b4e-8119-dbd41f83be2a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:06:50.669380 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:50.669357    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cbf9725-307f-4b4e-8119-dbd41f83be2a-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:50.670974 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:50.670947    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cbf9725-307f-4b4e-8119-dbd41f83be2a-maas-api-xks-ca" pod "2cbf9725-307f-4b4e-8119-dbd41f83be2a" (UID: "2cbf9725-307f-4b4e-8119-dbd41f83be2a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:06:50.670974 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:50.670967    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cbf9725-307f-4b4e-8119-dbd41f83be2a-kube-api-access-cjrfm" pod "2cbf9725-307f-4b4e-8119-dbd41f83be2a" (UID: "2cbf9725-307f-4b4e-8119-dbd41f83be2a"). InnerVolumeSpecName "kube-api-access-cjrfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:06:50.771436 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:50.771397    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjrfm\" (UniqueName: \"kubernetes.io/projected/2cbf9725-307f-4b4e-8119-dbd41f83be2a-kube-api-access-cjrfm\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:50.771436 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:50.771430    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cbf9725-307f-4b4e-8119-dbd41f83be2a-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:51.442821 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:51.442775    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-388e5b76-4tqxt"
Oct 30 21:06:51.443216 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:51.442844    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-388e5b76-4tqxt" event={"ID":"2cbf9725-307f-4b4e-8119-dbd41f83be2a","Type":"ContainerDied","Data":"1594cf7ec1410bafbdfbc455deea36ec2baebf546ecd14134e150beeec7df7f8"}
Oct 30 21:06:51.443216 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:51.442892    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1594cf7ec1410bafbdfbc455deea36ec2baebf546ecd14134e150beeec7df7f8"
Oct 30 21:06:52.465302 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:52.465264    5713 patch_prober.go:29] interesting pod/e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:06:52.465806 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:52.465329    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9" podUID="0a88fd37-ae88-4075-8203-024e856be6ee" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:06:54.460105 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:06:54.460065    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:06:55.165546 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.165521    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff"
Oct 30 21:06:55.217617 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.217595    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9"
Oct 30 21:06:55.254995 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.254973    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c58cfb06-4eaa-4e30-8382-ab66354eec5a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c58cfb06-4eaa-4e30-8382-ab66354eec5a-workload-certs\") pod \"c58cfb06-4eaa-4e30-8382-ab66354eec5a\" (UID: \"c58cfb06-4eaa-4e30-8382-ab66354eec5a\") "
Oct 30 21:06:55.255113 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.255020    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c58cfb06-4eaa-4e30-8382-ab66354eec5a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c58cfb06-4eaa-4e30-8382-ab66354eec5a-istio-data\") pod \"c58cfb06-4eaa-4e30-8382-ab66354eec5a\" (UID: \"c58cfb06-4eaa-4e30-8382-ab66354eec5a\") "
Oct 30 21:06:55.255113 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.255061    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c58cfb06-4eaa-4e30-8382-ab66354eec5a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c58cfb06-4eaa-4e30-8382-ab66354eec5a-workload-socket\") pod \"c58cfb06-4eaa-4e30-8382-ab66354eec5a\" (UID: \"c58cfb06-4eaa-4e30-8382-ab66354eec5a\") "
Oct 30 21:06:55.255113 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.255107    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c58cfb06-4eaa-4e30-8382-ab66354eec5a-istio-token\" (UniqueName: \"kubernetes.io/projected/c58cfb06-4eaa-4e30-8382-ab66354eec5a-istio-token\") pod \"c58cfb06-4eaa-4e30-8382-ab66354eec5a\" (UID: \"c58cfb06-4eaa-4e30-8382-ab66354eec5a\") "
Oct 30 21:06:55.255228 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.255155    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c58cfb06-4eaa-4e30-8382-ab66354eec5a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c58cfb06-4eaa-4e30-8382-ab66354eec5a-istio-podinfo\") pod \"c58cfb06-4eaa-4e30-8382-ab66354eec5a\" (UID: \"c58cfb06-4eaa-4e30-8382-ab66354eec5a\") "
Oct 30 21:06:55.255228 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.255201    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c58cfb06-4eaa-4e30-8382-ab66354eec5a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c58cfb06-4eaa-4e30-8382-ab66354eec5a-istiod-ca-cert\") pod \"c58cfb06-4eaa-4e30-8382-ab66354eec5a\" (UID: \"c58cfb06-4eaa-4e30-8382-ab66354eec5a\") "
Oct 30 21:06:55.255305 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.255255    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c58cfb06-4eaa-4e30-8382-ab66354eec5a-workload-certs" pod "c58cfb06-4eaa-4e30-8382-ab66354eec5a" (UID: "c58cfb06-4eaa-4e30-8382-ab66354eec5a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:55.255305 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.255278    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c58cfb06-4eaa-4e30-8382-ab66354eec5a-istio-data" pod "c58cfb06-4eaa-4e30-8382-ab66354eec5a" (UID: "c58cfb06-4eaa-4e30-8382-ab66354eec5a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:55.255305 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.255285    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c58cfb06-4eaa-4e30-8382-ab66354eec5a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c58cfb06-4eaa-4e30-8382-ab66354eec5a-credential-socket\") pod \"c58cfb06-4eaa-4e30-8382-ab66354eec5a\" (UID: \"c58cfb06-4eaa-4e30-8382-ab66354eec5a\") "
Oct 30 21:06:55.255464 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.255403    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c58cfb06-4eaa-4e30-8382-ab66354eec5a-workload-socket" pod "c58cfb06-4eaa-4e30-8382-ab66354eec5a" (UID: "c58cfb06-4eaa-4e30-8382-ab66354eec5a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:55.255464 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.255451    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c58cfb06-4eaa-4e30-8382-ab66354eec5a-kube-api-access-x4sm2\" (UniqueName: \"kubernetes.io/projected/c58cfb06-4eaa-4e30-8382-ab66354eec5a-kube-api-access-x4sm2\") pod \"c58cfb06-4eaa-4e30-8382-ab66354eec5a\" (UID: \"c58cfb06-4eaa-4e30-8382-ab66354eec5a\") "
Oct 30 21:06:55.255575 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.255492    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c58cfb06-4eaa-4e30-8382-ab66354eec5a-credential-socket" pod "c58cfb06-4eaa-4e30-8382-ab66354eec5a" (UID: "c58cfb06-4eaa-4e30-8382-ab66354eec5a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:55.255575 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.255563    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c58cfb06-4eaa-4e30-8382-ab66354eec5a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c58cfb06-4eaa-4e30-8382-ab66354eec5a-istio-envoy\") pod \"c58cfb06-4eaa-4e30-8382-ab66354eec5a\" (UID: \"c58cfb06-4eaa-4e30-8382-ab66354eec5a\") "
Oct 30 21:06:55.255727 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.255643    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c58cfb06-4eaa-4e30-8382-ab66354eec5a-istiod-ca-cert" pod "c58cfb06-4eaa-4e30-8382-ab66354eec5a" (UID: "c58cfb06-4eaa-4e30-8382-ab66354eec5a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:06:55.256424 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.256398    5713 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c58cfb06-4eaa-4e30-8382-ab66354eec5a-workload-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:55.256538 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.256425    5713 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c58cfb06-4eaa-4e30-8382-ab66354eec5a-istiod-ca-cert\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:55.256538 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.256441    5713 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c58cfb06-4eaa-4e30-8382-ab66354eec5a-credential-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:55.256538 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.256454    5713 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c58cfb06-4eaa-4e30-8382-ab66354eec5a-workload-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:55.256538 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.256468    5713 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c58cfb06-4eaa-4e30-8382-ab66354eec5a-istio-data\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:55.257491 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.257466    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c58cfb06-4eaa-4e30-8382-ab66354eec5a-istio-podinfo" pod "c58cfb06-4eaa-4e30-8382-ab66354eec5a" (UID: "c58cfb06-4eaa-4e30-8382-ab66354eec5a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:06:55.257587 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.257499    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c58cfb06-4eaa-4e30-8382-ab66354eec5a-istio-token" pod "c58cfb06-4eaa-4e30-8382-ab66354eec5a" (UID: "c58cfb06-4eaa-4e30-8382-ab66354eec5a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:06:55.257641 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.257630    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c58cfb06-4eaa-4e30-8382-ab66354eec5a-istio-envoy" pod "c58cfb06-4eaa-4e30-8382-ab66354eec5a" (UID: "c58cfb06-4eaa-4e30-8382-ab66354eec5a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:55.257847 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.257822    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c58cfb06-4eaa-4e30-8382-ab66354eec5a-kube-api-access-x4sm2" pod "c58cfb06-4eaa-4e30-8382-ab66354eec5a" (UID: "c58cfb06-4eaa-4e30-8382-ab66354eec5a"). InnerVolumeSpecName "kube-api-access-x4sm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:06:55.357065 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.357026    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a88fd37-ae88-4075-8203-024e856be6ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a88fd37-ae88-4075-8203-024e856be6ee-credential-socket\") pod \"0a88fd37-ae88-4075-8203-024e856be6ee\" (UID: \"0a88fd37-ae88-4075-8203-024e856be6ee\") "
Oct 30 21:06:55.357065 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.357072    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a88fd37-ae88-4075-8203-024e856be6ee-istio-token\" (UniqueName: \"kubernetes.io/projected/0a88fd37-ae88-4075-8203-024e856be6ee-istio-token\") pod \"0a88fd37-ae88-4075-8203-024e856be6ee\" (UID: \"0a88fd37-ae88-4075-8203-024e856be6ee\") "
Oct 30 21:06:55.357258 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.357127    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a88fd37-ae88-4075-8203-024e856be6ee-kube-api-access-6hzrh\" (UniqueName: \"kubernetes.io/projected/0a88fd37-ae88-4075-8203-024e856be6ee-kube-api-access-6hzrh\") pod \"0a88fd37-ae88-4075-8203-024e856be6ee\" (UID: \"0a88fd37-ae88-4075-8203-024e856be6ee\") "
Oct 30 21:06:55.357320 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.357307    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a88fd37-ae88-4075-8203-024e856be6ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a88fd37-ae88-4075-8203-024e856be6ee-workload-certs\") pod \"0a88fd37-ae88-4075-8203-024e856be6ee\" (UID: \"0a88fd37-ae88-4075-8203-024e856be6ee\") "
Oct 30 21:06:55.357378 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.357329    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a88fd37-ae88-4075-8203-024e856be6ee-credential-socket" pod "0a88fd37-ae88-4075-8203-024e856be6ee" (UID: "0a88fd37-ae88-4075-8203-024e856be6ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:55.357378 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.357352    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a88fd37-ae88-4075-8203-024e856be6ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a88fd37-ae88-4075-8203-024e856be6ee-istio-podinfo\") pod \"0a88fd37-ae88-4075-8203-024e856be6ee\" (UID: \"0a88fd37-ae88-4075-8203-024e856be6ee\") "
Oct 30 21:06:55.357486 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.357388    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a88fd37-ae88-4075-8203-024e856be6ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a88fd37-ae88-4075-8203-024e856be6ee-istiod-ca-cert\") pod \"0a88fd37-ae88-4075-8203-024e856be6ee\" (UID: \"0a88fd37-ae88-4075-8203-024e856be6ee\") "
Oct 30 21:06:55.357486 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.357441    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a88fd37-ae88-4075-8203-024e856be6ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a88fd37-ae88-4075-8203-024e856be6ee-istio-data\") pod \"0a88fd37-ae88-4075-8203-024e856be6ee\" (UID: \"0a88fd37-ae88-4075-8203-024e856be6ee\") "
Oct 30 21:06:55.357589 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.357531    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a88fd37-ae88-4075-8203-024e856be6ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a88fd37-ae88-4075-8203-024e856be6ee-workload-socket\") pod \"0a88fd37-ae88-4075-8203-024e856be6ee\" (UID: \"0a88fd37-ae88-4075-8203-024e856be6ee\") "
Oct 30 21:06:55.357589 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.357545    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a88fd37-ae88-4075-8203-024e856be6ee-workload-certs" pod "0a88fd37-ae88-4075-8203-024e856be6ee" (UID: "0a88fd37-ae88-4075-8203-024e856be6ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:55.357589 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.357569    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a88fd37-ae88-4075-8203-024e856be6ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a88fd37-ae88-4075-8203-024e856be6ee-istio-envoy\") pod \"0a88fd37-ae88-4075-8203-024e856be6ee\" (UID: \"0a88fd37-ae88-4075-8203-024e856be6ee\") "
Oct 30 21:06:55.357782 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.357762    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a88fd37-ae88-4075-8203-024e856be6ee-istio-data" pod "0a88fd37-ae88-4075-8203-024e856be6ee" (UID: "0a88fd37-ae88-4075-8203-024e856be6ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:55.357867 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.357852    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a88fd37-ae88-4075-8203-024e856be6ee-istiod-ca-cert" pod "0a88fd37-ae88-4075-8203-024e856be6ee" (UID: "0a88fd37-ae88-4075-8203-024e856be6ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:06:55.358178 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.358156    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a88fd37-ae88-4075-8203-024e856be6ee-workload-socket" pod "0a88fd37-ae88-4075-8203-024e856be6ee" (UID: "0a88fd37-ae88-4075-8203-024e856be6ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:55.358617 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.358593    5713 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a88fd37-ae88-4075-8203-024e856be6ee-workload-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:55.358749 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.358624    5713 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a88fd37-ae88-4075-8203-024e856be6ee-istiod-ca-cert\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:55.358749 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.358640    5713 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a88fd37-ae88-4075-8203-024e856be6ee-istio-data\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:55.358749 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.358654    5713 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c58cfb06-4eaa-4e30-8382-ab66354eec5a-istio-token\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:55.358749 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.358670    5713 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c58cfb06-4eaa-4e30-8382-ab66354eec5a-istio-podinfo\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:55.358749 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.358686    5713 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a88fd37-ae88-4075-8203-024e856be6ee-workload-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:55.358749 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.358701    5713 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a88fd37-ae88-4075-8203-024e856be6ee-credential-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:55.358749 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.358717    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4sm2\" (UniqueName: \"kubernetes.io/projected/c58cfb06-4eaa-4e30-8382-ab66354eec5a-kube-api-access-x4sm2\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:55.358749 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.358731    5713 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c58cfb06-4eaa-4e30-8382-ab66354eec5a-istio-envoy\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:55.360265 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.360239    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a88fd37-ae88-4075-8203-024e856be6ee-istio-podinfo" pod "0a88fd37-ae88-4075-8203-024e856be6ee" (UID: "0a88fd37-ae88-4075-8203-024e856be6ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:06:55.360383 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.360362    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a88fd37-ae88-4075-8203-024e856be6ee-istio-token" pod "0a88fd37-ae88-4075-8203-024e856be6ee" (UID: "0a88fd37-ae88-4075-8203-024e856be6ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:06:55.360551 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.360529    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a88fd37-ae88-4075-8203-024e856be6ee-istio-envoy" pod "0a88fd37-ae88-4075-8203-024e856be6ee" (UID: "0a88fd37-ae88-4075-8203-024e856be6ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:55.361806 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.361772    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a88fd37-ae88-4075-8203-024e856be6ee-kube-api-access-6hzrh" pod "0a88fd37-ae88-4075-8203-024e856be6ee" (UID: "0a88fd37-ae88-4075-8203-024e856be6ee"). InnerVolumeSpecName "kube-api-access-6hzrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:06:55.461010 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.460978    5713 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a88fd37-ae88-4075-8203-024e856be6ee-istio-token\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:55.461010 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.461005    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hzrh\" (UniqueName: \"kubernetes.io/projected/0a88fd37-ae88-4075-8203-024e856be6ee-kube-api-access-6hzrh\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:55.461010 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.461015    5713 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a88fd37-ae88-4075-8203-024e856be6ee-istio-podinfo\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:55.461579 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.461024    5713 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a88fd37-ae88-4075-8203-024e856be6ee-istio-envoy\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:06:55.486411 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.486372    5713 generic.go:356] "Generic (PLEG): container finished" podID="c58cfb06-4eaa-4e30-8382-ab66354eec5a" containerID="78cb10ce4647f90759b455fd84f99cb23c65d7290f042a1f206bca954246fb5a" exitCode=0
Oct 30 21:06:55.486564 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.486478    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff"
Oct 30 21:06:55.486564 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.486509    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff" event={"ID":"c58cfb06-4eaa-4e30-8382-ab66354eec5a","Type":"ContainerDied","Data":"78cb10ce4647f90759b455fd84f99cb23c65d7290f042a1f206bca954246fb5a"}
Oct 30 21:06:55.486693 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.486574    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff" event={"ID":"c58cfb06-4eaa-4e30-8382-ab66354eec5a","Type":"ContainerDied","Data":"f478fd1aa02965e07f08eb0089840c6092b22524de6c07341a741072ccbf8ac2"}
Oct 30 21:06:55.486693 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.486607    5713 scope.go:122] "RemoveContainer" containerID="78cb10ce4647f90759b455fd84f99cb23c65d7290f042a1f206bca954246fb5a"
Oct 30 21:06:55.490062 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.490036    5713 generic.go:356] "Generic (PLEG): container finished" podID="0a88fd37-ae88-4075-8203-024e856be6ee" containerID="4bfc0ffcf59e5706d3ec5d7854427e5e6b5ea158791048ed7fb2487dea0f7656" exitCode=0
Oct 30 21:06:55.490198 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.490090    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9"
Oct 30 21:06:55.490198 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.490123    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9" event={"ID":"0a88fd37-ae88-4075-8203-024e856be6ee","Type":"ContainerDied","Data":"4bfc0ffcf59e5706d3ec5d7854427e5e6b5ea158791048ed7fb2487dea0f7656"}
Oct 30 21:06:55.490198 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.490181    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9" event={"ID":"0a88fd37-ae88-4075-8203-024e856be6ee","Type":"ContainerDied","Data":"bd0b37e3c654461d3f8ca4a1005729b134ddad3803064ade331c1a6dbc7f8a4f"}
Oct 30 21:06:55.503047 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.503025    5713 scope.go:122] "RemoveContainer" containerID="78cb10ce4647f90759b455fd84f99cb23c65d7290f042a1f206bca954246fb5a"
Oct 30 21:06:55.503407 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:06:55.503364    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78cb10ce4647f90759b455fd84f99cb23c65d7290f042a1f206bca954246fb5a\": container with ID starting with 78cb10ce4647f90759b455fd84f99cb23c65d7290f042a1f206bca954246fb5a not found: ID does not exist" containerID="78cb10ce4647f90759b455fd84f99cb23c65d7290f042a1f206bca954246fb5a"
Oct 30 21:06:55.503502 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.503400    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78cb10ce4647f90759b455fd84f99cb23c65d7290f042a1f206bca954246fb5a"} err="failed to get container status \"78cb10ce4647f90759b455fd84f99cb23c65d7290f042a1f206bca954246fb5a\": rpc error: code = NotFound desc = could not find container \"78cb10ce4647f90759b455fd84f99cb23c65d7290f042a1f206bca954246fb5a\": container with ID starting with 78cb10ce4647f90759b455fd84f99cb23c65d7290f042a1f206bca954246fb5a not found: ID does not exist"
Oct 30 21:06:55.503502 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.503423    5713 scope.go:122] "RemoveContainer" containerID="4bfc0ffcf59e5706d3ec5d7854427e5e6b5ea158791048ed7fb2487dea0f7656"
Oct 30 21:06:55.515630 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.515614    5713 scope.go:122] "RemoveContainer" containerID="4bfc0ffcf59e5706d3ec5d7854427e5e6b5ea158791048ed7fb2487dea0f7656"
Oct 30 21:06:55.515896 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:06:55.515877    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bfc0ffcf59e5706d3ec5d7854427e5e6b5ea158791048ed7fb2487dea0f7656\": container with ID starting with 4bfc0ffcf59e5706d3ec5d7854427e5e6b5ea158791048ed7fb2487dea0f7656 not found: ID does not exist" containerID="4bfc0ffcf59e5706d3ec5d7854427e5e6b5ea158791048ed7fb2487dea0f7656"
Oct 30 21:06:55.515969 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.515903    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bfc0ffcf59e5706d3ec5d7854427e5e6b5ea158791048ed7fb2487dea0f7656"} err="failed to get container status \"4bfc0ffcf59e5706d3ec5d7854427e5e6b5ea158791048ed7fb2487dea0f7656\": rpc error: code = NotFound desc = could not find container \"4bfc0ffcf59e5706d3ec5d7854427e5e6b5ea158791048ed7fb2487dea0f7656\": container with ID starting with 4bfc0ffcf59e5706d3ec5d7854427e5e6b5ea158791048ed7fb2487dea0f7656 not found: ID does not exist"
Oct 30 21:06:55.687564 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.686978    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9"]
Oct 30 21:06:55.692548 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.692516    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-7gww9"]
Oct 30 21:06:55.761681 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.761607    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-255f05-85b6fd8bf9-wdzgm"]
Oct 30 21:06:55.762373 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.762354    5713 cpu_manager.go:414] "containerMap: removing container" podUID="c58cfb06-4eaa-4e30-8382-ab66354eec5a" containerName="istio-proxy"
Oct 30 21:06:55.762448 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.762375    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c58cfb06-4eaa-4e30-8382-ab66354eec5a" containerName="istio-proxy"
Oct 30 21:06:55.762448 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.762390    5713 cpu_manager.go:414] "containerMap: removing container" podUID="2cbf9725-307f-4b4e-8119-dbd41f83be2a" containerName="revoke-keys"
Oct 30 21:06:55.762448 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.762396    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cbf9725-307f-4b4e-8119-dbd41f83be2a" containerName="revoke-keys"
Oct 30 21:06:55.762448 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.762408    5713 cpu_manager.go:414] "containerMap: removing container" podUID="0a88fd37-ae88-4075-8203-024e856be6ee" containerName="istio-proxy"
Oct 30 21:06:55.762448 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.762413    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a88fd37-ae88-4075-8203-024e856be6ee" containerName="istio-proxy"
Oct 30 21:06:55.762608 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.762518    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cbf9725-307f-4b4e-8119-dbd41f83be2a" containerName="revoke-keys"
Oct 30 21:06:55.762608 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.762526    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a88fd37-ae88-4075-8203-024e856be6ee" containerName="istio-proxy"
Oct 30 21:06:55.762608 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.762542    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="c58cfb06-4eaa-4e30-8382-ab66354eec5a" containerName="istio-proxy"
Oct 30 21:06:55.767447 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.767425    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-255f05-85b6fd8bf9-wdzgm"
Oct 30 21:06:55.784565 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.784515    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff"]
Oct 30 21:06:55.789471 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.789442    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-4tvff"]
Oct 30 21:06:55.821650 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.821620    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-255f05\""
Oct 30 21:06:55.821845 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.821757    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-255f05-dockercfg-pzcz5\""
Oct 30 21:06:55.824984 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.824940    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-255f05-85b6fd8bf9-wdzgm"]
Oct 30 21:06:55.889729 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.889685    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7d22651-156e-4a5f-82f3-3aa7f3afc2b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-255f05-85b6fd8bf9-wdzgm\" (UID: \"a7d22651-156e-4a5f-82f3-3aa7f3afc2b7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-255f05-85b6fd8bf9-wdzgm"
Oct 30 21:06:55.889979 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.889933    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2jd6\" (UniqueName: \"kubernetes.io/projected/a7d22651-156e-4a5f-82f3-3aa7f3afc2b7-kube-api-access-l2jd6\") pod \"payload-pre-processing-e2e-shared-b-w2-255f05-85b6fd8bf9-wdzgm\" (UID: \"a7d22651-156e-4a5f-82f3-3aa7f3afc2b7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-255f05-85b6fd8bf9-wdzgm"
Oct 30 21:06:55.925912 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:55.925863    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-255f05-677cd4895b-ncgcl"]
Oct 30 21:06:56.028865 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:56.010775    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7d22651-156e-4a5f-82f3-3aa7f3afc2b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-255f05-85b6fd8bf9-wdzgm\" (UID: \"a7d22651-156e-4a5f-82f3-3aa7f3afc2b7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-255f05-85b6fd8bf9-wdzgm"
Oct 30 21:06:56.064668 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:56.064633    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2jd6\" (UniqueName: \"kubernetes.io/projected/a7d22651-156e-4a5f-82f3-3aa7f3afc2b7-kube-api-access-l2jd6\") pod \"payload-pre-processing-e2e-shared-b-w2-255f05-85b6fd8bf9-wdzgm\" (UID: \"a7d22651-156e-4a5f-82f3-3aa7f3afc2b7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-255f05-85b6fd8bf9-wdzgm"
Oct 30 21:06:56.065536 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:56.050351    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7d22651-156e-4a5f-82f3-3aa7f3afc2b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-255f05-85b6fd8bf9-wdzgm\" (UID: \"a7d22651-156e-4a5f-82f3-3aa7f3afc2b7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-255f05-85b6fd8bf9-wdzgm"
Oct 30 21:06:56.074249 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:56.074218    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-255f05-677cd4895b-ncgcl"]
Oct 30 21:06:56.074475 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:56.074330    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-255f05-677cd4895b-ncgcl"
Oct 30 21:06:56.122461 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:56.122426    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2jd6\" (UniqueName: \"kubernetes.io/projected/a7d22651-156e-4a5f-82f3-3aa7f3afc2b7-kube-api-access-l2jd6\") pod \"payload-pre-processing-e2e-shared-b-w2-255f05-85b6fd8bf9-wdzgm\" (UID: \"a7d22651-156e-4a5f-82f3-3aa7f3afc2b7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-255f05-85b6fd8bf9-wdzgm"
Oct 30 21:06:56.166547 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:56.166503    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b673d422-2d71-4f9d-bc6a-5bf0d3d09d90-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-255f05-677cd4895b-ncgcl\" (UID: \"b673d422-2d71-4f9d-bc6a-5bf0d3d09d90\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-255f05-677cd4895b-ncgcl"
Oct 30 21:06:56.166750 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:56.166730    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kl7n\" (UniqueName: \"kubernetes.io/projected/b673d422-2d71-4f9d-bc6a-5bf0d3d09d90-kube-api-access-9kl7n\") pod \"payload-processing-e2e-shared-b-w2-255f05-677cd4895b-ncgcl\" (UID: \"b673d422-2d71-4f9d-bc6a-5bf0d3d09d90\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-255f05-677cd4895b-ncgcl"
Oct 30 21:06:56.268613 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:56.268576    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kl7n\" (UniqueName: \"kubernetes.io/projected/b673d422-2d71-4f9d-bc6a-5bf0d3d09d90-kube-api-access-9kl7n\") pod \"payload-processing-e2e-shared-b-w2-255f05-677cd4895b-ncgcl\" (UID: \"b673d422-2d71-4f9d-bc6a-5bf0d3d09d90\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-255f05-677cd4895b-ncgcl"
Oct 30 21:06:56.268782 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:56.268699    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b673d422-2d71-4f9d-bc6a-5bf0d3d09d90-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-255f05-677cd4895b-ncgcl\" (UID: \"b673d422-2d71-4f9d-bc6a-5bf0d3d09d90\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-255f05-677cd4895b-ncgcl"
Oct 30 21:06:56.269408 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:56.269385    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b673d422-2d71-4f9d-bc6a-5bf0d3d09d90-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-255f05-677cd4895b-ncgcl\" (UID: \"b673d422-2d71-4f9d-bc6a-5bf0d3d09d90\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-255f05-677cd4895b-ncgcl"
Oct 30 21:06:56.326581 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:56.326504    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kl7n\" (UniqueName: \"kubernetes.io/projected/b673d422-2d71-4f9d-bc6a-5bf0d3d09d90-kube-api-access-9kl7n\") pod \"payload-processing-e2e-shared-b-w2-255f05-677cd4895b-ncgcl\" (UID: \"b673d422-2d71-4f9d-bc6a-5bf0d3d09d90\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-255f05-677cd4895b-ncgcl"
Oct 30 21:06:56.382651 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:56.382617    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-255f05-85b6fd8bf9-wdzgm"
Oct 30 21:06:56.396014 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:56.395975    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-255f05-677cd4895b-ncgcl"
Oct 30 21:06:56.772848 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:56.772823    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-255f05-85b6fd8bf9-wdzgm"]
Oct 30 21:06:56.773742 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:06:56.773710    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7d22651_156e_4a5f_82f3_3aa7f3afc2b7.slice/crio-30f5abb67174c46c622b7c46364448eb2353929be3b68af83d12138c440b63f0 WatchSource:0}: Error finding container 30f5abb67174c46c622b7c46364448eb2353929be3b68af83d12138c440b63f0: Status 404 returned error can't find the container with id 30f5abb67174c46c622b7c46364448eb2353929be3b68af83d12138c440b63f0
Oct 30 21:06:56.826282 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:56.826252    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-255f05-677cd4895b-ncgcl"]
Oct 30 21:06:56.826539 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:06:56.826515    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb673d422_2d71_4f9d_bc6a_5bf0d3d09d90.slice/crio-fb2cf335c99b3034bab512bbb1a6aa7f14b5489f59b243c91f4cfb736421a1ba WatchSource:0}: Error finding container fb2cf335c99b3034bab512bbb1a6aa7f14b5489f59b243c91f4cfb736421a1ba: Status 404 returned error can't find the container with id fb2cf335c99b3034bab512bbb1a6aa7f14b5489f59b243c91f4cfb736421a1ba
Oct 30 21:06:57.066644 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:57.066555    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a88fd37-ae88-4075-8203-024e856be6ee" path="/var/lib/kubelet/pods/0a88fd37-ae88-4075-8203-024e856be6ee/volumes"
Oct 30 21:06:57.067343 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:57.067324    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c58cfb06-4eaa-4e30-8382-ab66354eec5a" path="/var/lib/kubelet/pods/c58cfb06-4eaa-4e30-8382-ab66354eec5a/volumes"
Oct 30 21:06:57.512125 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:57.512083    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-255f05-85b6fd8bf9-wdzgm" event={"ID":"a7d22651-156e-4a5f-82f3-3aa7f3afc2b7","Type":"ContainerStarted","Data":"b81408ee16cb6c4ccbcde1bfa3cf0935562c54ac8669af9b0d26b0f19e4b6c6c"}
Oct 30 21:06:57.512322 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:57.512165    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-255f05-85b6fd8bf9-wdzgm"
Oct 30 21:06:57.512322 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:57.512193    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-255f05-85b6fd8bf9-wdzgm" event={"ID":"a7d22651-156e-4a5f-82f3-3aa7f3afc2b7","Type":"ContainerStarted","Data":"30f5abb67174c46c622b7c46364448eb2353929be3b68af83d12138c440b63f0"}
Oct 30 21:06:57.513845 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:57.513817    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-255f05-677cd4895b-ncgcl" event={"ID":"b673d422-2d71-4f9d-bc6a-5bf0d3d09d90","Type":"ContainerStarted","Data":"7b3efda7df5ed1a2bc25088c8f89f63289c6de3ba8501285602de7bcaaf385d9"}
Oct 30 21:06:57.513990 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:57.513865    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-255f05-677cd4895b-ncgcl" event={"ID":"b673d422-2d71-4f9d-bc6a-5bf0d3d09d90","Type":"ContainerStarted","Data":"fb2cf335c99b3034bab512bbb1a6aa7f14b5489f59b243c91f4cfb736421a1ba"}
Oct 30 21:06:57.513990 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:57.513913    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-255f05-677cd4895b-ncgcl"
Oct 30 21:06:57.630039 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:57.629989    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-255f05-85b6fd8bf9-wdzgm" podStartSLOduration=2.629975591 podStartE2EDuration="2.629975591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:06:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:06:57.626168391 +0000 UTC m=+1700.690944790" watchObservedRunningTime="2026-10-30 21:06:57.629975591 +0000 UTC m=+1700.694751986"
Oct 30 21:06:58.844173 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:58.844101    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-255f05-677cd4895b-ncgcl" podStartSLOduration=3.844087918 podStartE2EDuration="3.844087918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:06:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:06:57.744042393 +0000 UTC m=+1700.808818792" watchObservedRunningTime="2026-10-30 21:06:58.844087918 +0000 UTC m=+1701.908864315"
Oct 30 21:06:58.844590 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:58.844452    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-19db3d5f-p9jn5"]
Oct 30 21:06:58.849885 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:58.849857    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-19db3d5f-p9jn5"
Oct 30 21:06:58.903715 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:58.903678    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:06:58.903972 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:58.903852    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:06:58.906767 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:58.906740    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-19db3d5f-p9jn5"]
Oct 30 21:06:58.909701 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:58.909676    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c4338e6-6bdc-4211-a2aa-f303d6255366-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-19db3d5f-p9jn5\" (UID: \"1c4338e6-6bdc-4211-a2aa-f303d6255366\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-19db3d5f-p9jn5"
Oct 30 21:06:58.909844 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:58.909719    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbjr4\" (UniqueName: \"kubernetes.io/projected/1c4338e6-6bdc-4211-a2aa-f303d6255366-kube-api-access-nbjr4\") pod \"maas-api-revoke-keys-e2e-mt-19db3d5f-p9jn5\" (UID: \"1c4338e6-6bdc-4211-a2aa-f303d6255366\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-19db3d5f-p9jn5"
Oct 30 21:06:58.910002 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:58.909979    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c4338e6-6bdc-4211-a2aa-f303d6255366-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-19db3d5f-p9jn5\" (UID: \"1c4338e6-6bdc-4211-a2aa-f303d6255366\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-19db3d5f-p9jn5"
Oct 30 21:06:58.951607 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:58.951581    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:06:58.951721 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:58.951636    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:06:59.011853 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:59.011817    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c4338e6-6bdc-4211-a2aa-f303d6255366-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-19db3d5f-p9jn5\" (UID: \"1c4338e6-6bdc-4211-a2aa-f303d6255366\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-19db3d5f-p9jn5"
Oct 30 21:06:59.012011 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:59.011992    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c4338e6-6bdc-4211-a2aa-f303d6255366-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-19db3d5f-p9jn5\" (UID: \"1c4338e6-6bdc-4211-a2aa-f303d6255366\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-19db3d5f-p9jn5"
Oct 30 21:06:59.012106 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:59.012034    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbjr4\" (UniqueName: \"kubernetes.io/projected/1c4338e6-6bdc-4211-a2aa-f303d6255366-kube-api-access-nbjr4\") pod \"maas-api-revoke-keys-e2e-mt-19db3d5f-p9jn5\" (UID: \"1c4338e6-6bdc-4211-a2aa-f303d6255366\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-19db3d5f-p9jn5"
Oct 30 21:06:59.012642 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:59.012599    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c4338e6-6bdc-4211-a2aa-f303d6255366-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-19db3d5f-p9jn5\" (UID: \"1c4338e6-6bdc-4211-a2aa-f303d6255366\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-19db3d5f-p9jn5"
Oct 30 21:06:59.015017 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:59.014993    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c4338e6-6bdc-4211-a2aa-f303d6255366-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-19db3d5f-p9jn5\" (UID: \"1c4338e6-6bdc-4211-a2aa-f303d6255366\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-19db3d5f-p9jn5"
Oct 30 21:06:59.077406 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:59.077379    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbjr4\" (UniqueName: \"kubernetes.io/projected/1c4338e6-6bdc-4211-a2aa-f303d6255366-kube-api-access-nbjr4\") pod \"maas-api-revoke-keys-e2e-mt-19db3d5f-p9jn5\" (UID: \"1c4338e6-6bdc-4211-a2aa-f303d6255366\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-19db3d5f-p9jn5"
Oct 30 21:06:59.163328 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:59.163253    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-19db3d5f-p9jn5"
Oct 30 21:06:59.550042 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:06:59.550012    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-19db3d5f-p9jn5"]
Oct 30 21:06:59.550374 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:06:59.550321    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c4338e6_6bdc_4211_a2aa_f303d6255366.slice/crio-48083dd1414f162a432f4d824e98c9164d5c9ce23ecd6fb6d57ee0a3224a84de WatchSource:0}: Error finding container 48083dd1414f162a432f4d824e98c9164d5c9ce23ecd6fb6d57ee0a3224a84de: Status 404 returned error can't find the container with id 48083dd1414f162a432f4d824e98c9164d5c9ce23ecd6fb6d57ee0a3224a84de
Oct 30 21:07:00.541152 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:00.541118    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-19db3d5f-p9jn5" event={"ID":"1c4338e6-6bdc-4211-a2aa-f303d6255366","Type":"ContainerStarted","Data":"763a2733f60e8ca6ca8111e5ae1da1f6bdc71c6e49d31a67fc45e13bc6015eed"}
Oct 30 21:07:00.541762 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:00.541173    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-19db3d5f-p9jn5" event={"ID":"1c4338e6-6bdc-4211-a2aa-f303d6255366","Type":"ContainerStarted","Data":"48083dd1414f162a432f4d824e98c9164d5c9ce23ecd6fb6d57ee0a3224a84de"}
Oct 30 21:07:00.656884 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:00.656829    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-19db3d5f-p9jn5" podStartSLOduration=2.656812321 podStartE2EDuration="2.656812321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:06:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:07:00.65420897 +0000 UTC m=+1703.718985379" watchObservedRunningTime="2026-10-30 21:07:00.656812321 +0000 UTC m=+1703.721588776"
Oct 30 21:07:01.551041 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:01.551010    5713 generic.go:356] "Generic (PLEG): container finished" podID="1c4338e6-6bdc-4211-a2aa-f303d6255366" containerID="763a2733f60e8ca6ca8111e5ae1da1f6bdc71c6e49d31a67fc45e13bc6015eed" exitCode=0
Oct 30 21:07:01.551448 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:01.551095    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-19db3d5f-p9jn5" event={"ID":"1c4338e6-6bdc-4211-a2aa-f303d6255366","Type":"ContainerDied","Data":"763a2733f60e8ca6ca8111e5ae1da1f6bdc71c6e49d31a67fc45e13bc6015eed"}
Oct 30 21:07:02.717139 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:02.717108    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-19db3d5f-p9jn5"
Oct 30 21:07:02.872071 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:02.871976    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c4338e6-6bdc-4211-a2aa-f303d6255366-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c4338e6-6bdc-4211-a2aa-f303d6255366-maas-api-xks-ca\") pod \"1c4338e6-6bdc-4211-a2aa-f303d6255366\" (UID: \"1c4338e6-6bdc-4211-a2aa-f303d6255366\") "
Oct 30 21:07:02.872071 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:02.872035    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c4338e6-6bdc-4211-a2aa-f303d6255366-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c4338e6-6bdc-4211-a2aa-f303d6255366-maas-api-openshift-ca\") pod \"1c4338e6-6bdc-4211-a2aa-f303d6255366\" (UID: \"1c4338e6-6bdc-4211-a2aa-f303d6255366\") "
Oct 30 21:07:02.872307 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:02.872080    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c4338e6-6bdc-4211-a2aa-f303d6255366-kube-api-access-nbjr4\" (UniqueName: \"kubernetes.io/projected/1c4338e6-6bdc-4211-a2aa-f303d6255366-kube-api-access-nbjr4\") pod \"1c4338e6-6bdc-4211-a2aa-f303d6255366\" (UID: \"1c4338e6-6bdc-4211-a2aa-f303d6255366\") "
Oct 30 21:07:02.872480 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:02.872451    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c4338e6-6bdc-4211-a2aa-f303d6255366-maas-api-openshift-ca" pod "1c4338e6-6bdc-4211-a2aa-f303d6255366" (UID: "1c4338e6-6bdc-4211-a2aa-f303d6255366"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:07:02.873058 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:02.873032    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c4338e6-6bdc-4211-a2aa-f303d6255366-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:07:02.874366 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:02.874341    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c4338e6-6bdc-4211-a2aa-f303d6255366-kube-api-access-nbjr4" pod "1c4338e6-6bdc-4211-a2aa-f303d6255366" (UID: "1c4338e6-6bdc-4211-a2aa-f303d6255366"). InnerVolumeSpecName "kube-api-access-nbjr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:07:02.874863 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:02.874778    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c4338e6-6bdc-4211-a2aa-f303d6255366-maas-api-xks-ca" pod "1c4338e6-6bdc-4211-a2aa-f303d6255366" (UID: "1c4338e6-6bdc-4211-a2aa-f303d6255366"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:07:02.974353 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:02.974317    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c4338e6-6bdc-4211-a2aa-f303d6255366-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:07:02.974353 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:02.974354    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbjr4\" (UniqueName: \"kubernetes.io/projected/1c4338e6-6bdc-4211-a2aa-f303d6255366-kube-api-access-nbjr4\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:07:03.568145 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:03.568109    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-19db3d5f-p9jn5"
Oct 30 21:07:03.568316 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:03.568147    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-19db3d5f-p9jn5" event={"ID":"1c4338e6-6bdc-4211-a2aa-f303d6255366","Type":"ContainerDied","Data":"48083dd1414f162a432f4d824e98c9164d5c9ce23ecd6fb6d57ee0a3224a84de"}
Oct 30 21:07:03.568316 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:03.568196    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48083dd1414f162a432f4d824e98c9164d5c9ce23ecd6fb6d57ee0a3224a84de"
Oct 30 21:07:08.525484 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:08.525456    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-255f05-677cd4895b-ncgcl"
Oct 30 21:07:08.526096 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:08.525520    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-255f05-85b6fd8bf9-wdzgm"
Oct 30 21:07:10.871052 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:10.871015    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-19db3d5f-7c8fc84467-lblmt"]
Oct 30 21:07:15.871614 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:15.871551    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-19db3d5f-7c8fc84467-lblmt" podUID="ed8f7bb0-daa5-45b6-bee7-2689677ea6a5" containerName="payload-pre-processing" containerID="cri-o://bd2c0893c37df605aa71e805f482e09d9460ab784c9ee2386f81ac75f6ee5523" gracePeriod=25
Oct 30 21:07:16.157109 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:16.157079    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-19db3d5f-7c8fc84467-lblmt"
Oct 30 21:07:16.280544 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:16.280513    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed8f7bb0-daa5-45b6-bee7-2689677ea6a5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed8f7bb0-daa5-45b6-bee7-2689677ea6a5-plugins-config-volume\") pod \"ed8f7bb0-daa5-45b6-bee7-2689677ea6a5\" (UID: \"ed8f7bb0-daa5-45b6-bee7-2689677ea6a5\") "
Oct 30 21:07:16.280740 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:16.280682    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed8f7bb0-daa5-45b6-bee7-2689677ea6a5-kube-api-access-r2pkq\" (UniqueName: \"kubernetes.io/projected/ed8f7bb0-daa5-45b6-bee7-2689677ea6a5-kube-api-access-r2pkq\") pod \"ed8f7bb0-daa5-45b6-bee7-2689677ea6a5\" (UID: \"ed8f7bb0-daa5-45b6-bee7-2689677ea6a5\") "
Oct 30 21:07:16.280884 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:16.280860    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed8f7bb0-daa5-45b6-bee7-2689677ea6a5-plugins-config-volume" pod "ed8f7bb0-daa5-45b6-bee7-2689677ea6a5" (UID: "ed8f7bb0-daa5-45b6-bee7-2689677ea6a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:07:16.281145 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:16.281128    5713 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed8f7bb0-daa5-45b6-bee7-2689677ea6a5-plugins-config-volume\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:07:16.283169 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:16.283051    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed8f7bb0-daa5-45b6-bee7-2689677ea6a5-kube-api-access-r2pkq" pod "ed8f7bb0-daa5-45b6-bee7-2689677ea6a5" (UID: "ed8f7bb0-daa5-45b6-bee7-2689677ea6a5"). InnerVolumeSpecName "kube-api-access-r2pkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:07:16.382546 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:16.382503    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2pkq\" (UniqueName: \"kubernetes.io/projected/ed8f7bb0-daa5-45b6-bee7-2689677ea6a5-kube-api-access-r2pkq\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:07:16.675619 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:16.675530    5713 generic.go:356] "Generic (PLEG): container finished" podID="ed8f7bb0-daa5-45b6-bee7-2689677ea6a5" containerID="bd2c0893c37df605aa71e805f482e09d9460ab784c9ee2386f81ac75f6ee5523" exitCode=0
Oct 30 21:07:16.675619 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:16.675573    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-19db3d5f-7c8fc84467-lblmt"
Oct 30 21:07:16.675619 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:16.675583    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-19db3d5f-7c8fc84467-lblmt" event={"ID":"ed8f7bb0-daa5-45b6-bee7-2689677ea6a5","Type":"ContainerDied","Data":"bd2c0893c37df605aa71e805f482e09d9460ab784c9ee2386f81ac75f6ee5523"}
Oct 30 21:07:16.675931 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:16.675634    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-19db3d5f-7c8fc84467-lblmt" event={"ID":"ed8f7bb0-daa5-45b6-bee7-2689677ea6a5","Type":"ContainerDied","Data":"3924a0d6e31b3b528445b349a7e0526750193a4c54385a1b7198f7437d5fdd35"}
Oct 30 21:07:16.675931 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:16.675662    5713 scope.go:122] "RemoveContainer" containerID="bd2c0893c37df605aa71e805f482e09d9460ab784c9ee2386f81ac75f6ee5523"
Oct 30 21:07:16.688087 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:16.688064    5713 scope.go:122] "RemoveContainer" containerID="bd2c0893c37df605aa71e805f482e09d9460ab784c9ee2386f81ac75f6ee5523"
Oct 30 21:07:16.688394 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:07:16.688374    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd2c0893c37df605aa71e805f482e09d9460ab784c9ee2386f81ac75f6ee5523\": container with ID starting with bd2c0893c37df605aa71e805f482e09d9460ab784c9ee2386f81ac75f6ee5523 not found: ID does not exist" containerID="bd2c0893c37df605aa71e805f482e09d9460ab784c9ee2386f81ac75f6ee5523"
Oct 30 21:07:16.688448 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:16.688402    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd2c0893c37df605aa71e805f482e09d9460ab784c9ee2386f81ac75f6ee5523"} err="failed to get container status \"bd2c0893c37df605aa71e805f482e09d9460ab784c9ee2386f81ac75f6ee5523\": rpc error: code = NotFound desc = could not find container \"bd2c0893c37df605aa71e805f482e09d9460ab784c9ee2386f81ac75f6ee5523\": container with ID starting with bd2c0893c37df605aa71e805f482e09d9460ab784c9ee2386f81ac75f6ee5523 not found: ID does not exist"
Oct 30 21:07:16.859496 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:16.859462    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-19db3d5f-7c8fc84467-lblmt"]
Oct 30 21:07:16.908425 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:16.908395    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-19db3d5f-7c8fc84467-lblmt"]
Oct 30 21:07:17.064056 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:17.064019    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed8f7bb0-daa5-45b6-bee7-2689677ea6a5" path="/var/lib/kubelet/pods/ed8f7bb0-daa5-45b6-bee7-2689677ea6a5/volumes"
Oct 30 21:07:24.459745 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:07:24.459703    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:07:38.335029 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:38.334999    5713 scope.go:122] "RemoveContainer" containerID="f9650b438486baaebd42a342d20223691b469a09e17a89de329aba9c3a302650"
Oct 30 21:07:38.345878 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:38.345857    5713 scope.go:122] "RemoveContainer" containerID="6c4f2e2984a531950649833eb2594d0d1b93e76a9d791882f3f0d2e35e43c07f"
Oct 30 21:07:38.355698 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:38.355681    5713 scope.go:122] "RemoveContainer" containerID="938bc496bb9e8f8d48fc0367306b91902888465072a459f7bcc820b511ca5319"
Oct 30 21:07:38.364871 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:38.364852    5713 scope.go:122] "RemoveContainer" containerID="dfc7d722ce40f8f2dc54f4c0035bffb3427dad285e0af969ae5e38086eb47105"
Oct 30 21:07:38.374222 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:38.374202    5713 scope.go:122] "RemoveContainer" containerID="6b5629c2a2cbde3e949ca0e55b04cfdf8785a5b9e5f750ac66d5df408de585db"
Oct 30 21:07:38.384105 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:38.384090    5713 scope.go:122] "RemoveContainer" containerID="3b4855f2379d671680e3bfab4ea34141f77ef81df034b2e58678405d8923aa2d"
Oct 30 21:07:38.393084 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:38.393065    5713 scope.go:122] "RemoveContainer" containerID="87c8b7a07675405eb16ccb472f8aa5a30c2cf1c1c0a90b9626ad1fda019d858f"
Oct 30 21:07:38.401718 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:38.401699    5713 scope.go:122] "RemoveContainer" containerID="82dca2879f5ad45dc89f16a1132a80d1e06b803231c358bb7202def49f156f3b"
Oct 30 21:07:40.284799 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:40.284761    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5"]
Oct 30 21:07:40.285397 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:40.285382    5713 cpu_manager.go:414] "containerMap: removing container" podUID="ed8f7bb0-daa5-45b6-bee7-2689677ea6a5" containerName="payload-pre-processing"
Oct 30 21:07:40.285442 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:40.285399    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed8f7bb0-daa5-45b6-bee7-2689677ea6a5" containerName="payload-pre-processing"
Oct 30 21:07:40.285482 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:40.285449    5713 cpu_manager.go:414] "containerMap: removing container" podUID="1c4338e6-6bdc-4211-a2aa-f303d6255366" containerName="revoke-keys"
Oct 30 21:07:40.285482 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:40.285454    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c4338e6-6bdc-4211-a2aa-f303d6255366" containerName="revoke-keys"
Oct 30 21:07:40.285556 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:40.285546    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c4338e6-6bdc-4211-a2aa-f303d6255366" containerName="revoke-keys"
Oct 30 21:07:40.285595 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:40.285559    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed8f7bb0-daa5-45b6-bee7-2689677ea6a5" containerName="payload-pre-processing"
Oct 30 21:07:40.291494 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:40.291470    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5"
Oct 30 21:07:40.399561 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:40.351513    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-b888d0-openshift-default-dockercfg-rnlkd\""
Oct 30 21:07:40.428780 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:40.426775    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3000e9c-41b7-4667-b9e1-c90ecf6c9b63-istiod-ca-cert\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5\" (UID: \"e3000e9c-41b7-4667-b9e1-c90ecf6c9b63\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5"
Oct 30 21:07:40.428780 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:40.426867    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3000e9c-41b7-4667-b9e1-c90ecf6c9b63-istio-token\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5\" (UID: \"e3000e9c-41b7-4667-b9e1-c90ecf6c9b63\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5"
Oct 30 21:07:40.428780 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:40.426917    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98z4s\" (UniqueName: \"kubernetes.io/projected/e3000e9c-41b7-4667-b9e1-c90ecf6c9b63-kube-api-access-98z4s\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5\" (UID: \"e3000e9c-41b7-4667-b9e1-c90ecf6c9b63\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5"
Oct 30 21:07:40.428780 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:40.426977    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3000e9c-41b7-4667-b9e1-c90ecf6c9b63-istio-data\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5\" (UID: \"e3000e9c-41b7-4667-b9e1-c90ecf6c9b63\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5"
Oct 30 21:07:40.428780 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:40.427012    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3000e9c-41b7-4667-b9e1-c90ecf6c9b63-workload-socket\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5\" (UID: \"e3000e9c-41b7-4667-b9e1-c90ecf6c9b63\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5"
Oct 30 21:07:40.428780 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:40.427094    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3000e9c-41b7-4667-b9e1-c90ecf6c9b63-workload-certs\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5\" (UID: \"e3000e9c-41b7-4667-b9e1-c90ecf6c9b63\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5"
Oct 30 21:07:40.428780 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:40.427185    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3000e9c-41b7-4667-b9e1-c90ecf6c9b63-credential-socket\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5\" (UID: \"e3000e9c-41b7-4667-b9e1-c90ecf6c9b63\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5"
Oct 30 21:07:40.428780 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:40.427304    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3000e9c-41b7-4667-b9e1-c90ecf6c9b63-istio-podinfo\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5\" (UID: \"e3000e9c-41b7-4667-b9e1-c90ecf6c9b63\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5"
Oct 30 21:07:40.428780 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:40.427373    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3000e9c-41b7-4667-b9e1-c90ecf6c9b63-istio-envoy\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5\" (UID: \"e3000e9c-41b7-4667-b9e1-c90ecf6c9b63\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5"
Oct 30 21:07:40.432252 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:40.432223    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5"]
Oct 30 21:07:40.528471 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:40.528437    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3000e9c-41b7-4667-b9e1-c90ecf6c9b63-workload-certs\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5\" (UID: \"e3000e9c-41b7-4667-b9e1-c90ecf6c9b63\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5"
Oct 30 21:07:40.528663 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:40.528517    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3000e9c-41b7-4667-b9e1-c90ecf6c9b63-credential-socket\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5\" (UID: \"e3000e9c-41b7-4667-b9e1-c90ecf6c9b63\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5"
Oct 30 21:07:40.528663 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:40.528606    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3000e9c-41b7-4667-b9e1-c90ecf6c9b63-istio-podinfo\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5\" (UID: \"e3000e9c-41b7-4667-b9e1-c90ecf6c9b63\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5"
Oct 30 21:07:40.528663 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:40.528647    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3000e9c-41b7-4667-b9e1-c90ecf6c9b63-istio-envoy\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5\" (UID: \"e3000e9c-41b7-4667-b9e1-c90ecf6c9b63\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5"
Oct 30 21:07:40.528922 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:40.528673    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3000e9c-41b7-4667-b9e1-c90ecf6c9b63-istiod-ca-cert\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5\" (UID: \"e3000e9c-41b7-4667-b9e1-c90ecf6c9b63\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5"
Oct 30 21:07:40.528922 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:40.528696    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3000e9c-41b7-4667-b9e1-c90ecf6c9b63-istio-token\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5\" (UID: \"e3000e9c-41b7-4667-b9e1-c90ecf6c9b63\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5"
Oct 30 21:07:40.528922 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:40.528738    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98z4s\" (UniqueName: \"kubernetes.io/projected/e3000e9c-41b7-4667-b9e1-c90ecf6c9b63-kube-api-access-98z4s\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5\" (UID: \"e3000e9c-41b7-4667-b9e1-c90ecf6c9b63\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5"
Oct 30 21:07:40.528922 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:40.528841    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3000e9c-41b7-4667-b9e1-c90ecf6c9b63-istio-data\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5\" (UID: \"e3000e9c-41b7-4667-b9e1-c90ecf6c9b63\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5"
Oct 30 21:07:40.528922 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:40.528881    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3000e9c-41b7-4667-b9e1-c90ecf6c9b63-workload-socket\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5\" (UID: \"e3000e9c-41b7-4667-b9e1-c90ecf6c9b63\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5"
Oct 30 21:07:40.528922 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:40.528910    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3000e9c-41b7-4667-b9e1-c90ecf6c9b63-workload-certs\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5\" (UID: \"e3000e9c-41b7-4667-b9e1-c90ecf6c9b63\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5"
Oct 30 21:07:40.529243 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:40.528941    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3000e9c-41b7-4667-b9e1-c90ecf6c9b63-credential-socket\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5\" (UID: \"e3000e9c-41b7-4667-b9e1-c90ecf6c9b63\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5"
Oct 30 21:07:40.529340 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:40.529318    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3000e9c-41b7-4667-b9e1-c90ecf6c9b63-istio-data\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5\" (UID: \"e3000e9c-41b7-4667-b9e1-c90ecf6c9b63\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5"
Oct 30 21:07:40.529381 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:40.529347    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3000e9c-41b7-4667-b9e1-c90ecf6c9b63-workload-socket\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5\" (UID: \"e3000e9c-41b7-4667-b9e1-c90ecf6c9b63\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5"
Oct 30 21:07:40.529439 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:40.529413    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3000e9c-41b7-4667-b9e1-c90ecf6c9b63-istiod-ca-cert\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5\" (UID: \"e3000e9c-41b7-4667-b9e1-c90ecf6c9b63\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5"
Oct 30 21:07:40.531223 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:40.531200    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3000e9c-41b7-4667-b9e1-c90ecf6c9b63-istio-envoy\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5\" (UID: \"e3000e9c-41b7-4667-b9e1-c90ecf6c9b63\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5"
Oct 30 21:07:40.531325 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:40.531304    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3000e9c-41b7-4667-b9e1-c90ecf6c9b63-istio-podinfo\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5\" (UID: \"e3000e9c-41b7-4667-b9e1-c90ecf6c9b63\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5"
Oct 30 21:07:40.587318 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:40.587228    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3000e9c-41b7-4667-b9e1-c90ecf6c9b63-istio-token\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5\" (UID: \"e3000e9c-41b7-4667-b9e1-c90ecf6c9b63\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5"
Oct 30 21:07:40.587479 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:40.587337    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98z4s\" (UniqueName: \"kubernetes.io/projected/e3000e9c-41b7-4667-b9e1-c90ecf6c9b63-kube-api-access-98z4s\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5\" (UID: \"e3000e9c-41b7-4667-b9e1-c90ecf6c9b63\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5"
Oct 30 21:07:40.606460 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:40.606425    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5"
Oct 30 21:07:41.088526 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:41.083165    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5"]
Oct 30 21:07:41.088952 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:07:41.088917    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3000e9c_41b7_4667_b9e1_c90ecf6c9b63.slice/crio-47cb405db7a850b2dec06d05e817bdd6d83202f5d089033655351c6b263fd8ca WatchSource:0}: Error finding container 47cb405db7a850b2dec06d05e817bdd6d83202f5d089033655351c6b263fd8ca: Status 404 returned error can't find the container with id 47cb405db7a850b2dec06d05e817bdd6d83202f5d089033655351c6b263fd8ca
Oct 30 21:07:41.091912 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:41.091875    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:07:41.092003 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:41.091956    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:07:41.092003 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:41.091995    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:07:41.951630 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:41.951591    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5" event={"ID":"e3000e9c-41b7-4667-b9e1-c90ecf6c9b63","Type":"ContainerStarted","Data":"75d41fbab6583d1312c43e434c531e6b47564a0ab9ed3aacb7382a1705032798"}
Oct 30 21:07:41.952073 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:41.951656    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5" event={"ID":"e3000e9c-41b7-4667-b9e1-c90ecf6c9b63","Type":"ContainerStarted","Data":"47cb405db7a850b2dec06d05e817bdd6d83202f5d089033655351c6b263fd8ca"}
Oct 30 21:07:42.095531 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:42.095481    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5" podStartSLOduration=2.095462619 podStartE2EDuration="2.095462619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:07:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:07:42.092236835 +0000 UTC m=+1745.157013216" watchObservedRunningTime="2026-10-30 21:07:42.095462619 +0000 UTC m=+1745.160239122"
Oct 30 21:07:42.606767 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:42.606732    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5"
Oct 30 21:07:42.612167 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:42.612135    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5"
Oct 30 21:07:42.959730 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:42.959697    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5"
Oct 30 21:07:42.960552 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:42.960526    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5"
Oct 30 21:07:50.366469 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:50.366431    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b888d0-75cbbbf7c5-rnfbj"]
Oct 30 21:07:50.388390 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:50.388342    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b888d0-75cbbbf7c5-rnfbj"
Oct 30 21:07:50.422705 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:50.422668    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppjz9\" (UniqueName: \"kubernetes.io/projected/8335fd09-7966-4952-8214-1e569d15f4fa-kube-api-access-ppjz9\") pod \"payload-pre-processing-e2e-api-a-b888d0-75cbbbf7c5-rnfbj\" (UID: \"8335fd09-7966-4952-8214-1e569d15f4fa\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b888d0-75cbbbf7c5-rnfbj"
Oct 30 21:07:50.422917 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:50.422841    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8335fd09-7966-4952-8214-1e569d15f4fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b888d0-75cbbbf7c5-rnfbj\" (UID: \"8335fd09-7966-4952-8214-1e569d15f4fa\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b888d0-75cbbbf7c5-rnfbj"
Oct 30 21:07:50.425871 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:50.425845    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b888d0-75cbbbf7c5-rnfbj"]
Oct 30 21:07:50.470858 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:50.470829    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-b888d0-dockercfg-ccrlm\""
Oct 30 21:07:50.471068 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:50.470882    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-b888d0\""
Oct 30 21:07:50.569817 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:50.564439    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppjz9\" (UniqueName: \"kubernetes.io/projected/8335fd09-7966-4952-8214-1e569d15f4fa-kube-api-access-ppjz9\") pod \"payload-pre-processing-e2e-api-a-b888d0-75cbbbf7c5-rnfbj\" (UID: \"8335fd09-7966-4952-8214-1e569d15f4fa\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b888d0-75cbbbf7c5-rnfbj"
Oct 30 21:07:50.569817 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:50.564929    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8335fd09-7966-4952-8214-1e569d15f4fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b888d0-75cbbbf7c5-rnfbj\" (UID: \"8335fd09-7966-4952-8214-1e569d15f4fa\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b888d0-75cbbbf7c5-rnfbj"
Oct 30 21:07:50.569817 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:50.566262    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8335fd09-7966-4952-8214-1e569d15f4fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b888d0-75cbbbf7c5-rnfbj\" (UID: \"8335fd09-7966-4952-8214-1e569d15f4fa\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b888d0-75cbbbf7c5-rnfbj"
Oct 30 21:07:50.570803 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:50.570758    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b888d0-79557999cd-rzz2r"]
Oct 30 21:07:50.578341 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:50.578315    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b888d0-79557999cd-rzz2r"
Oct 30 21:07:50.590914 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:50.590885    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b888d0-79557999cd-rzz2r"]
Oct 30 21:07:50.634251 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:50.622754    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppjz9\" (UniqueName: \"kubernetes.io/projected/8335fd09-7966-4952-8214-1e569d15f4fa-kube-api-access-ppjz9\") pod \"payload-pre-processing-e2e-api-a-b888d0-75cbbbf7c5-rnfbj\" (UID: \"8335fd09-7966-4952-8214-1e569d15f4fa\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b888d0-75cbbbf7c5-rnfbj"
Oct 30 21:07:50.666979 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:50.666933    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twdbl\" (UniqueName: \"kubernetes.io/projected/fbd6d24d-5227-4534-8fe8-1288fef79e88-kube-api-access-twdbl\") pod \"payload-processing-e2e-api-a-b888d0-79557999cd-rzz2r\" (UID: \"fbd6d24d-5227-4534-8fe8-1288fef79e88\") " pod="openshift-ingress/payload-processing-e2e-api-a-b888d0-79557999cd-rzz2r"
Oct 30 21:07:50.667382 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:50.667357    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbd6d24d-5227-4534-8fe8-1288fef79e88-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b888d0-79557999cd-rzz2r\" (UID: \"fbd6d24d-5227-4534-8fe8-1288fef79e88\") " pod="openshift-ingress/payload-processing-e2e-api-a-b888d0-79557999cd-rzz2r"
Oct 30 21:07:50.706544 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:50.706512    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b888d0-75cbbbf7c5-rnfbj"
Oct 30 21:07:50.771041 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:50.770990    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twdbl\" (UniqueName: \"kubernetes.io/projected/fbd6d24d-5227-4534-8fe8-1288fef79e88-kube-api-access-twdbl\") pod \"payload-processing-e2e-api-a-b888d0-79557999cd-rzz2r\" (UID: \"fbd6d24d-5227-4534-8fe8-1288fef79e88\") " pod="openshift-ingress/payload-processing-e2e-api-a-b888d0-79557999cd-rzz2r"
Oct 30 21:07:50.771179 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:50.771094    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbd6d24d-5227-4534-8fe8-1288fef79e88-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b888d0-79557999cd-rzz2r\" (UID: \"fbd6d24d-5227-4534-8fe8-1288fef79e88\") " pod="openshift-ingress/payload-processing-e2e-api-a-b888d0-79557999cd-rzz2r"
Oct 30 21:07:50.772260 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:50.772229    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbd6d24d-5227-4534-8fe8-1288fef79e88-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b888d0-79557999cd-rzz2r\" (UID: \"fbd6d24d-5227-4534-8fe8-1288fef79e88\") " pod="openshift-ingress/payload-processing-e2e-api-a-b888d0-79557999cd-rzz2r"
Oct 30 21:07:50.828685 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:50.828645    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twdbl\" (UniqueName: \"kubernetes.io/projected/fbd6d24d-5227-4534-8fe8-1288fef79e88-kube-api-access-twdbl\") pod \"payload-processing-e2e-api-a-b888d0-79557999cd-rzz2r\" (UID: \"fbd6d24d-5227-4534-8fe8-1288fef79e88\") " pod="openshift-ingress/payload-processing-e2e-api-a-b888d0-79557999cd-rzz2r"
Oct 30 21:07:50.915192 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:50.915115    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b888d0-79557999cd-rzz2r"
Oct 30 21:07:51.113420 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:51.113392    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b888d0-75cbbbf7c5-rnfbj"]
Oct 30 21:07:51.113881 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:07:51.113852    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8335fd09_7966_4952_8214_1e569d15f4fa.slice/crio-6b9a1b966d9288db770f0b9b757c2d0ee1e89d110d126afa57bdc22f0c237836 WatchSource:0}: Error finding container 6b9a1b966d9288db770f0b9b757c2d0ee1e89d110d126afa57bdc22f0c237836: Status 404 returned error can't find the container with id 6b9a1b966d9288db770f0b9b757c2d0ee1e89d110d126afa57bdc22f0c237836
Oct 30 21:07:51.299593 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:51.299561    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b888d0-79557999cd-rzz2r"]
Oct 30 21:07:51.300205 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:07:51.300178    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbd6d24d_5227_4534_8fe8_1288fef79e88.slice/crio-7c84ee175b3e4d58a86759147d5fb8a1c18ada0fa8e595a0501f11d64aea6336 WatchSource:0}: Error finding container 7c84ee175b3e4d58a86759147d5fb8a1c18ada0fa8e595a0501f11d64aea6336: Status 404 returned error can't find the container with id 7c84ee175b3e4d58a86759147d5fb8a1c18ada0fa8e595a0501f11d64aea6336
Oct 30 21:07:52.050283 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:52.050234    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b888d0-75cbbbf7c5-rnfbj" event={"ID":"8335fd09-7966-4952-8214-1e569d15f4fa","Type":"ContainerStarted","Data":"e260163d30b28d68dc8894d67faef810fc5c9f95afc79d7b92a1657ba94c01a7"}
Oct 30 21:07:52.050743 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:52.050323    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b888d0-75cbbbf7c5-rnfbj" event={"ID":"8335fd09-7966-4952-8214-1e569d15f4fa","Type":"ContainerStarted","Data":"6b9a1b966d9288db770f0b9b757c2d0ee1e89d110d126afa57bdc22f0c237836"}
Oct 30 21:07:52.050743 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:52.050381    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b888d0-75cbbbf7c5-rnfbj"
Oct 30 21:07:52.052639 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:52.052610    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b888d0-79557999cd-rzz2r" event={"ID":"fbd6d24d-5227-4534-8fe8-1288fef79e88","Type":"ContainerStarted","Data":"358ecb28cf28938ce27fbeae78493d6cc8dcac24f619f217e356572da4728a2c"}
Oct 30 21:07:52.052766 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:52.052661    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b888d0-79557999cd-rzz2r" event={"ID":"fbd6d24d-5227-4534-8fe8-1288fef79e88","Type":"ContainerStarted","Data":"7c84ee175b3e4d58a86759147d5fb8a1c18ada0fa8e595a0501f11d64aea6336"}
Oct 30 21:07:52.052766 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:52.052717    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-b888d0-79557999cd-rzz2r"
Oct 30 21:07:52.181025 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:52.180954    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b888d0-75cbbbf7c5-rnfbj" podStartSLOduration=2.180932497 podStartE2EDuration="2.180932497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:07:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:07:52.173157025 +0000 UTC m=+1755.237933425" watchObservedRunningTime="2026-10-30 21:07:52.180932497 +0000 UTC m=+1755.245708896"
Oct 30 21:07:52.290457 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:52.290395    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-b888d0-79557999cd-rzz2r" podStartSLOduration=2.290376017 podStartE2EDuration="2.290376017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:07:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:07:52.289577757 +0000 UTC m=+1755.354354155" watchObservedRunningTime="2026-10-30 21:07:52.290376017 +0000 UTC m=+1755.355152415"
Oct 30 21:07:54.460010 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:07:54.459972    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:07:56.048419 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:56.048387    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5fc0962-5mcg4"]
Oct 30 21:07:56.098157 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:56.098129    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5fc0962-5mcg4"]
Oct 30 21:07:57.067081 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:57.067048    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5104496-d047-41dc-9ed4-550e8c89725b" path="/var/lib/kubelet/pods/a5104496-d047-41dc-9ed4-550e8c89725b/volumes"
Oct 30 21:07:59.341705 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:59.341672    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-19db3d5f-p9jn5"]
Oct 30 21:07:59.390390 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:07:59.390358    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-19db3d5f-p9jn5"]
Oct 30 21:08:01.064486 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:01.064447    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c4338e6-6bdc-4211-a2aa-f303d6255366" path="/var/lib/kubelet/pods/1c4338e6-6bdc-4211-a2aa-f303d6255366/volumes"
Oct 30 21:08:03.064677 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:03.064644    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b888d0-75cbbbf7c5-rnfbj"
Oct 30 21:08:03.065102 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:03.064709    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-b888d0-79557999cd-rzz2r"
Oct 30 21:08:08.281264 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:08.281228    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q"]
Oct 30 21:08:08.285962 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:08.285850    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q"
Oct 30 21:08:08.299940 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:08.299908    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec28cb7b-925f-4b32-9ff8-d75caf2f3d14-credential-socket\") pod \"e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q\" (UID: \"ec28cb7b-925f-4b32-9ff8-d75caf2f3d14\") " pod="openshift-ingress/e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q"
Oct 30 21:08:08.300100 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:08.299979    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec28cb7b-925f-4b32-9ff8-d75caf2f3d14-istio-envoy\") pod \"e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q\" (UID: \"ec28cb7b-925f-4b32-9ff8-d75caf2f3d14\") " pod="openshift-ingress/e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q"
Oct 30 21:08:08.300100 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:08.300003    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec28cb7b-925f-4b32-9ff8-d75caf2f3d14-istio-podinfo\") pod \"e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q\" (UID: \"ec28cb7b-925f-4b32-9ff8-d75caf2f3d14\") " pod="openshift-ingress/e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q"
Oct 30 21:08:08.300100 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:08.300030    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec28cb7b-925f-4b32-9ff8-d75caf2f3d14-workload-socket\") pod \"e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q\" (UID: \"ec28cb7b-925f-4b32-9ff8-d75caf2f3d14\") " pod="openshift-ingress/e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q"
Oct 30 21:08:08.300285 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:08.300127    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec28cb7b-925f-4b32-9ff8-d75caf2f3d14-workload-certs\") pod \"e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q\" (UID: \"ec28cb7b-925f-4b32-9ff8-d75caf2f3d14\") " pod="openshift-ingress/e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q"
Oct 30 21:08:08.300285 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:08.300228    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gklj\" (UniqueName: \"kubernetes.io/projected/ec28cb7b-925f-4b32-9ff8-d75caf2f3d14-kube-api-access-4gklj\") pod \"e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q\" (UID: \"ec28cb7b-925f-4b32-9ff8-d75caf2f3d14\") " pod="openshift-ingress/e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q"
Oct 30 21:08:08.300285 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:08.300274    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec28cb7b-925f-4b32-9ff8-d75caf2f3d14-istiod-ca-cert\") pod \"e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q\" (UID: \"ec28cb7b-925f-4b32-9ff8-d75caf2f3d14\") " pod="openshift-ingress/e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q"
Oct 30 21:08:08.300436 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:08.300417    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec28cb7b-925f-4b32-9ff8-d75caf2f3d14-istio-token\") pod \"e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q\" (UID: \"ec28cb7b-925f-4b32-9ff8-d75caf2f3d14\") " pod="openshift-ingress/e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q"
Oct 30 21:08:08.300492 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:08.300485    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec28cb7b-925f-4b32-9ff8-d75caf2f3d14-istio-data\") pod \"e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q\" (UID: \"ec28cb7b-925f-4b32-9ff8-d75caf2f3d14\") " pod="openshift-ingress/e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q"
Oct 30 21:08:08.411683 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:08.411647    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-9926f7-openshift-default-dockercfg-5x8t5\""
Oct 30 21:08:08.412736 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:08.412673    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec28cb7b-925f-4b32-9ff8-d75caf2f3d14-istiod-ca-cert\") pod \"e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q\" (UID: \"ec28cb7b-925f-4b32-9ff8-d75caf2f3d14\") " pod="openshift-ingress/e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q"
Oct 30 21:08:08.413008 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:08.412954    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec28cb7b-925f-4b32-9ff8-d75caf2f3d14-istiod-ca-cert\") pod \"e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q\" (UID: \"ec28cb7b-925f-4b32-9ff8-d75caf2f3d14\") " pod="openshift-ingress/e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q"
Oct 30 21:08:08.413219 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:08.413204    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec28cb7b-925f-4b32-9ff8-d75caf2f3d14-istio-token\") pod \"e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q\" (UID: \"ec28cb7b-925f-4b32-9ff8-d75caf2f3d14\") " pod="openshift-ingress/e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q"
Oct 30 21:08:08.413380 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:08.413366    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec28cb7b-925f-4b32-9ff8-d75caf2f3d14-istio-data\") pod \"e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q\" (UID: \"ec28cb7b-925f-4b32-9ff8-d75caf2f3d14\") " pod="openshift-ingress/e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q"
Oct 30 21:08:08.414234 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:08.414212    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec28cb7b-925f-4b32-9ff8-d75caf2f3d14-credential-socket\") pod \"e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q\" (UID: \"ec28cb7b-925f-4b32-9ff8-d75caf2f3d14\") " pod="openshift-ingress/e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q"
Oct 30 21:08:08.414469 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:08.414452    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec28cb7b-925f-4b32-9ff8-d75caf2f3d14-istio-envoy\") pod \"e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q\" (UID: \"ec28cb7b-925f-4b32-9ff8-d75caf2f3d14\") " pod="openshift-ingress/e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q"
Oct 30 21:08:08.414576 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:08.414562    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec28cb7b-925f-4b32-9ff8-d75caf2f3d14-istio-podinfo\") pod \"e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q\" (UID: \"ec28cb7b-925f-4b32-9ff8-d75caf2f3d14\") " pod="openshift-ingress/e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q"
Oct 30 21:08:08.414682 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:08.414670    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec28cb7b-925f-4b32-9ff8-d75caf2f3d14-workload-socket\") pod \"e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q\" (UID: \"ec28cb7b-925f-4b32-9ff8-d75caf2f3d14\") " pod="openshift-ingress/e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q"
Oct 30 21:08:08.414782 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:08.414770    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec28cb7b-925f-4b32-9ff8-d75caf2f3d14-workload-certs\") pod \"e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q\" (UID: \"ec28cb7b-925f-4b32-9ff8-d75caf2f3d14\") " pod="openshift-ingress/e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q"
Oct 30 21:08:08.416333 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:08.416313    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gklj\" (UniqueName: \"kubernetes.io/projected/ec28cb7b-925f-4b32-9ff8-d75caf2f3d14-kube-api-access-4gklj\") pod \"e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q\" (UID: \"ec28cb7b-925f-4b32-9ff8-d75caf2f3d14\") " pod="openshift-ingress/e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q"
Oct 30 21:08:08.416578 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:08.413730    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec28cb7b-925f-4b32-9ff8-d75caf2f3d14-istio-data\") pod \"e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q\" (UID: \"ec28cb7b-925f-4b32-9ff8-d75caf2f3d14\") " pod="openshift-ingress/e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q"
Oct 30 21:08:08.416699 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:08.415564    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec28cb7b-925f-4b32-9ff8-d75caf2f3d14-credential-socket\") pod \"e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q\" (UID: \"ec28cb7b-925f-4b32-9ff8-d75caf2f3d14\") " pod="openshift-ingress/e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q"
Oct 30 21:08:08.416810 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:08.416192    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec28cb7b-925f-4b32-9ff8-d75caf2f3d14-workload-socket\") pod \"e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q\" (UID: \"ec28cb7b-925f-4b32-9ff8-d75caf2f3d14\") " pod="openshift-ingress/e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q"
Oct 30 21:08:08.416921 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:08.415131    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec28cb7b-925f-4b32-9ff8-d75caf2f3d14-workload-certs\") pod \"e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q\" (UID: \"ec28cb7b-925f-4b32-9ff8-d75caf2f3d14\") " pod="openshift-ingress/e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q"
Oct 30 21:08:08.421962 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:08.421934    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q"]
Oct 30 21:08:08.432582 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:08.432545    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec28cb7b-925f-4b32-9ff8-d75caf2f3d14-istio-envoy\") pod \"e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q\" (UID: \"ec28cb7b-925f-4b32-9ff8-d75caf2f3d14\") " pod="openshift-ingress/e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q"
Oct 30 21:08:08.440294 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:08.440227    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec28cb7b-925f-4b32-9ff8-d75caf2f3d14-istio-podinfo\") pod \"e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q\" (UID: \"ec28cb7b-925f-4b32-9ff8-d75caf2f3d14\") " pod="openshift-ingress/e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q"
Oct 30 21:08:08.471591 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:08.471561    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec28cb7b-925f-4b32-9ff8-d75caf2f3d14-istio-token\") pod \"e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q\" (UID: \"ec28cb7b-925f-4b32-9ff8-d75caf2f3d14\") " pod="openshift-ingress/e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q"
Oct 30 21:08:08.474193 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:08.474165    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gklj\" (UniqueName: \"kubernetes.io/projected/ec28cb7b-925f-4b32-9ff8-d75caf2f3d14-kube-api-access-4gklj\") pod \"e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q\" (UID: \"ec28cb7b-925f-4b32-9ff8-d75caf2f3d14\") " pod="openshift-ingress/e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q"
Oct 30 21:08:08.602260 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:08.602160    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q"
Oct 30 21:08:09.048320 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:09.048285    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q"]
Oct 30 21:08:09.052665 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:09.052629    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:08:09.052857 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:09.052721    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:08:09.052857 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:09.052766    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:08:09.243378 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:09.243341    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q" event={"ID":"ec28cb7b-925f-4b32-9ff8-d75caf2f3d14","Type":"ContainerStarted","Data":"00705fe9f3bf9f96294b219bba02b81f3ba219ce426d7b3825bfee73c1b9f2d7"}
Oct 30 21:08:09.243574 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:09.243397    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q" event={"ID":"ec28cb7b-925f-4b32-9ff8-d75caf2f3d14","Type":"ContainerStarted","Data":"674d527d0732fa066e00a978fb6b289f3b5b3ebd50f28d16724b43c7129937be"}
Oct 30 21:08:09.362016 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:09.361903    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q" podStartSLOduration=1.361886608 podStartE2EDuration="1.361886608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:08:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:08:09.359940893 +0000 UTC m=+1772.424717293" watchObservedRunningTime="2026-10-30 21:08:09.361886608 +0000 UTC m=+1772.426663007"
Oct 30 21:08:09.602408 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:09.602376    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q"
Oct 30 21:08:10.607925 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:10.607893    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q"
Oct 30 21:08:11.267181 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:11.267143    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q"
Oct 30 21:08:11.268068 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:11.268049    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q"
Oct 30 21:08:24.459735 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:08:24.459697    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:08:29.344233 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:29.344203    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40b7b51a-8dtkf"]
Oct 30 21:08:29.348554 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:29.348536    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40b7b51a-8dtkf"
Oct 30 21:08:29.402298 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:29.402264    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:08:29.450477 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:29.450437    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szvjb\" (UniqueName: \"kubernetes.io/projected/3da62021-1436-4278-9434-26b7d9feea6b-kube-api-access-szvjb\") pod \"maas-api-revoke-sub-40b7b51a-8dtkf\" (UID: \"3da62021-1436-4278-9434-26b7d9feea6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40b7b51a-8dtkf"
Oct 30 21:08:29.450671 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:29.450518    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3da62021-1436-4278-9434-26b7d9feea6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40b7b51a-8dtkf\" (UID: \"3da62021-1436-4278-9434-26b7d9feea6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40b7b51a-8dtkf"
Oct 30 21:08:29.450671 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:29.450618    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3da62021-1436-4278-9434-26b7d9feea6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40b7b51a-8dtkf\" (UID: \"3da62021-1436-4278-9434-26b7d9feea6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40b7b51a-8dtkf"
Oct 30 21:08:29.451098 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:29.451069    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:08:29.451279 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:29.451080    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:08:29.451279 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:29.451176    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:08:29.455076 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:29.455054    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40b7b51a-8dtkf"]
Oct 30 21:08:29.551965 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:29.551927    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szvjb\" (UniqueName: \"kubernetes.io/projected/3da62021-1436-4278-9434-26b7d9feea6b-kube-api-access-szvjb\") pod \"maas-api-revoke-sub-40b7b51a-8dtkf\" (UID: \"3da62021-1436-4278-9434-26b7d9feea6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40b7b51a-8dtkf"
Oct 30 21:08:29.552196 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:29.552000    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3da62021-1436-4278-9434-26b7d9feea6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40b7b51a-8dtkf\" (UID: \"3da62021-1436-4278-9434-26b7d9feea6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40b7b51a-8dtkf"
Oct 30 21:08:29.552196 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:29.552045    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3da62021-1436-4278-9434-26b7d9feea6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40b7b51a-8dtkf\" (UID: \"3da62021-1436-4278-9434-26b7d9feea6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40b7b51a-8dtkf"
Oct 30 21:08:29.552662 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:29.552633    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3da62021-1436-4278-9434-26b7d9feea6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40b7b51a-8dtkf\" (UID: \"3da62021-1436-4278-9434-26b7d9feea6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40b7b51a-8dtkf"
Oct 30 21:08:29.554639 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:29.554622    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3da62021-1436-4278-9434-26b7d9feea6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40b7b51a-8dtkf\" (UID: \"3da62021-1436-4278-9434-26b7d9feea6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40b7b51a-8dtkf"
Oct 30 21:08:29.610601 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:29.610534    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szvjb\" (UniqueName: \"kubernetes.io/projected/3da62021-1436-4278-9434-26b7d9feea6b-kube-api-access-szvjb\") pod \"maas-api-revoke-sub-40b7b51a-8dtkf\" (UID: \"3da62021-1436-4278-9434-26b7d9feea6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40b7b51a-8dtkf"
Oct 30 21:08:29.663753 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:29.663632    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40b7b51a-8dtkf"
Oct 30 21:08:30.043776 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:30.043742    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40b7b51a-8dtkf"]
Oct 30 21:08:30.043917 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:08:30.043773    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3da62021_1436_4278_9434_26b7d9feea6b.slice/crio-8862aba8326eeff02d380e239a46c58bc700db166b82cf0923bae24be2ece004 WatchSource:0}: Error finding container 8862aba8326eeff02d380e239a46c58bc700db166b82cf0923bae24be2ece004: Status 404 returned error can't find the container with id 8862aba8326eeff02d380e239a46c58bc700db166b82cf0923bae24be2ece004
Oct 30 21:08:30.423325 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:30.423291    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40b7b51a-8dtkf" event={"ID":"3da62021-1436-4278-9434-26b7d9feea6b","Type":"ContainerStarted","Data":"94cff754d8a6cf5cfe7ee7b26444ae6c4d46e15361940e6a4c03da889d1844c7"}
Oct 30 21:08:30.423729 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:30.423346    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40b7b51a-8dtkf" event={"ID":"3da62021-1436-4278-9434-26b7d9feea6b","Type":"ContainerStarted","Data":"8862aba8326eeff02d380e239a46c58bc700db166b82cf0923bae24be2ece004"}
Oct 30 21:08:30.541402 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:30.541353    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40b7b51a-8dtkf" podStartSLOduration=1.5413362290000001 podStartE2EDuration="1.541336229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:08:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:08:30.537025896 +0000 UTC m=+1793.601802307" watchObservedRunningTime="2026-10-30 21:08:30.541336229 +0000 UTC m=+1793.606112627"
Oct 30 21:08:31.445475 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:31.445437    5713 generic.go:356] "Generic (PLEG): container finished" podID="3da62021-1436-4278-9434-26b7d9feea6b" containerID="94cff754d8a6cf5cfe7ee7b26444ae6c4d46e15361940e6a4c03da889d1844c7" exitCode=0
Oct 30 21:08:31.445908 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:31.445520    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40b7b51a-8dtkf" event={"ID":"3da62021-1436-4278-9434-26b7d9feea6b","Type":"ContainerDied","Data":"94cff754d8a6cf5cfe7ee7b26444ae6c4d46e15361940e6a4c03da889d1844c7"}
Oct 30 21:08:32.618553 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:32.618526    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40b7b51a-8dtkf"
Oct 30 21:08:32.704069 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:32.703991    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3da62021-1436-4278-9434-26b7d9feea6b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3da62021-1436-4278-9434-26b7d9feea6b-maas-api-xks-ca\") pod \"3da62021-1436-4278-9434-26b7d9feea6b\" (UID: \"3da62021-1436-4278-9434-26b7d9feea6b\") "
Oct 30 21:08:32.704229 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:32.704119    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3da62021-1436-4278-9434-26b7d9feea6b-kube-api-access-szvjb\" (UniqueName: \"kubernetes.io/projected/3da62021-1436-4278-9434-26b7d9feea6b-kube-api-access-szvjb\") pod \"3da62021-1436-4278-9434-26b7d9feea6b\" (UID: \"3da62021-1436-4278-9434-26b7d9feea6b\") "
Oct 30 21:08:32.704409 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:32.704388    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3da62021-1436-4278-9434-26b7d9feea6b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3da62021-1436-4278-9434-26b7d9feea6b-maas-api-openshift-ca\") pod \"3da62021-1436-4278-9434-26b7d9feea6b\" (UID: \"3da62021-1436-4278-9434-26b7d9feea6b\") "
Oct 30 21:08:32.704748 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:32.704719    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3da62021-1436-4278-9434-26b7d9feea6b-maas-api-openshift-ca" pod "3da62021-1436-4278-9434-26b7d9feea6b" (UID: "3da62021-1436-4278-9434-26b7d9feea6b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:08:32.705076 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:32.705055    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3da62021-1436-4278-9434-26b7d9feea6b-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:08:32.706371 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:32.706342    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3da62021-1436-4278-9434-26b7d9feea6b-maas-api-xks-ca" pod "3da62021-1436-4278-9434-26b7d9feea6b" (UID: "3da62021-1436-4278-9434-26b7d9feea6b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:08:32.706451 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:32.706389    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3da62021-1436-4278-9434-26b7d9feea6b-kube-api-access-szvjb" pod "3da62021-1436-4278-9434-26b7d9feea6b" (UID: "3da62021-1436-4278-9434-26b7d9feea6b"). InnerVolumeSpecName "kube-api-access-szvjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:08:32.807075 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:32.807037    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3da62021-1436-4278-9434-26b7d9feea6b-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:08:32.807075 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:32.807070    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szvjb\" (UniqueName: \"kubernetes.io/projected/3da62021-1436-4278-9434-26b7d9feea6b-kube-api-access-szvjb\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:08:33.462517 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:33.462486    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40b7b51a-8dtkf"
Oct 30 21:08:33.462703 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:33.462517    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40b7b51a-8dtkf" event={"ID":"3da62021-1436-4278-9434-26b7d9feea6b","Type":"ContainerDied","Data":"8862aba8326eeff02d380e239a46c58bc700db166b82cf0923bae24be2ece004"}
Oct 30 21:08:33.462703 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:33.462566    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8862aba8326eeff02d380e239a46c58bc700db166b82cf0923bae24be2ece004"
Oct 30 21:08:37.487289 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:37.487258    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d4597073-8ff0-4a4e-933c-c64e934fe977/prometheus/0.log"
Oct 30 21:08:37.491907 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:37.491886    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d4597073-8ff0-4a4e-933c-c64e934fe977/prometheus/0.log"
Oct 30 21:08:37.502713 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:37.502689    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r7lpf_764461d5-e0b2-4856-a97e-0c5d3d34a497/ovn-acl-logging/0.log"
Oct 30 21:08:37.506752 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:08:37.506731    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r7lpf_764461d5-e0b2-4856-a97e-0c5d3d34a497/ovn-acl-logging/0.log"
Oct 30 21:08:54.459556 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:08:54.459465    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:09:03.921814 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:03.920966    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-682999/distinct-w3-682999-kserve-74d49d4955-nndlc" podUID="2a35314b-31ea-42e6-b49a-15ab1305b338" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:09:03.935273 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:03.935228    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7" podUID="b2193213-ebae-498a-911a-6056a8e94312" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:09:03.992042 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:03.991991    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm" podUID="cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:09:04.021133 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:04.021086    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b425d/simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk" podUID="e8108824-03ff-4c1d-9f5b-07be3a3a92d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:09:04.027548 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:04.027507    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b425d/premium-w5-2b425d-kserve-78756b6c5b-9x5xb" podUID="abf7b512-50e5-4400-9ca3-9ffccaead3d7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:09:04.054920 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:04.054871    5713 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2b425d/simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk" podUID="e8108824-03ff-4c1d-9f5b-07be3a3a92d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:09:04.133487 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:04.132231    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-682999/distinct-2-w3-682999-kserve-7bc687885b-gq5x7" podUID="0dcf0422-081f-46d7-b246-7a49b074c88c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:09:04.157711 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:04.156912    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b425d/distinct-2-w5-2b425d-kserve-6694867c68-bzpvn" podUID="b9d391e6-de6f-476d-a926-d1b333474109" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:09:04.183814 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:04.179902    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-781058/premium-w0-781058-kserve-7f5c8964c9-z5fmk" podUID="57626f4f-844e-4ec9-ab89-359d015923af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 30 21:09:04.205815 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:04.204884    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-682999/simulator-w3-682999-kserve-57cfcb6f8f-6mgrg" podUID="53296e7f-c6bd-45f2-ac31-004feddc63a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:09:04.205815 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:04.205742    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l" podUID="fc9510c5-2180-41d6-90bc-9acdff0f3616" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:09:04.438992 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:04.438851    5713 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l" podUID="fc9510c5-2180-41d6-90bc-9acdff0f3616" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:09:05.047421 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:05.047387    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2e56076-87sh8"]
Oct 30 21:09:05.096679 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:05.096645    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2e56076-87sh8"]
Oct 30 21:09:07.066567 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:07.066529    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6124e97b-a754-4495-a0d9-61b63533a579" path="/var/lib/kubelet/pods/6124e97b-a754-4495-a0d9-61b63533a579/volumes"
Oct 30 21:09:17.022632 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:17.022592    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ef7e2d-77489c76c6-vf7cz"]
Oct 30 21:09:17.024604 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:17.024576    5713 cpu_manager.go:414] "containerMap: removing container" podUID="3da62021-1436-4278-9434-26b7d9feea6b" containerName="revoke-keys"
Oct 30 21:09:17.024604 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:17.024604    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3da62021-1436-4278-9434-26b7d9feea6b" containerName="revoke-keys"
Oct 30 21:09:17.024882 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:17.024867    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="3da62021-1436-4278-9434-26b7d9feea6b" containerName="revoke-keys"
Oct 30 21:09:17.036110 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:17.033239    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ef7e2d-77489c76c6-vf7cz"
Oct 30 21:09:17.047479 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:17.047439    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwxsr\" (UniqueName: \"kubernetes.io/projected/1afb9a0a-9a10-4f71-b66d-a7cc873ed48b-kube-api-access-zwxsr\") pod \"payload-processing-e2e-api-a-ef7e2d-77489c76c6-vf7cz\" (UID: \"1afb9a0a-9a10-4f71-b66d-a7cc873ed48b\") " pod="openshift-ingress/payload-processing-e2e-api-a-ef7e2d-77489c76c6-vf7cz"
Oct 30 21:09:17.047721 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:17.047702    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1afb9a0a-9a10-4f71-b66d-a7cc873ed48b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ef7e2d-77489c76c6-vf7cz\" (UID: \"1afb9a0a-9a10-4f71-b66d-a7cc873ed48b\") " pod="openshift-ingress/payload-processing-e2e-api-a-ef7e2d-77489c76c6-vf7cz"
Oct 30 21:09:17.080405 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:17.080371    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ef7e2d-77489c76c6-vf7cz"]
Oct 30 21:09:17.128719 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:17.128686    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-ef7e2d\""
Oct 30 21:09:17.128983 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:17.128686    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-ef7e2d-dockercfg-rdvgr\""
Oct 30 21:09:17.151878 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:17.150171    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwxsr\" (UniqueName: \"kubernetes.io/projected/1afb9a0a-9a10-4f71-b66d-a7cc873ed48b-kube-api-access-zwxsr\") pod \"payload-processing-e2e-api-a-ef7e2d-77489c76c6-vf7cz\" (UID: \"1afb9a0a-9a10-4f71-b66d-a7cc873ed48b\") " pod="openshift-ingress/payload-processing-e2e-api-a-ef7e2d-77489c76c6-vf7cz"
Oct 30 21:09:17.151878 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:17.150381    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1afb9a0a-9a10-4f71-b66d-a7cc873ed48b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ef7e2d-77489c76c6-vf7cz\" (UID: \"1afb9a0a-9a10-4f71-b66d-a7cc873ed48b\") " pod="openshift-ingress/payload-processing-e2e-api-a-ef7e2d-77489c76c6-vf7cz"
Oct 30 21:09:17.151878 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:17.151644    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1afb9a0a-9a10-4f71-b66d-a7cc873ed48b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ef7e2d-77489c76c6-vf7cz\" (UID: \"1afb9a0a-9a10-4f71-b66d-a7cc873ed48b\") " pod="openshift-ingress/payload-processing-e2e-api-a-ef7e2d-77489c76c6-vf7cz"
Oct 30 21:09:17.214830 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:17.214765    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwxsr\" (UniqueName: \"kubernetes.io/projected/1afb9a0a-9a10-4f71-b66d-a7cc873ed48b-kube-api-access-zwxsr\") pod \"payload-processing-e2e-api-a-ef7e2d-77489c76c6-vf7cz\" (UID: \"1afb9a0a-9a10-4f71-b66d-a7cc873ed48b\") " pod="openshift-ingress/payload-processing-e2e-api-a-ef7e2d-77489c76c6-vf7cz"
Oct 30 21:09:17.271955 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:17.271870    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ef7e2d-6784699ff9-mcvmc"]
Oct 30 21:09:17.281175 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:17.281106    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef7e2d-6784699ff9-mcvmc"
Oct 30 21:09:17.293132 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:17.293099    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ef7e2d-6784699ff9-mcvmc"]
Oct 30 21:09:17.355989 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:17.355957    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r9qt\" (UniqueName: \"kubernetes.io/projected/e9200d55-faeb-4087-85cb-949f64353a55-kube-api-access-9r9qt\") pod \"payload-pre-processing-e2e-api-a-ef7e2d-6784699ff9-mcvmc\" (UID: \"e9200d55-faeb-4087-85cb-949f64353a55\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef7e2d-6784699ff9-mcvmc"
Oct 30 21:09:17.356182 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:17.356110    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9200d55-faeb-4087-85cb-949f64353a55-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ef7e2d-6784699ff9-mcvmc\" (UID: \"e9200d55-faeb-4087-85cb-949f64353a55\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef7e2d-6784699ff9-mcvmc"
Oct 30 21:09:17.362498 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:17.362470    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ef7e2d-77489c76c6-vf7cz"
Oct 30 21:09:17.477915 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:17.463261    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r9qt\" (UniqueName: \"kubernetes.io/projected/e9200d55-faeb-4087-85cb-949f64353a55-kube-api-access-9r9qt\") pod \"payload-pre-processing-e2e-api-a-ef7e2d-6784699ff9-mcvmc\" (UID: \"e9200d55-faeb-4087-85cb-949f64353a55\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef7e2d-6784699ff9-mcvmc"
Oct 30 21:09:17.477915 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:17.463409    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9200d55-faeb-4087-85cb-949f64353a55-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ef7e2d-6784699ff9-mcvmc\" (UID: \"e9200d55-faeb-4087-85cb-949f64353a55\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef7e2d-6784699ff9-mcvmc"
Oct 30 21:09:17.477915 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:17.464449    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9200d55-faeb-4087-85cb-949f64353a55-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ef7e2d-6784699ff9-mcvmc\" (UID: \"e9200d55-faeb-4087-85cb-949f64353a55\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef7e2d-6784699ff9-mcvmc"
Oct 30 21:09:17.525555 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:17.525458    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r9qt\" (UniqueName: \"kubernetes.io/projected/e9200d55-faeb-4087-85cb-949f64353a55-kube-api-access-9r9qt\") pod \"payload-pre-processing-e2e-api-a-ef7e2d-6784699ff9-mcvmc\" (UID: \"e9200d55-faeb-4087-85cb-949f64353a55\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef7e2d-6784699ff9-mcvmc"
Oct 30 21:09:17.607160 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:17.607128    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef7e2d-6784699ff9-mcvmc"
Oct 30 21:09:17.873518 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:17.873489    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ef7e2d-77489c76c6-vf7cz"]
Oct 30 21:09:17.875293 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:09:17.875259    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1afb9a0a_9a10_4f71_b66d_a7cc873ed48b.slice/crio-60a7de002565b5a1d27966a0fdaeadd7f15ed0ab4527f0b936b4bd200efa40b1 WatchSource:0}: Error finding container 60a7de002565b5a1d27966a0fdaeadd7f15ed0ab4527f0b936b4bd200efa40b1: Status 404 returned error can't find the container with id 60a7de002565b5a1d27966a0fdaeadd7f15ed0ab4527f0b936b4bd200efa40b1
Oct 30 21:09:17.936863 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:17.936829    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ef7e2d-77489c76c6-vf7cz" event={"ID":"1afb9a0a-9a10-4f71-b66d-a7cc873ed48b","Type":"ContainerStarted","Data":"60a7de002565b5a1d27966a0fdaeadd7f15ed0ab4527f0b936b4bd200efa40b1"}
Oct 30 21:09:17.992475 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:17.992452    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ef7e2d-6784699ff9-mcvmc"]
Oct 30 21:09:17.992969 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:09:17.992940    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9200d55_faeb_4087_85cb_949f64353a55.slice/crio-bf8ad350c9a382b03c3ace48692854066b8f837b51a592284b09af8cbdce595f WatchSource:0}: Error finding container bf8ad350c9a382b03c3ace48692854066b8f837b51a592284b09af8cbdce595f: Status 404 returned error can't find the container with id bf8ad350c9a382b03c3ace48692854066b8f837b51a592284b09af8cbdce595f
Oct 30 21:09:18.947765 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:18.947730    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef7e2d-6784699ff9-mcvmc" event={"ID":"e9200d55-faeb-4087-85cb-949f64353a55","Type":"ContainerStarted","Data":"3eff780b5ce92b1e268fbe5531fd500409f36519abc1c8c66dd97e1a44d90e24"}
Oct 30 21:09:18.948276 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:18.947782    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef7e2d-6784699ff9-mcvmc" event={"ID":"e9200d55-faeb-4087-85cb-949f64353a55","Type":"ContainerStarted","Data":"bf8ad350c9a382b03c3ace48692854066b8f837b51a592284b09af8cbdce595f"}
Oct 30 21:09:18.948276 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:18.947829    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef7e2d-6784699ff9-mcvmc"
Oct 30 21:09:18.949468 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:18.949444    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ef7e2d-77489c76c6-vf7cz" event={"ID":"1afb9a0a-9a10-4f71-b66d-a7cc873ed48b","Type":"ContainerStarted","Data":"1e64ef0131528605e58d15b41272cf5521a51caaa6017cb61da4e52d0a137107"}
Oct 30 21:09:18.949603 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:18.949565    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-ef7e2d-77489c76c6-vf7cz"
Oct 30 21:09:19.063177 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:19.063125    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef7e2d-6784699ff9-mcvmc" podStartSLOduration=2.063111923 podStartE2EDuration="2.063111923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:19.062015357 +0000 UTC m=+1842.126791755" watchObservedRunningTime="2026-10-30 21:09:19.063111923 +0000 UTC m=+1842.127888320"
Oct 30 21:09:19.198302 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:19.198189    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-ef7e2d-77489c76c6-vf7cz" podStartSLOduration=3.198171223 podStartE2EDuration="3.198171223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:19.194596448 +0000 UTC m=+1842.259372846" watchObservedRunningTime="2026-10-30 21:09:19.198171223 +0000 UTC m=+1842.262947620"
Oct 30 21:09:23.067771 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:23.067743    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk"]
Oct 30 21:09:23.072570 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:23.072545    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk"
Oct 30 21:09:23.134309 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:23.134273    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk"]
Oct 30 21:09:23.151672 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:23.151640    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbdw9\" (UniqueName: \"kubernetes.io/projected/ad23561f-a24f-4c13-9df7-0ecce2596034-kube-api-access-vbdw9\") pod \"e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk\" (UID: \"ad23561f-a24f-4c13-9df7-0ecce2596034\") " pod="openshift-ingress/e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk"
Oct 30 21:09:23.151872 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:23.151736    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad23561f-a24f-4c13-9df7-0ecce2596034-credential-socket\") pod \"e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk\" (UID: \"ad23561f-a24f-4c13-9df7-0ecce2596034\") " pod="openshift-ingress/e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk"
Oct 30 21:09:23.151872 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:23.151783    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad23561f-a24f-4c13-9df7-0ecce2596034-workload-socket\") pod \"e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk\" (UID: \"ad23561f-a24f-4c13-9df7-0ecce2596034\") " pod="openshift-ingress/e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk"
Oct 30 21:09:23.152013 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:23.151993    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad23561f-a24f-4c13-9df7-0ecce2596034-istiod-ca-cert\") pod \"e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk\" (UID: \"ad23561f-a24f-4c13-9df7-0ecce2596034\") " pod="openshift-ingress/e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk"
Oct 30 21:09:23.152117 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:23.152080    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad23561f-a24f-4c13-9df7-0ecce2596034-istio-token\") pod \"e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk\" (UID: \"ad23561f-a24f-4c13-9df7-0ecce2596034\") " pod="openshift-ingress/e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk"
Oct 30 21:09:23.152197 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:23.152176    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad23561f-a24f-4c13-9df7-0ecce2596034-workload-certs\") pod \"e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk\" (UID: \"ad23561f-a24f-4c13-9df7-0ecce2596034\") " pod="openshift-ingress/e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk"
Oct 30 21:09:23.152313 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:23.152298    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad23561f-a24f-4c13-9df7-0ecce2596034-istio-data\") pod \"e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk\" (UID: \"ad23561f-a24f-4c13-9df7-0ecce2596034\") " pod="openshift-ingress/e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk"
Oct 30 21:09:23.152368 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:23.152331    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad23561f-a24f-4c13-9df7-0ecce2596034-istio-podinfo\") pod \"e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk\" (UID: \"ad23561f-a24f-4c13-9df7-0ecce2596034\") " pod="openshift-ingress/e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk"
Oct 30 21:09:23.152435 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:23.152417    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad23561f-a24f-4c13-9df7-0ecce2596034-istio-envoy\") pod \"e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk\" (UID: \"ad23561f-a24f-4c13-9df7-0ecce2596034\") " pod="openshift-ingress/e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk"
Oct 30 21:09:23.177847 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:23.177826    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-677b45-openshift-default-dockercfg-cdjqs\""
Oct 30 21:09:23.254456 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:23.254421    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad23561f-a24f-4c13-9df7-0ecce2596034-istio-data\") pod \"e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk\" (UID: \"ad23561f-a24f-4c13-9df7-0ecce2596034\") " pod="openshift-ingress/e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk"
Oct 30 21:09:23.254456 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:23.254462    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad23561f-a24f-4c13-9df7-0ecce2596034-istio-podinfo\") pod \"e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk\" (UID: \"ad23561f-a24f-4c13-9df7-0ecce2596034\") " pod="openshift-ingress/e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk"
Oct 30 21:09:23.254731 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:23.254497    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad23561f-a24f-4c13-9df7-0ecce2596034-istio-envoy\") pod \"e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk\" (UID: \"ad23561f-a24f-4c13-9df7-0ecce2596034\") " pod="openshift-ingress/e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk"
Oct 30 21:09:23.254731 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:23.254583    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbdw9\" (UniqueName: \"kubernetes.io/projected/ad23561f-a24f-4c13-9df7-0ecce2596034-kube-api-access-vbdw9\") pod \"e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk\" (UID: \"ad23561f-a24f-4c13-9df7-0ecce2596034\") " pod="openshift-ingress/e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk"
Oct 30 21:09:23.254731 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:23.254616    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad23561f-a24f-4c13-9df7-0ecce2596034-credential-socket\") pod \"e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk\" (UID: \"ad23561f-a24f-4c13-9df7-0ecce2596034\") " pod="openshift-ingress/e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk"
Oct 30 21:09:23.254731 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:23.254646    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad23561f-a24f-4c13-9df7-0ecce2596034-workload-socket\") pod \"e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk\" (UID: \"ad23561f-a24f-4c13-9df7-0ecce2596034\") " pod="openshift-ingress/e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk"
Oct 30 21:09:23.254974 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:23.254763    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad23561f-a24f-4c13-9df7-0ecce2596034-istiod-ca-cert\") pod \"e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk\" (UID: \"ad23561f-a24f-4c13-9df7-0ecce2596034\") " pod="openshift-ingress/e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk"
Oct 30 21:09:23.254974 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:23.254892    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad23561f-a24f-4c13-9df7-0ecce2596034-istio-token\") pod \"e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk\" (UID: \"ad23561f-a24f-4c13-9df7-0ecce2596034\") " pod="openshift-ingress/e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk"
Oct 30 21:09:23.254974 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:23.254950    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad23561f-a24f-4c13-9df7-0ecce2596034-workload-certs\") pod \"e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk\" (UID: \"ad23561f-a24f-4c13-9df7-0ecce2596034\") " pod="openshift-ingress/e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk"
Oct 30 21:09:23.254974 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:23.254956    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad23561f-a24f-4c13-9df7-0ecce2596034-istio-data\") pod \"e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk\" (UID: \"ad23561f-a24f-4c13-9df7-0ecce2596034\") " pod="openshift-ingress/e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk"
Oct 30 21:09:23.255182 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:23.255050    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad23561f-a24f-4c13-9df7-0ecce2596034-credential-socket\") pod \"e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk\" (UID: \"ad23561f-a24f-4c13-9df7-0ecce2596034\") " pod="openshift-ingress/e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk"
Oct 30 21:09:23.255300 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:23.255282    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad23561f-a24f-4c13-9df7-0ecce2596034-workload-certs\") pod \"e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk\" (UID: \"ad23561f-a24f-4c13-9df7-0ecce2596034\") " pod="openshift-ingress/e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk"
Oct 30 21:09:23.255341 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:23.255318    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad23561f-a24f-4c13-9df7-0ecce2596034-workload-socket\") pod \"e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk\" (UID: \"ad23561f-a24f-4c13-9df7-0ecce2596034\") " pod="openshift-ingress/e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk"
Oct 30 21:09:23.255736 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:23.255718    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad23561f-a24f-4c13-9df7-0ecce2596034-istiod-ca-cert\") pod \"e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk\" (UID: \"ad23561f-a24f-4c13-9df7-0ecce2596034\") " pod="openshift-ingress/e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk"
Oct 30 21:09:23.257294 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:23.257269    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad23561f-a24f-4c13-9df7-0ecce2596034-istio-envoy\") pod \"e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk\" (UID: \"ad23561f-a24f-4c13-9df7-0ecce2596034\") " pod="openshift-ingress/e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk"
Oct 30 21:09:23.258335 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:23.258311    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad23561f-a24f-4c13-9df7-0ecce2596034-istio-podinfo\") pod \"e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk\" (UID: \"ad23561f-a24f-4c13-9df7-0ecce2596034\") " pod="openshift-ingress/e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk"
Oct 30 21:09:23.311593 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:23.311561    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad23561f-a24f-4c13-9df7-0ecce2596034-istio-token\") pod \"e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk\" (UID: \"ad23561f-a24f-4c13-9df7-0ecce2596034\") " pod="openshift-ingress/e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk"
Oct 30 21:09:23.311839 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:23.311778    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbdw9\" (UniqueName: \"kubernetes.io/projected/ad23561f-a24f-4c13-9df7-0ecce2596034-kube-api-access-vbdw9\") pod \"e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk\" (UID: \"ad23561f-a24f-4c13-9df7-0ecce2596034\") " pod="openshift-ingress/e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk"
Oct 30 21:09:23.390367 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:23.390287    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk"
Oct 30 21:09:23.823467 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:23.823431    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk"]
Oct 30 21:09:23.824420 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:09:23.824391    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad23561f_a24f_4c13_9df7_0ecce2596034.slice/crio-080cf27cf9517f4edf56f7c309334ca693f071836c64293a8161f556b1753c00 WatchSource:0}: Error finding container 080cf27cf9517f4edf56f7c309334ca693f071836c64293a8161f556b1753c00: Status 404 returned error can't find the container with id 080cf27cf9517f4edf56f7c309334ca693f071836c64293a8161f556b1753c00
Oct 30 21:09:23.827266 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:23.827231    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:09:23.827370 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:23.827312    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:09:23.827435 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:23.827385    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:09:23.996129 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:23.996090    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk" event={"ID":"ad23561f-a24f-4c13-9df7-0ecce2596034","Type":"ContainerStarted","Data":"4157e3bb57ce1a7d4978d8fa5903723c81ad1f43ca0a805cd809861f4e54edcc"}
Oct 30 21:09:23.996302 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:23.996184    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk" event={"ID":"ad23561f-a24f-4c13-9df7-0ecce2596034","Type":"ContainerStarted","Data":"080cf27cf9517f4edf56f7c309334ca693f071836c64293a8161f556b1753c00"}
Oct 30 21:09:24.117217 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:24.117073    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk" podStartSLOduration=2.117053132 podStartE2EDuration="2.117053132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:24.111401129 +0000 UTC m=+1847.176177528" watchObservedRunningTime="2026-10-30 21:09:24.117053132 +0000 UTC m=+1847.181829531"
Oct 30 21:09:24.403860 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:24.403349    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk"
Oct 30 21:09:24.403860 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:24.403537    5713 patch_prober.go:29] interesting pod/e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Oct 30 21:09:24.403860 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:24.403581    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk" podUID="ad23561f-a24f-4c13-9df7-0ecce2596034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Oct 30 21:09:24.472863 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:09:24.472783    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:09:25.395946 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:25.395919    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk"
Oct 30 21:09:26.013946 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:26.013912    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk"
Oct 30 21:09:26.014929 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:26.014912    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk"
Oct 30 21:09:29.959886 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:29.959853    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef7e2d-6784699ff9-mcvmc"
Oct 30 21:09:29.960424 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:29.959941    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-ef7e2d-77489c76c6-vf7cz"
Oct 30 21:09:35.032812 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:35.017845    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-677b45-955d68994-fdprd"]
Oct 30 21:09:35.037843 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:35.037813    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-677b45-955d68994-fdprd"
Oct 30 21:09:35.190403 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:35.190371    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-677b45-955d68994-fdprd"]
Oct 30 21:09:35.207984 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:35.205184    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-677b45\""
Oct 30 21:09:35.207984 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:35.205274    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-677b45-dockercfg-nmln7\""
Oct 30 21:09:35.211721 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:35.211382    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b15b706-c879-4ada-a2b0-852c21ff3c8b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-677b45-955d68994-fdprd\" (UID: \"0b15b706-c879-4ada-a2b0-852c21ff3c8b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-677b45-955d68994-fdprd"
Oct 30 21:09:35.211953 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:35.211913    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxzjw\" (UniqueName: \"kubernetes.io/projected/0b15b706-c879-4ada-a2b0-852c21ff3c8b-kube-api-access-kxzjw\") pod \"payload-processing-e2e-shared-b-w1-677b45-955d68994-fdprd\" (UID: \"0b15b706-c879-4ada-a2b0-852c21ff3c8b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-677b45-955d68994-fdprd"
Oct 30 21:09:35.326874 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:35.326722    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b15b706-c879-4ada-a2b0-852c21ff3c8b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-677b45-955d68994-fdprd\" (UID: \"0b15b706-c879-4ada-a2b0-852c21ff3c8b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-677b45-955d68994-fdprd"
Oct 30 21:09:35.327078 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:35.326955    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxzjw\" (UniqueName: \"kubernetes.io/projected/0b15b706-c879-4ada-a2b0-852c21ff3c8b-kube-api-access-kxzjw\") pod \"payload-processing-e2e-shared-b-w1-677b45-955d68994-fdprd\" (UID: \"0b15b706-c879-4ada-a2b0-852c21ff3c8b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-677b45-955d68994-fdprd"
Oct 30 21:09:35.362330 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:35.328031    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b15b706-c879-4ada-a2b0-852c21ff3c8b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-677b45-955d68994-fdprd\" (UID: \"0b15b706-c879-4ada-a2b0-852c21ff3c8b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-677b45-955d68994-fdprd"
Oct 30 21:09:35.463987 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:35.461517    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxzjw\" (UniqueName: \"kubernetes.io/projected/0b15b706-c879-4ada-a2b0-852c21ff3c8b-kube-api-access-kxzjw\") pod \"payload-processing-e2e-shared-b-w1-677b45-955d68994-fdprd\" (UID: \"0b15b706-c879-4ada-a2b0-852c21ff3c8b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-677b45-955d68994-fdprd"
Oct 30 21:09:35.527371 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:35.527330    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-677b45-955d68994-fdprd"
Oct 30 21:09:35.997808 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:35.993995    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-677b45-955d68994-fdprd"]
Oct 30 21:09:36.230734 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:36.230688    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-677b45-955d68994-fdprd" event={"ID":"0b15b706-c879-4ada-a2b0-852c21ff3c8b","Type":"ContainerStarted","Data":"d93b196730fd53491955c5258bcb9ed5d5a893db6fc0ae4c993a08081b728dea"}
Oct 30 21:09:36.231442 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:36.230831    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-677b45-955d68994-fdprd" event={"ID":"0b15b706-c879-4ada-a2b0-852c21ff3c8b","Type":"ContainerStarted","Data":"71d7130fae6d7c221b25ba0169da2e9db6153f6cbecd5c961304d681016e1bf4"}
Oct 30 21:09:36.231442 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:36.231071    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-677b45-955d68994-fdprd"
Oct 30 21:09:36.345869 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:36.345781    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-677b45-955d68994-fdprd" podStartSLOduration=2.345763401 podStartE2EDuration="2.345763401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:36.343592718 +0000 UTC m=+1859.408369142" watchObservedRunningTime="2026-10-30 21:09:36.345763401 +0000 UTC m=+1859.410539801"
Oct 30 21:09:38.511929 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:38.511890    5713 scope.go:122] "RemoveContainer" containerID="6ecb5620cfd99d8e90af5d7586a1de5de17fd2c7ea1f14f7a347f1c9e27ba3a0"
Oct 30 21:09:38.526450 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:38.526432    5713 scope.go:122] "RemoveContainer" containerID="eedb447238bd70fcb9505a3d2556fc0d5221fc0e7e006962e6b51e11637c657e"
Oct 30 21:09:38.536335 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:38.536310    5713 scope.go:122] "RemoveContainer" containerID="b981cd0d93974018b3b4f6f27068e7d9663fc47d94df51a35f1de1c233ede85b"
Oct 30 21:09:38.545890 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:38.545872    5713 scope.go:122] "RemoveContainer" containerID="9953ef86dc0084bbe4cdbfc8869b7630538de952f56d92d6d10a0f10d3095f5e"
Oct 30 21:09:41.302421 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:41.302389    5713 generic.go:356] "Generic (PLEG): container finished" podID="e3000e9c-41b7-4667-b9e1-c90ecf6c9b63" containerID="75d41fbab6583d1312c43e434c531e6b47564a0ab9ed3aacb7382a1705032798" exitCode=0
Oct 30 21:09:41.302884 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:41.302446    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5" event={"ID":"e3000e9c-41b7-4667-b9e1-c90ecf6c9b63","Type":"ContainerDied","Data":"75d41fbab6583d1312c43e434c531e6b47564a0ab9ed3aacb7382a1705032798"}
Oct 30 21:09:41.302954 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:41.302935    5713 scope.go:122] "RemoveContainer" containerID="75d41fbab6583d1312c43e434c531e6b47564a0ab9ed3aacb7382a1705032798"
Oct 30 21:09:41.304667 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:41.304638    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:09:41.304737 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:41.304714    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:09:41.304774 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:41.304745    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:09:41.413771 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:41.413725    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz"]
Oct 30 21:09:41.418309 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:41.417729    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz"
Oct 30 21:09:41.482765 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:41.482713    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a26a6cf1-a0b9-4470-93ea-557cae471765-credential-socket\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz\" (UID: \"a26a6cf1-a0b9-4470-93ea-557cae471765\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz"
Oct 30 21:09:41.482965 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:41.482831    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6tj2\" (UniqueName: \"kubernetes.io/projected/a26a6cf1-a0b9-4470-93ea-557cae471765-kube-api-access-l6tj2\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz\" (UID: \"a26a6cf1-a0b9-4470-93ea-557cae471765\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz"
Oct 30 21:09:41.482965 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:41.482869    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a26a6cf1-a0b9-4470-93ea-557cae471765-istio-token\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz\" (UID: \"a26a6cf1-a0b9-4470-93ea-557cae471765\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz"
Oct 30 21:09:41.482965 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:41.482903    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a26a6cf1-a0b9-4470-93ea-557cae471765-istio-envoy\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz\" (UID: \"a26a6cf1-a0b9-4470-93ea-557cae471765\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz"
Oct 30 21:09:41.482965 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:41.482933    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a26a6cf1-a0b9-4470-93ea-557cae471765-istio-podinfo\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz\" (UID: \"a26a6cf1-a0b9-4470-93ea-557cae471765\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz"
Oct 30 21:09:41.483237 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:41.483029    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a26a6cf1-a0b9-4470-93ea-557cae471765-istio-data\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz\" (UID: \"a26a6cf1-a0b9-4470-93ea-557cae471765\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz"
Oct 30 21:09:41.483237 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:41.483061    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a26a6cf1-a0b9-4470-93ea-557cae471765-workload-certs\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz\" (UID: \"a26a6cf1-a0b9-4470-93ea-557cae471765\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz"
Oct 30 21:09:41.483237 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:41.483213    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a26a6cf1-a0b9-4470-93ea-557cae471765-istiod-ca-cert\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz\" (UID: \"a26a6cf1-a0b9-4470-93ea-557cae471765\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz"
Oct 30 21:09:41.483403 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:41.483286    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a26a6cf1-a0b9-4470-93ea-557cae471765-workload-socket\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz\" (UID: \"a26a6cf1-a0b9-4470-93ea-557cae471765\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz"
Oct 30 21:09:41.559627 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:41.559542    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz"]
Oct 30 21:09:41.585289 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:41.585250    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a26a6cf1-a0b9-4470-93ea-557cae471765-istiod-ca-cert\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz\" (UID: \"a26a6cf1-a0b9-4470-93ea-557cae471765\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz"
Oct 30 21:09:41.585530 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:41.585355    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a26a6cf1-a0b9-4470-93ea-557cae471765-workload-socket\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz\" (UID: \"a26a6cf1-a0b9-4470-93ea-557cae471765\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz"
Oct 30 21:09:41.585839 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:41.585767    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a26a6cf1-a0b9-4470-93ea-557cae471765-workload-socket\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz\" (UID: \"a26a6cf1-a0b9-4470-93ea-557cae471765\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz"
Oct 30 21:09:41.585939 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:41.585924    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a26a6cf1-a0b9-4470-93ea-557cae471765-credential-socket\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz\" (UID: \"a26a6cf1-a0b9-4470-93ea-557cae471765\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz"
Oct 30 21:09:41.586023 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:41.586004    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6tj2\" (UniqueName: \"kubernetes.io/projected/a26a6cf1-a0b9-4470-93ea-557cae471765-kube-api-access-l6tj2\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz\" (UID: \"a26a6cf1-a0b9-4470-93ea-557cae471765\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz"
Oct 30 21:09:41.586089 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:41.586044    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a26a6cf1-a0b9-4470-93ea-557cae471765-istio-token\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz\" (UID: \"a26a6cf1-a0b9-4470-93ea-557cae471765\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz"
Oct 30 21:09:41.586089 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:41.586075    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a26a6cf1-a0b9-4470-93ea-557cae471765-istio-envoy\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz\" (UID: \"a26a6cf1-a0b9-4470-93ea-557cae471765\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz"
Oct 30 21:09:41.586208 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:41.586104    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a26a6cf1-a0b9-4470-93ea-557cae471765-istio-podinfo\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz\" (UID: \"a26a6cf1-a0b9-4470-93ea-557cae471765\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz"
Oct 30 21:09:41.586208 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:41.586113    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a26a6cf1-a0b9-4470-93ea-557cae471765-istiod-ca-cert\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz\" (UID: \"a26a6cf1-a0b9-4470-93ea-557cae471765\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz"
Oct 30 21:09:41.586315 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:41.586209    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a26a6cf1-a0b9-4470-93ea-557cae471765-istio-data\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz\" (UID: \"a26a6cf1-a0b9-4470-93ea-557cae471765\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz"
Oct 30 21:09:41.586315 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:41.586243    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a26a6cf1-a0b9-4470-93ea-557cae471765-workload-certs\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz\" (UID: \"a26a6cf1-a0b9-4470-93ea-557cae471765\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz"
Oct 30 21:09:41.586693 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:41.586666    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a26a6cf1-a0b9-4470-93ea-557cae471765-workload-certs\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz\" (UID: \"a26a6cf1-a0b9-4470-93ea-557cae471765\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz"
Oct 30 21:09:41.586842 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:41.586759    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a26a6cf1-a0b9-4470-93ea-557cae471765-istio-data\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz\" (UID: \"a26a6cf1-a0b9-4470-93ea-557cae471765\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz"
Oct 30 21:09:41.587117 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:41.587080    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a26a6cf1-a0b9-4470-93ea-557cae471765-credential-socket\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz\" (UID: \"a26a6cf1-a0b9-4470-93ea-557cae471765\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz"
Oct 30 21:09:41.589549 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:41.589521    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a26a6cf1-a0b9-4470-93ea-557cae471765-istio-envoy\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz\" (UID: \"a26a6cf1-a0b9-4470-93ea-557cae471765\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz"
Oct 30 21:09:41.589671 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:41.589546    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a26a6cf1-a0b9-4470-93ea-557cae471765-istio-podinfo\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz\" (UID: \"a26a6cf1-a0b9-4470-93ea-557cae471765\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz"
Oct 30 21:09:41.647262 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:41.647230    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6tj2\" (UniqueName: \"kubernetes.io/projected/a26a6cf1-a0b9-4470-93ea-557cae471765-kube-api-access-l6tj2\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz\" (UID: \"a26a6cf1-a0b9-4470-93ea-557cae471765\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz"
Oct 30 21:09:41.647440 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:41.647418    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a26a6cf1-a0b9-4470-93ea-557cae471765-istio-token\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz\" (UID: \"a26a6cf1-a0b9-4470-93ea-557cae471765\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz"
Oct 30 21:09:41.732707 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:41.732671    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz"
Oct 30 21:09:42.189833 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:42.188271    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz"]
Oct 30 21:09:42.212468 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:42.212422    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:09:42.212610 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:42.212574    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:09:42.212682 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:42.212667    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:09:42.312241 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:42.312201    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz" event={"ID":"a26a6cf1-a0b9-4470-93ea-557cae471765","Type":"ContainerStarted","Data":"8503397f8a4b7f19b6aebdbf1a4ce1477d04d95f031db242087d95b1a1834e01"}
Oct 30 21:09:42.312683 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:42.312308    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz" event={"ID":"a26a6cf1-a0b9-4470-93ea-557cae471765","Type":"ContainerStarted","Data":"0e8d8cae54e5943d5c8047f811d2b73a37c3bca9d6f3e5d67bb921712e3233a3"}
Oct 30 21:09:42.314736 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:42.314698    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5" event={"ID":"e3000e9c-41b7-4667-b9e1-c90ecf6c9b63","Type":"ContainerStarted","Data":"0adf7a2bfc102d7a99bc4e81c775046c82ee85f9a7463bbc28fb6db8fe9760f2"}
Oct 30 21:09:42.460213 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:42.460091    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz" podStartSLOduration=1.460064302 podStartE2EDuration="1.460064302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:42.441945904 +0000 UTC m=+1865.506722303" watchObservedRunningTime="2026-10-30 21:09:42.460064302 +0000 UTC m=+1865.524840700"
Oct 30 21:09:42.607412 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:42.607377    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5"
Oct 30 21:09:42.608932 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:42.608882    5713 patch_prober.go:29] interesting pod/e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Oct 30 21:09:42.609069 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:42.608933    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5" podUID="e3000e9c-41b7-4667-b9e1-c90ecf6c9b63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Oct 30 21:09:42.733604 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:42.733525    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz"
Oct 30 21:09:42.983600 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:42.983565    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5"
Oct 30 21:09:43.607923 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:43.607880    5713 patch_prober.go:29] interesting pod/e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Oct 30 21:09:43.608357 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:43.607944    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5" podUID="e3000e9c-41b7-4667-b9e1-c90ecf6c9b63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Oct 30 21:09:43.772814 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:43.771998    5713 patch_prober.go:29] interesting pod/e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Oct 30 21:09:43.772814 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:43.772081    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz" podUID="a26a6cf1-a0b9-4470-93ea-557cae471765" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Oct 30 21:09:44.607271 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:44.607237    5713 patch_prober.go:29] interesting pod/e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Oct 30 21:09:44.607433 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:44.607293    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5" podUID="e3000e9c-41b7-4667-b9e1-c90ecf6c9b63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Oct 30 21:09:44.733717 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:44.733685    5713 patch_prober.go:29] interesting pod/e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Oct 30 21:09:44.734289 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:44.733743    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz" podUID="a26a6cf1-a0b9-4470-93ea-557cae471765" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Oct 30 21:09:45.606857 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:45.606826    5713 patch_prober.go:29] interesting pod/e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Oct 30 21:09:45.607077 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:45.607055    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5" podUID="e3000e9c-41b7-4667-b9e1-c90ecf6c9b63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Oct 30 21:09:45.733095 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:45.733061    5713 patch_prober.go:29] interesting pod/e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Oct 30 21:09:45.733257 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:45.733119    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz" podUID="a26a6cf1-a0b9-4470-93ea-557cae471765" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Oct 30 21:09:46.607376 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:46.607337    5713 patch_prober.go:29] interesting pod/e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Oct 30 21:09:46.607908 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:46.607402    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5" podUID="e3000e9c-41b7-4667-b9e1-c90ecf6c9b63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Oct 30 21:09:46.754815 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:46.733453    5713 patch_prober.go:29] interesting pod/e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Oct 30 21:09:46.754815 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:46.733514    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz" podUID="a26a6cf1-a0b9-4470-93ea-557cae471765" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Oct 30 21:09:47.315816 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:47.315011    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-677b45-955d68994-fdprd"
Oct 30 21:09:47.520847 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:47.503083    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f3b172-5c85644f98-5wgnl"]
Oct 30 21:09:47.612984 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:47.612234    5713 patch_prober.go:29] interesting pod/e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Oct 30 21:09:47.612984 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:47.612311    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5" podUID="e3000e9c-41b7-4667-b9e1-c90ecf6c9b63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Oct 30 21:09:47.640863 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:47.624313    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f3b172-5c85644f98-5wgnl"
Oct 30 21:09:47.729103 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:47.716062    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f3b172-5c85644f98-5wgnl"]
Oct 30 21:09:47.754024 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:47.737433    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-f3b172\""
Oct 30 21:09:47.754299 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:47.745312    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-f3b172-dockercfg-8qvkr\""
Oct 30 21:09:47.754428 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:47.745618    5713 patch_prober.go:29] interesting pod/e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Oct 30 21:09:47.815842 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:47.754560    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz" podUID="a26a6cf1-a0b9-4470-93ea-557cae471765" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Oct 30 21:09:47.824289 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:47.816990    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9bb0c3d-cf42-4f2b-9dde-b16b890cddab-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f3b172-5c85644f98-5wgnl\" (UID: \"c9bb0c3d-cf42-4f2b-9dde-b16b890cddab\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f3b172-5c85644f98-5wgnl"
Oct 30 21:09:47.824662 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:47.824639    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4ctd\" (UniqueName: \"kubernetes.io/projected/c9bb0c3d-cf42-4f2b-9dde-b16b890cddab-kube-api-access-s4ctd\") pod \"payload-processing-e2e-inf-a-f3b172-5c85644f98-5wgnl\" (UID: \"c9bb0c3d-cf42-4f2b-9dde-b16b890cddab\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f3b172-5c85644f98-5wgnl"
Oct 30 21:09:47.913986 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:47.913119    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f3b172-79c58fc975-d9k5m"]
Oct 30 21:09:47.920240 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:47.920214    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f3b172-79c58fc975-d9k5m"]
Oct 30 21:09:47.921009 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:47.920334    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3b172-79c58fc975-d9k5m"
Oct 30 21:09:47.926403 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:47.926361    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4ctd\" (UniqueName: \"kubernetes.io/projected/c9bb0c3d-cf42-4f2b-9dde-b16b890cddab-kube-api-access-s4ctd\") pod \"payload-processing-e2e-inf-a-f3b172-5c85644f98-5wgnl\" (UID: \"c9bb0c3d-cf42-4f2b-9dde-b16b890cddab\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f3b172-5c85644f98-5wgnl"
Oct 30 21:09:47.927169 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:47.927140    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9bb0c3d-cf42-4f2b-9dde-b16b890cddab-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f3b172-5c85644f98-5wgnl\" (UID: \"c9bb0c3d-cf42-4f2b-9dde-b16b890cddab\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f3b172-5c85644f98-5wgnl"
Oct 30 21:09:47.928243 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:47.928215    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9bb0c3d-cf42-4f2b-9dde-b16b890cddab-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f3b172-5c85644f98-5wgnl\" (UID: \"c9bb0c3d-cf42-4f2b-9dde-b16b890cddab\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f3b172-5c85644f98-5wgnl"
Oct 30 21:09:47.985829 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:47.985230    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4ctd\" (UniqueName: \"kubernetes.io/projected/c9bb0c3d-cf42-4f2b-9dde-b16b890cddab-kube-api-access-s4ctd\") pod \"payload-processing-e2e-inf-a-f3b172-5c85644f98-5wgnl\" (UID: \"c9bb0c3d-cf42-4f2b-9dde-b16b890cddab\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f3b172-5c85644f98-5wgnl"
Oct 30 21:09:48.028744 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:48.028705    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n98fn\" (UniqueName: \"kubernetes.io/projected/6ed86aa9-530c-4122-99c9-23c7f28c2b08-kube-api-access-n98fn\") pod \"payload-pre-processing-e2e-inf-a-f3b172-79c58fc975-d9k5m\" (UID: \"6ed86aa9-530c-4122-99c9-23c7f28c2b08\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3b172-79c58fc975-d9k5m"
Oct 30 21:09:48.029126 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:48.029098    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ed86aa9-530c-4122-99c9-23c7f28c2b08-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f3b172-79c58fc975-d9k5m\" (UID: \"6ed86aa9-530c-4122-99c9-23c7f28c2b08\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3b172-79c58fc975-d9k5m"
Oct 30 21:09:48.131843 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:48.131770    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n98fn\" (UniqueName: \"kubernetes.io/projected/6ed86aa9-530c-4122-99c9-23c7f28c2b08-kube-api-access-n98fn\") pod \"payload-pre-processing-e2e-inf-a-f3b172-79c58fc975-d9k5m\" (UID: \"6ed86aa9-530c-4122-99c9-23c7f28c2b08\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3b172-79c58fc975-d9k5m"
Oct 30 21:09:48.132024 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:48.131885    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ed86aa9-530c-4122-99c9-23c7f28c2b08-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f3b172-79c58fc975-d9k5m\" (UID: \"6ed86aa9-530c-4122-99c9-23c7f28c2b08\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3b172-79c58fc975-d9k5m"
Oct 30 21:09:48.132777 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:48.132746    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ed86aa9-530c-4122-99c9-23c7f28c2b08-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f3b172-79c58fc975-d9k5m\" (UID: \"6ed86aa9-530c-4122-99c9-23c7f28c2b08\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3b172-79c58fc975-d9k5m"
Oct 30 21:09:48.189327 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:48.189292    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n98fn\" (UniqueName: \"kubernetes.io/projected/6ed86aa9-530c-4122-99c9-23c7f28c2b08-kube-api-access-n98fn\") pod \"payload-pre-processing-e2e-inf-a-f3b172-79c58fc975-d9k5m\" (UID: \"6ed86aa9-530c-4122-99c9-23c7f28c2b08\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3b172-79c58fc975-d9k5m"
Oct 30 21:09:48.204350 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:48.204319    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f3b172-5c85644f98-5wgnl"
Oct 30 21:09:48.250399 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:48.250081    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3b172-79c58fc975-d9k5m"
Oct 30 21:09:48.612620 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:48.612532    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5"
Oct 30 21:09:48.613772 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:48.613751    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5"
Oct 30 21:09:48.627405 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:09:48.627377    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9bb0c3d_cf42_4f2b_9dde_b16b890cddab.slice/crio-df84c5f8038486244f71377a8bdf694657df828da733a35e971b3defe60cc4cc WatchSource:0}: Error finding container df84c5f8038486244f71377a8bdf694657df828da733a35e971b3defe60cc4cc: Status 404 returned error can't find the container with id df84c5f8038486244f71377a8bdf694657df828da733a35e971b3defe60cc4cc
Oct 30 21:09:48.627530 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:48.627507    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f3b172-5c85644f98-5wgnl"]
Oct 30 21:09:48.727466 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:48.727435    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f3b172-79c58fc975-d9k5m"]
Oct 30 21:09:48.729392 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:09:48.729361    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ed86aa9_530c_4122_99c9_23c7f28c2b08.slice/crio-a6ef9d0d1bd8c96c0d0f54a78b7ef31d8852b0427c57dba3c0eec914bd93372e WatchSource:0}: Error finding container a6ef9d0d1bd8c96c0d0f54a78b7ef31d8852b0427c57dba3c0eec914bd93372e: Status 404 returned error can't find the container with id a6ef9d0d1bd8c96c0d0f54a78b7ef31d8852b0427c57dba3c0eec914bd93372e
Oct 30 21:09:48.733565 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:48.733217    5713 patch_prober.go:29] interesting pod/e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Oct 30 21:09:48.733565 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:48.733266    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz" podUID="a26a6cf1-a0b9-4470-93ea-557cae471765" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Oct 30 21:09:49.593911 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:49.593863    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f3b172-5c85644f98-5wgnl" event={"ID":"c9bb0c3d-cf42-4f2b-9dde-b16b890cddab","Type":"ContainerStarted","Data":"a7dcbc295d693e3eff75509312e957e6f4e328b17bcd9550744516ddb9611e12"}
Oct 30 21:09:49.594261 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:49.594229    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f3b172-5c85644f98-5wgnl" event={"ID":"c9bb0c3d-cf42-4f2b-9dde-b16b890cddab","Type":"ContainerStarted","Data":"df84c5f8038486244f71377a8bdf694657df828da733a35e971b3defe60cc4cc"}
Oct 30 21:09:49.595097 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:49.595067    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-f3b172-5c85644f98-5wgnl"
Oct 30 21:09:49.597301 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:49.597268    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3b172-79c58fc975-d9k5m" event={"ID":"6ed86aa9-530c-4122-99c9-23c7f28c2b08","Type":"ContainerStarted","Data":"6a355a95f6e2ff2347dc8601c9a28335b87bf791563f28d7d32a49f3f6353713"}
Oct 30 21:09:49.597429 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:49.597328    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3b172-79c58fc975-d9k5m" event={"ID":"6ed86aa9-530c-4122-99c9-23c7f28c2b08","Type":"ContainerStarted","Data":"a6ef9d0d1bd8c96c0d0f54a78b7ef31d8852b0427c57dba3c0eec914bd93372e"}
Oct 30 21:09:49.597593 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:49.597578    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3b172-79c58fc975-d9k5m"
Oct 30 21:09:49.711063 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:49.711005    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-f3b172-5c85644f98-5wgnl" podStartSLOduration=2.710990419 podStartE2EDuration="2.710990419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:49.708704815 +0000 UTC m=+1872.773481214" watchObservedRunningTime="2026-10-30 21:09:49.710990419 +0000 UTC m=+1872.775766845"
Oct 30 21:09:49.739126 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:49.739094    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz"
Oct 30 21:09:49.739514 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:49.739492    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz"
Oct 30 21:09:49.740216 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:49.740196    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz"
Oct 30 21:09:49.837763 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:09:49.837693    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3b172-79c58fc975-d9k5m" podStartSLOduration=2.837674893 podStartE2EDuration="2.837674893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:49.830513643 +0000 UTC m=+1872.895290038" watchObservedRunningTime="2026-10-30 21:09:49.837674893 +0000 UTC m=+1872.902451291"
Oct 30 21:09:54.460229 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:09:54.460187    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:10:00.622677 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:00.622645    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3b172-79c58fc975-d9k5m"
Oct 30 21:10:00.623214 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:00.622715    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-f3b172-5c85644f98-5wgnl"
Oct 30 21:10:24.461686 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:10:24.461644    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:10:26.733625 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:26.733592    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd"]
Oct 30 21:10:26.739639 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:26.739620    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd"
Oct 30 21:10:26.785525 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:26.785488    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49e80bbd-dd2a-42f8-bdaf-e83abde1e625-workload-socket\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd\" (UID: \"49e80bbd-dd2a-42f8-bdaf-e83abde1e625\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd"
Oct 30 21:10:26.785696 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:26.785638    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbg7k\" (UniqueName: \"kubernetes.io/projected/49e80bbd-dd2a-42f8-bdaf-e83abde1e625-kube-api-access-tbg7k\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd\" (UID: \"49e80bbd-dd2a-42f8-bdaf-e83abde1e625\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd"
Oct 30 21:10:26.785741 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:26.785700    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49e80bbd-dd2a-42f8-bdaf-e83abde1e625-istio-envoy\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd\" (UID: \"49e80bbd-dd2a-42f8-bdaf-e83abde1e625\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd"
Oct 30 21:10:26.785773 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:26.785755    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49e80bbd-dd2a-42f8-bdaf-e83abde1e625-istio-data\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd\" (UID: \"49e80bbd-dd2a-42f8-bdaf-e83abde1e625\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd"
Oct 30 21:10:26.785834 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:26.785778    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49e80bbd-dd2a-42f8-bdaf-e83abde1e625-credential-socket\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd\" (UID: \"49e80bbd-dd2a-42f8-bdaf-e83abde1e625\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd"
Oct 30 21:10:26.785949 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:26.785927    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49e80bbd-dd2a-42f8-bdaf-e83abde1e625-workload-certs\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd\" (UID: \"49e80bbd-dd2a-42f8-bdaf-e83abde1e625\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd"
Oct 30 21:10:26.786018 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:26.785963    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49e80bbd-dd2a-42f8-bdaf-e83abde1e625-istio-podinfo\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd\" (UID: \"49e80bbd-dd2a-42f8-bdaf-e83abde1e625\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd"
Oct 30 21:10:26.786018 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:26.785996    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49e80bbd-dd2a-42f8-bdaf-e83abde1e625-istio-token\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd\" (UID: \"49e80bbd-dd2a-42f8-bdaf-e83abde1e625\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd"
Oct 30 21:10:26.786164 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:26.786148    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49e80bbd-dd2a-42f8-bdaf-e83abde1e625-istiod-ca-cert\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd\" (UID: \"49e80bbd-dd2a-42f8-bdaf-e83abde1e625\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd"
Oct 30 21:10:26.887870 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:26.887828    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49e80bbd-dd2a-42f8-bdaf-e83abde1e625-istio-data\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd\" (UID: \"49e80bbd-dd2a-42f8-bdaf-e83abde1e625\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd"
Oct 30 21:10:26.887870 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:26.887872    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49e80bbd-dd2a-42f8-bdaf-e83abde1e625-credential-socket\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd\" (UID: \"49e80bbd-dd2a-42f8-bdaf-e83abde1e625\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd"
Oct 30 21:10:26.888143 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:26.887903    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49e80bbd-dd2a-42f8-bdaf-e83abde1e625-workload-certs\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd\" (UID: \"49e80bbd-dd2a-42f8-bdaf-e83abde1e625\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd"
Oct 30 21:10:26.888143 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:26.887919    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49e80bbd-dd2a-42f8-bdaf-e83abde1e625-istio-podinfo\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd\" (UID: \"49e80bbd-dd2a-42f8-bdaf-e83abde1e625\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd"
Oct 30 21:10:26.888143 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:26.887941    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49e80bbd-dd2a-42f8-bdaf-e83abde1e625-istio-token\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd\" (UID: \"49e80bbd-dd2a-42f8-bdaf-e83abde1e625\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd"
Oct 30 21:10:26.888143 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:26.888037    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49e80bbd-dd2a-42f8-bdaf-e83abde1e625-istiod-ca-cert\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd\" (UID: \"49e80bbd-dd2a-42f8-bdaf-e83abde1e625\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd"
Oct 30 21:10:26.888368 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:26.888223    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49e80bbd-dd2a-42f8-bdaf-e83abde1e625-workload-socket\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd\" (UID: \"49e80bbd-dd2a-42f8-bdaf-e83abde1e625\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd"
Oct 30 21:10:26.888368 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:26.888322    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49e80bbd-dd2a-42f8-bdaf-e83abde1e625-istio-data\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd\" (UID: \"49e80bbd-dd2a-42f8-bdaf-e83abde1e625\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd"
Oct 30 21:10:26.888368 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:26.888348    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49e80bbd-dd2a-42f8-bdaf-e83abde1e625-credential-socket\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd\" (UID: \"49e80bbd-dd2a-42f8-bdaf-e83abde1e625\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd"
Oct 30 21:10:26.888524 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:26.888389    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbg7k\" (UniqueName: \"kubernetes.io/projected/49e80bbd-dd2a-42f8-bdaf-e83abde1e625-kube-api-access-tbg7k\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd\" (UID: \"49e80bbd-dd2a-42f8-bdaf-e83abde1e625\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd"
Oct 30 21:10:26.888524 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:26.888468    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49e80bbd-dd2a-42f8-bdaf-e83abde1e625-istio-envoy\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd\" (UID: \"49e80bbd-dd2a-42f8-bdaf-e83abde1e625\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd"
Oct 30 21:10:26.888641 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:26.888622    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49e80bbd-dd2a-42f8-bdaf-e83abde1e625-workload-socket\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd\" (UID: \"49e80bbd-dd2a-42f8-bdaf-e83abde1e625\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd"
Oct 30 21:10:26.888999 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:26.888973    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49e80bbd-dd2a-42f8-bdaf-e83abde1e625-workload-certs\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd\" (UID: \"49e80bbd-dd2a-42f8-bdaf-e83abde1e625\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd"
Oct 30 21:10:26.889129 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:26.889027    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49e80bbd-dd2a-42f8-bdaf-e83abde1e625-istiod-ca-cert\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd\" (UID: \"49e80bbd-dd2a-42f8-bdaf-e83abde1e625\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd"
Oct 30 21:10:26.891414 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:26.891381    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49e80bbd-dd2a-42f8-bdaf-e83abde1e625-istio-podinfo\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd\" (UID: \"49e80bbd-dd2a-42f8-bdaf-e83abde1e625\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd"
Oct 30 21:10:26.891414 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:26.891408    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49e80bbd-dd2a-42f8-bdaf-e83abde1e625-istio-envoy\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd\" (UID: \"49e80bbd-dd2a-42f8-bdaf-e83abde1e625\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd"
Oct 30 21:10:26.943243 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:26.943209    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd"]
Oct 30 21:10:26.946101 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:26.946069    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49e80bbd-dd2a-42f8-bdaf-e83abde1e625-istio-token\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd\" (UID: \"49e80bbd-dd2a-42f8-bdaf-e83abde1e625\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd"
Oct 30 21:10:26.961708 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:26.961676    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbg7k\" (UniqueName: \"kubernetes.io/projected/49e80bbd-dd2a-42f8-bdaf-e83abde1e625-kube-api-access-tbg7k\") pod \"e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd\" (UID: \"49e80bbd-dd2a-42f8-bdaf-e83abde1e625\") " pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd"
Oct 30 21:10:27.055643 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:27.055561    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd"
Oct 30 21:10:27.533869 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:27.533833    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd"]
Oct 30 21:10:27.534328 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:10:27.534295    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49e80bbd_dd2a_42f8_bdaf_e83abde1e625.slice/crio-83b34a4d32906d12bc4cd69129fc8fcf0ba8c69658d1035e9c7e57694b554b26 WatchSource:0}: Error finding container 83b34a4d32906d12bc4cd69129fc8fcf0ba8c69658d1035e9c7e57694b554b26: Status 404 returned error can't find the container with id 83b34a4d32906d12bc4cd69129fc8fcf0ba8c69658d1035e9c7e57694b554b26
Oct 30 21:10:27.537351 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:27.537318    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:10:27.537453 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:27.537388    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:10:27.537453 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:27.537416    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:10:27.924584 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:27.924487    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd" event={"ID":"49e80bbd-dd2a-42f8-bdaf-e83abde1e625","Type":"ContainerStarted","Data":"927cfc6c9e6ca68d76ac93f90f1ee49f2d654dd2f1445aa215b30ca43c515c7a"}
Oct 30 21:10:27.924584 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:27.924564    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd" event={"ID":"49e80bbd-dd2a-42f8-bdaf-e83abde1e625","Type":"ContainerStarted","Data":"83b34a4d32906d12bc4cd69129fc8fcf0ba8c69658d1035e9c7e57694b554b26"}
Oct 30 21:10:28.048004 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:28.047938    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd" podStartSLOduration=2.047920103 podStartE2EDuration="2.047920103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:10:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:10:28.040904365 +0000 UTC m=+1911.105680766" watchObservedRunningTime="2026-10-30 21:10:28.047920103 +0000 UTC m=+1911.112696508"
Oct 30 21:10:28.055826 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:28.055761    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd"
Oct 30 21:10:28.057331 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:28.057301    5713 patch_prober.go:29] interesting pod/e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Oct 30 21:10:28.057480 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:28.057356    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd" podUID="49e80bbd-dd2a-42f8-bdaf-e83abde1e625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Oct 30 21:10:29.062041 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:29.062003    5713 patch_prober.go:29] interesting pod/e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Oct 30 21:10:29.062570 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:29.062068    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd" podUID="49e80bbd-dd2a-42f8-bdaf-e83abde1e625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Oct 30 21:10:30.059816 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:30.059542    5713 patch_prober.go:29] interesting pod/e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Oct 30 21:10:30.059816 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:30.059616    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd" podUID="49e80bbd-dd2a-42f8-bdaf-e83abde1e625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Oct 30 21:10:31.060304 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:31.060267    5713 patch_prober.go:29] interesting pod/e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Oct 30 21:10:31.060881 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:31.060322    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd" podUID="49e80bbd-dd2a-42f8-bdaf-e83abde1e625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Oct 30 21:10:32.059203 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:32.059167    5713 patch_prober.go:29] interesting pod/e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Oct 30 21:10:32.059411 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:32.059312    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd" podUID="49e80bbd-dd2a-42f8-bdaf-e83abde1e625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Oct 30 21:10:33.060261 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:33.060222    5713 patch_prober.go:29] interesting pod/e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Oct 30 21:10:33.060762 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:33.060294    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd" podUID="49e80bbd-dd2a-42f8-bdaf-e83abde1e625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Oct 30 21:10:34.170556 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:34.170523    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd"
Oct 30 21:10:34.171547 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:34.171521    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd"
Oct 30 21:10:34.173729 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:34.173710    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd"
Oct 30 21:10:35.769269 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:35.769229    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk"]
Oct 30 21:10:35.773827 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:35.773766    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk"
Oct 30 21:10:35.826845 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:35.826812    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-cb6404-openshift-default-dockercfg-qw5rv\""
Oct 30 21:10:35.880739 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:35.880705    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk"]
Oct 30 21:10:35.888060 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:35.888032    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/237d222b-50be-4b97-b53d-49187434c09b-workload-certs\") pod \"e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk\" (UID: \"237d222b-50be-4b97-b53d-49187434c09b\") " pod="openshift-ingress/e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk"
Oct 30 21:10:35.888206 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:35.888183    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/237d222b-50be-4b97-b53d-49187434c09b-istio-envoy\") pod \"e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk\" (UID: \"237d222b-50be-4b97-b53d-49187434c09b\") " pod="openshift-ingress/e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk"
Oct 30 21:10:35.888281 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:35.888262    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/237d222b-50be-4b97-b53d-49187434c09b-istiod-ca-cert\") pod \"e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk\" (UID: \"237d222b-50be-4b97-b53d-49187434c09b\") " pod="openshift-ingress/e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk"
Oct 30 21:10:35.888338 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:35.888325    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/237d222b-50be-4b97-b53d-49187434c09b-istio-token\") pod \"e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk\" (UID: \"237d222b-50be-4b97-b53d-49187434c09b\") " pod="openshift-ingress/e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk"
Oct 30 21:10:35.888374 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:35.888356    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/237d222b-50be-4b97-b53d-49187434c09b-istio-data\") pod \"e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk\" (UID: \"237d222b-50be-4b97-b53d-49187434c09b\") " pod="openshift-ingress/e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk"
Oct 30 21:10:35.888442 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:35.888428    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/237d222b-50be-4b97-b53d-49187434c09b-istio-podinfo\") pod \"e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk\" (UID: \"237d222b-50be-4b97-b53d-49187434c09b\") " pod="openshift-ingress/e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk"
Oct 30 21:10:35.888480 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:35.888465    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp6ng\" (UniqueName: \"kubernetes.io/projected/237d222b-50be-4b97-b53d-49187434c09b-kube-api-access-zp6ng\") pod \"e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk\" (UID: \"237d222b-50be-4b97-b53d-49187434c09b\") " pod="openshift-ingress/e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk"
Oct 30 21:10:35.888584 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:35.888570    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/237d222b-50be-4b97-b53d-49187434c09b-credential-socket\") pod \"e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk\" (UID: \"237d222b-50be-4b97-b53d-49187434c09b\") " pod="openshift-ingress/e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk"
Oct 30 21:10:35.888623 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:35.888615    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/237d222b-50be-4b97-b53d-49187434c09b-workload-socket\") pod \"e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk\" (UID: \"237d222b-50be-4b97-b53d-49187434c09b\") " pod="openshift-ingress/e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk"
Oct 30 21:10:35.991107 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:35.991067    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/237d222b-50be-4b97-b53d-49187434c09b-credential-socket\") pod \"e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk\" (UID: \"237d222b-50be-4b97-b53d-49187434c09b\") " pod="openshift-ingress/e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk"
Oct 30 21:10:35.991296 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:35.991160    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/237d222b-50be-4b97-b53d-49187434c09b-workload-socket\") pod \"e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk\" (UID: \"237d222b-50be-4b97-b53d-49187434c09b\") " pod="openshift-ingress/e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk"
Oct 30 21:10:35.991347 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:35.991310    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/237d222b-50be-4b97-b53d-49187434c09b-workload-certs\") pod \"e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk\" (UID: \"237d222b-50be-4b97-b53d-49187434c09b\") " pod="openshift-ingress/e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk"
Oct 30 21:10:35.991462 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:35.991441    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/237d222b-50be-4b97-b53d-49187434c09b-istio-envoy\") pod \"e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk\" (UID: \"237d222b-50be-4b97-b53d-49187434c09b\") " pod="openshift-ingress/e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk"
Oct 30 21:10:35.991552 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:35.991484    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/237d222b-50be-4b97-b53d-49187434c09b-istiod-ca-cert\") pod \"e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk\" (UID: \"237d222b-50be-4b97-b53d-49187434c09b\") " pod="openshift-ingress/e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk"
Oct 30 21:10:35.991622 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:35.991571    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/237d222b-50be-4b97-b53d-49187434c09b-credential-socket\") pod \"e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk\" (UID: \"237d222b-50be-4b97-b53d-49187434c09b\") " pod="openshift-ingress/e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk"
Oct 30 21:10:35.991622 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:35.991580    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/237d222b-50be-4b97-b53d-49187434c09b-istio-token\") pod \"e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk\" (UID: \"237d222b-50be-4b97-b53d-49187434c09b\") " pod="openshift-ingress/e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk"
Oct 30 21:10:35.991743 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:35.991631    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/237d222b-50be-4b97-b53d-49187434c09b-istio-data\") pod \"e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk\" (UID: \"237d222b-50be-4b97-b53d-49187434c09b\") " pod="openshift-ingress/e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk"
Oct 30 21:10:35.991743 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:35.991632    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/237d222b-50be-4b97-b53d-49187434c09b-workload-socket\") pod \"e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk\" (UID: \"237d222b-50be-4b97-b53d-49187434c09b\") " pod="openshift-ingress/e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk"
Oct 30 21:10:35.991743 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:35.991676    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/237d222b-50be-4b97-b53d-49187434c09b-istio-podinfo\") pod \"e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk\" (UID: \"237d222b-50be-4b97-b53d-49187434c09b\") " pod="openshift-ingress/e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk"
Oct 30 21:10:35.991743 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:35.991682    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/237d222b-50be-4b97-b53d-49187434c09b-workload-certs\") pod \"e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk\" (UID: \"237d222b-50be-4b97-b53d-49187434c09b\") " pod="openshift-ingress/e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk"
Oct 30 21:10:35.991743 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:35.991723    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp6ng\" (UniqueName: \"kubernetes.io/projected/237d222b-50be-4b97-b53d-49187434c09b-kube-api-access-zp6ng\") pod \"e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk\" (UID: \"237d222b-50be-4b97-b53d-49187434c09b\") " pod="openshift-ingress/e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk"
Oct 30 21:10:35.992290 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:35.991916    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/237d222b-50be-4b97-b53d-49187434c09b-istio-data\") pod \"e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk\" (UID: \"237d222b-50be-4b97-b53d-49187434c09b\") " pod="openshift-ingress/e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk"
Oct 30 21:10:35.992425 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:35.992404    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/237d222b-50be-4b97-b53d-49187434c09b-istiod-ca-cert\") pod \"e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk\" (UID: \"237d222b-50be-4b97-b53d-49187434c09b\") " pod="openshift-ingress/e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk"
Oct 30 21:10:35.994351 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:35.994323    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/237d222b-50be-4b97-b53d-49187434c09b-istio-envoy\") pod \"e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk\" (UID: \"237d222b-50be-4b97-b53d-49187434c09b\") " pod="openshift-ingress/e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk"
Oct 30 21:10:35.998187 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:35.998165    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/237d222b-50be-4b97-b53d-49187434c09b-istio-podinfo\") pod \"e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk\" (UID: \"237d222b-50be-4b97-b53d-49187434c09b\") " pod="openshift-ingress/e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk"
Oct 30 21:10:36.050116 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:36.050041    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp6ng\" (UniqueName: \"kubernetes.io/projected/237d222b-50be-4b97-b53d-49187434c09b-kube-api-access-zp6ng\") pod \"e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk\" (UID: \"237d222b-50be-4b97-b53d-49187434c09b\") " pod="openshift-ingress/e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk"
Oct 30 21:10:36.051084 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:36.051066    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/237d222b-50be-4b97-b53d-49187434c09b-istio-token\") pod \"e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk\" (UID: \"237d222b-50be-4b97-b53d-49187434c09b\") " pod="openshift-ingress/e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk"
Oct 30 21:10:36.089682 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:36.089652    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk"
Oct 30 21:10:36.529484 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:36.529454    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk"]
Oct 30 21:10:36.529618 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:10:36.529508    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod237d222b_50be_4b97_b53d_49187434c09b.slice/crio-e7c993d7d4761ad0a18fcf483bedb6a62ad9701ddc15eaa9afed646bbe024c72 WatchSource:0}: Error finding container e7c993d7d4761ad0a18fcf483bedb6a62ad9701ddc15eaa9afed646bbe024c72: Status 404 returned error can't find the container with id e7c993d7d4761ad0a18fcf483bedb6a62ad9701ddc15eaa9afed646bbe024c72
Oct 30 21:10:36.532402 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:36.532345    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:10:36.532499 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:36.532436    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:10:36.532499 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:36.532466    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:10:37.039710 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:37.039665    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk" event={"ID":"237d222b-50be-4b97-b53d-49187434c09b","Type":"ContainerStarted","Data":"05163563985b8d23bde1a51105a5b2d6ba5a9dd9196c54c6ca0a059993cd8339"}
Oct 30 21:10:37.040153 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:37.039731    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk" event={"ID":"237d222b-50be-4b97-b53d-49187434c09b","Type":"ContainerStarted","Data":"e7c993d7d4761ad0a18fcf483bedb6a62ad9701ddc15eaa9afed646bbe024c72"}
Oct 30 21:10:37.089859 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:37.089826    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk"
Oct 30 21:10:37.095733 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:37.095706    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk"
Oct 30 21:10:37.162995 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:37.162929    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk" podStartSLOduration=2.162911151 podStartE2EDuration="2.162911151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:10:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:10:37.157034165 +0000 UTC m=+1920.221810564" watchObservedRunningTime="2026-10-30 21:10:37.162911151 +0000 UTC m=+1920.227687549"
Oct 30 21:10:38.060328 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:38.060291    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk"
Oct 30 21:10:38.061263 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:38.061240    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk"
Oct 30 21:10:38.598097 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:10:38.598066    5713 scope.go:122] "RemoveContainer" containerID="240ed7d821f503693218f530d2cdfe559f91b26fb3ffbe8a59394e387156a040"
Oct 30 21:10:54.460905 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:10:54.460854    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:11:08.983605 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:08.983569    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf"]
Oct 30 21:11:08.989097 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:08.989075    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf"
Oct 30 21:11:09.043467 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:09.043434    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-7eaf74-openshift-default-dockercfg-n66xg\""
Oct 30 21:11:09.099644 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:09.099603    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf"]
Oct 30 21:11:09.112264 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:09.112230    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c59144f6-c298-4a9d-a3c3-7a2e81bb8967-istiod-ca-cert\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf\" (UID: \"c59144f6-c298-4a9d-a3c3-7a2e81bb8967\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf"
Oct 30 21:11:09.112453 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:09.112299    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c59144f6-c298-4a9d-a3c3-7a2e81bb8967-istio-podinfo\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf\" (UID: \"c59144f6-c298-4a9d-a3c3-7a2e81bb8967\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf"
Oct 30 21:11:09.112453 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:09.112396    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c59144f6-c298-4a9d-a3c3-7a2e81bb8967-istio-data\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf\" (UID: \"c59144f6-c298-4a9d-a3c3-7a2e81bb8967\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf"
Oct 30 21:11:09.112453 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:09.112424    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c59144f6-c298-4a9d-a3c3-7a2e81bb8967-credential-socket\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf\" (UID: \"c59144f6-c298-4a9d-a3c3-7a2e81bb8967\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf"
Oct 30 21:11:09.112631 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:09.112610    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c59144f6-c298-4a9d-a3c3-7a2e81bb8967-workload-certs\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf\" (UID: \"c59144f6-c298-4a9d-a3c3-7a2e81bb8967\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf"
Oct 30 21:11:09.112709 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:09.112675    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c59144f6-c298-4a9d-a3c3-7a2e81bb8967-workload-socket\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf\" (UID: \"c59144f6-c298-4a9d-a3c3-7a2e81bb8967\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf"
Oct 30 21:11:09.112864 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:09.112724    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c59144f6-c298-4a9d-a3c3-7a2e81bb8967-istio-envoy\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf\" (UID: \"c59144f6-c298-4a9d-a3c3-7a2e81bb8967\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf"
Oct 30 21:11:09.112911 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:09.112874    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v5vv\" (UniqueName: \"kubernetes.io/projected/c59144f6-c298-4a9d-a3c3-7a2e81bb8967-kube-api-access-7v5vv\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf\" (UID: \"c59144f6-c298-4a9d-a3c3-7a2e81bb8967\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf"
Oct 30 21:11:09.112953 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:09.112924    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c59144f6-c298-4a9d-a3c3-7a2e81bb8967-istio-token\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf\" (UID: \"c59144f6-c298-4a9d-a3c3-7a2e81bb8967\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf"
Oct 30 21:11:09.214117 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:09.214084    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v5vv\" (UniqueName: \"kubernetes.io/projected/c59144f6-c298-4a9d-a3c3-7a2e81bb8967-kube-api-access-7v5vv\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf\" (UID: \"c59144f6-c298-4a9d-a3c3-7a2e81bb8967\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf"
Oct 30 21:11:09.214332 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:09.214140    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c59144f6-c298-4a9d-a3c3-7a2e81bb8967-istio-token\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf\" (UID: \"c59144f6-c298-4a9d-a3c3-7a2e81bb8967\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf"
Oct 30 21:11:09.214332 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:09.214242    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c59144f6-c298-4a9d-a3c3-7a2e81bb8967-istiod-ca-cert\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf\" (UID: \"c59144f6-c298-4a9d-a3c3-7a2e81bb8967\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf"
Oct 30 21:11:09.214332 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:09.214300    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c59144f6-c298-4a9d-a3c3-7a2e81bb8967-istio-podinfo\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf\" (UID: \"c59144f6-c298-4a9d-a3c3-7a2e81bb8967\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf"
Oct 30 21:11:09.214515 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:09.214379    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c59144f6-c298-4a9d-a3c3-7a2e81bb8967-istio-data\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf\" (UID: \"c59144f6-c298-4a9d-a3c3-7a2e81bb8967\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf"
Oct 30 21:11:09.214515 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:09.214415    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c59144f6-c298-4a9d-a3c3-7a2e81bb8967-credential-socket\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf\" (UID: \"c59144f6-c298-4a9d-a3c3-7a2e81bb8967\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf"
Oct 30 21:11:09.214626 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:09.214545    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c59144f6-c298-4a9d-a3c3-7a2e81bb8967-workload-certs\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf\" (UID: \"c59144f6-c298-4a9d-a3c3-7a2e81bb8967\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf"
Oct 30 21:11:09.214626 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:09.214596    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c59144f6-c298-4a9d-a3c3-7a2e81bb8967-workload-socket\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf\" (UID: \"c59144f6-c298-4a9d-a3c3-7a2e81bb8967\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf"
Oct 30 21:11:09.214733 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:09.214641    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c59144f6-c298-4a9d-a3c3-7a2e81bb8967-istio-envoy\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf\" (UID: \"c59144f6-c298-4a9d-a3c3-7a2e81bb8967\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf"
Oct 30 21:11:09.215062 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:09.215034    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c59144f6-c298-4a9d-a3c3-7a2e81bb8967-workload-socket\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf\" (UID: \"c59144f6-c298-4a9d-a3c3-7a2e81bb8967\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf"
Oct 30 21:11:09.215218 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:09.215185    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c59144f6-c298-4a9d-a3c3-7a2e81bb8967-credential-socket\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf\" (UID: \"c59144f6-c298-4a9d-a3c3-7a2e81bb8967\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf"
Oct 30 21:11:09.215301 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:09.215153    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c59144f6-c298-4a9d-a3c3-7a2e81bb8967-workload-certs\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf\" (UID: \"c59144f6-c298-4a9d-a3c3-7a2e81bb8967\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf"
Oct 30 21:11:09.215367 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:09.215172    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c59144f6-c298-4a9d-a3c3-7a2e81bb8967-istiod-ca-cert\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf\" (UID: \"c59144f6-c298-4a9d-a3c3-7a2e81bb8967\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf"
Oct 30 21:11:09.215429 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:09.215402    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c59144f6-c298-4a9d-a3c3-7a2e81bb8967-istio-data\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf\" (UID: \"c59144f6-c298-4a9d-a3c3-7a2e81bb8967\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf"
Oct 30 21:11:09.217361 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:09.217337    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c59144f6-c298-4a9d-a3c3-7a2e81bb8967-istio-podinfo\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf\" (UID: \"c59144f6-c298-4a9d-a3c3-7a2e81bb8967\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf"
Oct 30 21:11:09.217972 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:09.217943    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c59144f6-c298-4a9d-a3c3-7a2e81bb8967-istio-envoy\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf\" (UID: \"c59144f6-c298-4a9d-a3c3-7a2e81bb8967\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf"
Oct 30 21:11:09.274546 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:09.274471    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c59144f6-c298-4a9d-a3c3-7a2e81bb8967-istio-token\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf\" (UID: \"c59144f6-c298-4a9d-a3c3-7a2e81bb8967\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf"
Oct 30 21:11:09.274546 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:09.274522    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v5vv\" (UniqueName: \"kubernetes.io/projected/c59144f6-c298-4a9d-a3c3-7a2e81bb8967-kube-api-access-7v5vv\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf\" (UID: \"c59144f6-c298-4a9d-a3c3-7a2e81bb8967\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf"
Oct 30 21:11:09.299594 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:09.299564    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck"]
Oct 30 21:11:09.305288 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:09.305266    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck"
Oct 30 21:11:09.307051 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:09.307029    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf"
Oct 30 21:11:09.362622 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:09.361152    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-ef7e2d-openshift-default-dockercfg-fhxft\""
Oct 30 21:11:09.368533 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:09.368499    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck"]
Oct 30 21:11:09.422031 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:09.421986    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da8c670-dd6c-4a19-856e-0241a7494dbb-credential-socket\") pod \"e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck\" (UID: \"0da8c670-dd6c-4a19-856e-0241a7494dbb\") " pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck"
Oct 30 21:11:09.422031 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:09.422031    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0da8c670-dd6c-4a19-856e-0241a7494dbb-istio-podinfo\") pod \"e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck\" (UID: \"0da8c670-dd6c-4a19-856e-0241a7494dbb\") " pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck"
Oct 30 21:11:09.422270 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:09.422150    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0da8c670-dd6c-4a19-856e-0241a7494dbb-istio-data\") pod \"e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck\" (UID: \"0da8c670-dd6c-4a19-856e-0241a7494dbb\") " pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck"
Oct 30 21:11:09.422270 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:09.422256    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0da8c670-dd6c-4a19-856e-0241a7494dbb-istio-envoy\") pod \"e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck\" (UID: \"0da8c670-dd6c-4a19-856e-0241a7494dbb\") " pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck"
Oct 30 21:11:09.422356 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:09.422334    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jjx8\" (UniqueName: \"kubernetes.io/projected/0da8c670-dd6c-4a19-856e-0241a7494dbb-kube-api-access-8jjx8\") pod \"e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck\" (UID: \"0da8c670-dd6c-4a19-856e-0241a7494dbb\") " pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck"
Oct 30 21:11:09.422445 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:09.422427    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0da8c670-dd6c-4a19-856e-0241a7494dbb-istio-token\") pod \"e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck\" (UID: \"0da8c670-dd6c-4a19-856e-0241a7494dbb\") " pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck"
Oct 30 21:11:09.422540 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:09.422525    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0da8c670-dd6c-4a19-856e-0241a7494dbb-workload-certs\") pod \"e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck\" (UID: \"0da8c670-dd6c-4a19-856e-0241a7494dbb\") " pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck"
Oct 30 21:11:09.422617 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:09.422603    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da8c670-dd6c-4a19-856e-0241a7494dbb-workload-socket\") pod \"e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck\" (UID: \"0da8c670-dd6c-4a19-856e-0241a7494dbb\") " pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck"
Oct 30 21:11:09.422707 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:09.422692    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0da8c670-dd6c-4a19-856e-0241a7494dbb-istiod-ca-cert\") pod \"e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck\" (UID: \"0da8c670-dd6c-4a19-856e-0241a7494dbb\") " pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck"
Oct 30 21:11:09.524152 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:09.524111    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0da8c670-dd6c-4a19-856e-0241a7494dbb-workload-certs\") pod \"e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck\" (UID: \"0da8c670-dd6c-4a19-856e-0241a7494dbb\") " pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck"
Oct 30 21:11:09.524342 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:09.524184    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da8c670-dd6c-4a19-856e-0241a7494dbb-workload-socket\") pod \"e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck\" (UID: \"0da8c670-dd6c-4a19-856e-0241a7494dbb\") " pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck"
Oct 30 21:11:09.524342 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:09.524238    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0da8c670-dd6c-4a19-856e-0241a7494dbb-istiod-ca-cert\") pod \"e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck\" (UID: \"0da8c670-dd6c-4a19-856e-0241a7494dbb\") " pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck"
Oct 30 21:11:09.524342 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:09.524338    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da8c670-dd6c-4a19-856e-0241a7494dbb-credential-socket\") pod \"e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck\" (UID: \"0da8c670-dd6c-4a19-856e-0241a7494dbb\") " pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck"
Oct 30 21:11:09.524518 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:09.524362    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0da8c670-dd6c-4a19-856e-0241a7494dbb-istio-podinfo\") pod \"e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck\" (UID: \"0da8c670-dd6c-4a19-856e-0241a7494dbb\") " pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck"
Oct 30 21:11:09.524518 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:09.524397    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0da8c670-dd6c-4a19-856e-0241a7494dbb-istio-data\") pod \"e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck\" (UID: \"0da8c670-dd6c-4a19-856e-0241a7494dbb\") " pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck"
Oct 30 21:11:09.524518 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:09.524466    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0da8c670-dd6c-4a19-856e-0241a7494dbb-istio-envoy\") pod \"e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck\" (UID: \"0da8c670-dd6c-4a19-856e-0241a7494dbb\") " pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck"
Oct 30 21:11:09.524709 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:09.524539    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jjx8\" (UniqueName: \"kubernetes.io/projected/0da8c670-dd6c-4a19-856e-0241a7494dbb-kube-api-access-8jjx8\") pod \"e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck\" (UID: \"0da8c670-dd6c-4a19-856e-0241a7494dbb\") " pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck"
Oct 30 21:11:09.524709 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:09.524574    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0da8c670-dd6c-4a19-856e-0241a7494dbb-workload-certs\") pod \"e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck\" (UID: \"0da8c670-dd6c-4a19-856e-0241a7494dbb\") " pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck"
Oct 30 21:11:09.524709 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:09.524607    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da8c670-dd6c-4a19-856e-0241a7494dbb-workload-socket\") pod \"e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck\" (UID: \"0da8c670-dd6c-4a19-856e-0241a7494dbb\") " pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck"
Oct 30 21:11:09.524709 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:09.524644    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0da8c670-dd6c-4a19-856e-0241a7494dbb-istio-token\") pod \"e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck\" (UID: \"0da8c670-dd6c-4a19-856e-0241a7494dbb\") " pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck"
Oct 30 21:11:09.524709 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:09.524674    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da8c670-dd6c-4a19-856e-0241a7494dbb-credential-socket\") pod \"e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck\" (UID: \"0da8c670-dd6c-4a19-856e-0241a7494dbb\") " pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck"
Oct 30 21:11:09.525038 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:09.524824    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0da8c670-dd6c-4a19-856e-0241a7494dbb-istio-data\") pod \"e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck\" (UID: \"0da8c670-dd6c-4a19-856e-0241a7494dbb\") " pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck"
Oct 30 21:11:09.525122 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:09.525100    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0da8c670-dd6c-4a19-856e-0241a7494dbb-istiod-ca-cert\") pod \"e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck\" (UID: \"0da8c670-dd6c-4a19-856e-0241a7494dbb\") " pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck"
Oct 30 21:11:09.526995 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:09.526977    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0da8c670-dd6c-4a19-856e-0241a7494dbb-istio-envoy\") pod \"e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck\" (UID: \"0da8c670-dd6c-4a19-856e-0241a7494dbb\") " pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck"
Oct 30 21:11:09.527233 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:09.527215    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0da8c670-dd6c-4a19-856e-0241a7494dbb-istio-podinfo\") pod \"e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck\" (UID: \"0da8c670-dd6c-4a19-856e-0241a7494dbb\") " pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck"
Oct 30 21:11:09.581757 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:09.581726    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0da8c670-dd6c-4a19-856e-0241a7494dbb-istio-token\") pod \"e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck\" (UID: \"0da8c670-dd6c-4a19-856e-0241a7494dbb\") " pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck"
Oct 30 21:11:09.581959 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:09.581892    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jjx8\" (UniqueName: \"kubernetes.io/projected/0da8c670-dd6c-4a19-856e-0241a7494dbb-kube-api-access-8jjx8\") pod \"e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck\" (UID: \"0da8c670-dd6c-4a19-856e-0241a7494dbb\") " pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck"
Oct 30 21:11:09.626847 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:09.626769    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck"
Oct 30 21:11:09.759421 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:11:09.759385    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc59144f6_c298_4a9d_a3c3_7a2e81bb8967.slice/crio-89c29e1944e8726a040ff32d992f7784032cdb388c6e47ae84b3f91706f6ea4a WatchSource:0}: Error finding container 89c29e1944e8726a040ff32d992f7784032cdb388c6e47ae84b3f91706f6ea4a: Status 404 returned error can't find the container with id 89c29e1944e8726a040ff32d992f7784032cdb388c6e47ae84b3f91706f6ea4a
Oct 30 21:11:09.761395 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:09.761369    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf"]
Oct 30 21:11:09.763738 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:09.763698    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:11:09.763882 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:09.763775    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:11:09.763882 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:09.763830    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:11:10.016895 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:10.016868    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck"]
Oct 30 21:11:10.017293 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:11:10.016921    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0da8c670_dd6c_4a19_856e_0241a7494dbb.slice/crio-1a7d823eb45db6e55829818f4cd233185aa41ee695da34da583cf670d4b5f08e WatchSource:0}: Error finding container 1a7d823eb45db6e55829818f4cd233185aa41ee695da34da583cf670d4b5f08e: Status 404 returned error can't find the container with id 1a7d823eb45db6e55829818f4cd233185aa41ee695da34da583cf670d4b5f08e
Oct 30 21:11:10.020129 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:10.020098    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:11:10.020206 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:10.020162    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:11:10.020206 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:10.020189    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:11:10.366269 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:10.366160    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf" event={"ID":"c59144f6-c298-4a9d-a3c3-7a2e81bb8967","Type":"ContainerStarted","Data":"0fcb783b7658411186f3e3c318cf06459e180201e511a6ef12564779e28124fe"}
Oct 30 21:11:10.366269 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:10.366231    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf" event={"ID":"c59144f6-c298-4a9d-a3c3-7a2e81bb8967","Type":"ContainerStarted","Data":"89c29e1944e8726a040ff32d992f7784032cdb388c6e47ae84b3f91706f6ea4a"}
Oct 30 21:11:10.368255 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:10.368227    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck" event={"ID":"0da8c670-dd6c-4a19-856e-0241a7494dbb","Type":"ContainerStarted","Data":"5976c2a9326676d051cc805f8fd5bc59ab319b0c58f895444242fc6d2ed8c2bc"}
Oct 30 21:11:10.368393 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:10.368282    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck" event={"ID":"0da8c670-dd6c-4a19-856e-0241a7494dbb","Type":"ContainerStarted","Data":"1a7d823eb45db6e55829818f4cd233185aa41ee695da34da583cf670d4b5f08e"}
Oct 30 21:11:10.490357 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:10.489671    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf" podStartSLOduration=2.489650272 podStartE2EDuration="2.489650272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:11:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:10.482039264 +0000 UTC m=+1953.546815662" watchObservedRunningTime="2026-10-30 21:11:10.489650272 +0000 UTC m=+1953.554426671"
Oct 30 21:11:10.606020 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:10.605968    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck" podStartSLOduration=1.605953333 podStartE2EDuration="1.605953333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:11:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:10.600733637 +0000 UTC m=+1953.665510035" watchObservedRunningTime="2026-10-30 21:11:10.605953333 +0000 UTC m=+1953.670729730"
Oct 30 21:11:10.627709 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:10.627635    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck"
Oct 30 21:11:11.308128 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:11.308095    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf"
Oct 30 21:11:11.309387 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:11.309358    5713 patch_prober.go:29] interesting pod/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Oct 30 21:11:11.309523 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:11.309408    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf" podUID="c59144f6-c298-4a9d-a3c3-7a2e81bb8967" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Oct 30 21:11:11.393979 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:11.393937    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9926f7-8ldbl"]
Oct 30 21:11:11.399739 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:11.399719    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9926f7-8ldbl"
Oct 30 21:11:11.482817 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:11.473119    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:11:11.483037 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:11.482984    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:11:11.492873 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:11.491234    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c7c5226-001b-4e5f-90ee-a0392597425d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9926f7-8ldbl\" (UID: \"0c7c5226-001b-4e5f-90ee-a0392597425d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9926f7-8ldbl"
Oct 30 21:11:11.492873 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:11.491376    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c7c5226-001b-4e5f-90ee-a0392597425d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9926f7-8ldbl\" (UID: \"0c7c5226-001b-4e5f-90ee-a0392597425d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9926f7-8ldbl"
Oct 30 21:11:11.492873 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:11.491457    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhxgn\" (UniqueName: \"kubernetes.io/projected/0c7c5226-001b-4e5f-90ee-a0392597425d-kube-api-access-hhxgn\") pod \"maas-api-revoke-keys-e2e-api-a-9926f7-8ldbl\" (UID: \"0c7c5226-001b-4e5f-90ee-a0392597425d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9926f7-8ldbl"
Oct 30 21:11:11.493772 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:11.493200    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9926f7-8ldbl"]
Oct 30 21:11:11.503089 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:11.503061    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:11:11.503452 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:11.503431    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:11:11.593425 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:11.592987    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c7c5226-001b-4e5f-90ee-a0392597425d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9926f7-8ldbl\" (UID: \"0c7c5226-001b-4e5f-90ee-a0392597425d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9926f7-8ldbl"
Oct 30 21:11:11.593425 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:11.593081    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c7c5226-001b-4e5f-90ee-a0392597425d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9926f7-8ldbl\" (UID: \"0c7c5226-001b-4e5f-90ee-a0392597425d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9926f7-8ldbl"
Oct 30 21:11:11.593425 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:11.593118    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhxgn\" (UniqueName: \"kubernetes.io/projected/0c7c5226-001b-4e5f-90ee-a0392597425d-kube-api-access-hhxgn\") pod \"maas-api-revoke-keys-e2e-api-a-9926f7-8ldbl\" (UID: \"0c7c5226-001b-4e5f-90ee-a0392597425d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9926f7-8ldbl"
Oct 30 21:11:11.593710 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:11.593689    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c7c5226-001b-4e5f-90ee-a0392597425d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9926f7-8ldbl\" (UID: \"0c7c5226-001b-4e5f-90ee-a0392597425d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9926f7-8ldbl"
Oct 30 21:11:11.596161 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:11.596132    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c7c5226-001b-4e5f-90ee-a0392597425d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9926f7-8ldbl\" (UID: \"0c7c5226-001b-4e5f-90ee-a0392597425d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9926f7-8ldbl"
Oct 30 21:11:11.629193 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:11.629166    5713 patch_prober.go:29] interesting pod/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Oct 30 21:11:11.629347 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:11.629214    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck" podUID="0da8c670-dd6c-4a19-856e-0241a7494dbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Oct 30 21:11:11.660813 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:11.660548    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhxgn\" (UniqueName: \"kubernetes.io/projected/0c7c5226-001b-4e5f-90ee-a0392597425d-kube-api-access-hhxgn\") pod \"maas-api-revoke-keys-e2e-api-a-9926f7-8ldbl\" (UID: \"0c7c5226-001b-4e5f-90ee-a0392597425d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9926f7-8ldbl"
Oct 30 21:11:11.715385 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:11.715349    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9926f7-8ldbl"
Oct 30 21:11:12.307485 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:12.307457    5713 patch_prober.go:29] interesting pod/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Oct 30 21:11:12.307651 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:12.307501    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf" podUID="c59144f6-c298-4a9d-a3c3-7a2e81bb8967" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Oct 30 21:11:12.417961 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:12.417937    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9926f7-8ldbl"]
Oct 30 21:11:12.420303 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:11:12.420274    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c7c5226_001b_4e5f_90ee_a0392597425d.slice/crio-1ea53ed136ffc28f3f6398f1800d107316c4368bfa7c885ea4580302ad229065 WatchSource:0}: Error finding container 1ea53ed136ffc28f3f6398f1800d107316c4368bfa7c885ea4580302ad229065: Status 404 returned error can't find the container with id 1ea53ed136ffc28f3f6398f1800d107316c4368bfa7c885ea4580302ad229065
Oct 30 21:11:12.627666 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:12.627537    5713 patch_prober.go:29] interesting pod/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Oct 30 21:11:12.627666 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:12.627595    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck" podUID="0da8c670-dd6c-4a19-856e-0241a7494dbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Oct 30 21:11:13.307871 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:13.307757    5713 patch_prober.go:29] interesting pod/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Oct 30 21:11:13.307871 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:13.307822    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf" podUID="c59144f6-c298-4a9d-a3c3-7a2e81bb8967" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Oct 30 21:11:13.414602 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:13.414569    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9926f7-8ldbl" event={"ID":"0c7c5226-001b-4e5f-90ee-a0392597425d","Type":"ContainerStarted","Data":"a01279cbb593eac7fc7dfaef3670291c2c6940a4f73e45a8ad18a3f0b75eaafd"}
Oct 30 21:11:13.414804 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:13.414628    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9926f7-8ldbl" event={"ID":"0c7c5226-001b-4e5f-90ee-a0392597425d","Type":"ContainerStarted","Data":"1ea53ed136ffc28f3f6398f1800d107316c4368bfa7c885ea4580302ad229065"}
Oct 30 21:11:13.627739 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:13.627649    5713 patch_prober.go:29] interesting pod/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Oct 30 21:11:13.627739 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:13.627711    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck" podUID="0da8c670-dd6c-4a19-856e-0241a7494dbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Oct 30 21:11:14.308320 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:14.308284    5713 patch_prober.go:29] interesting pod/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Oct 30 21:11:14.308479 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:14.308340    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf" podUID="c59144f6-c298-4a9d-a3c3-7a2e81bb8967" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Oct 30 21:11:14.423636 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:14.423597    5713 generic.go:356] "Generic (PLEG): container finished" podID="0c7c5226-001b-4e5f-90ee-a0392597425d" containerID="a01279cbb593eac7fc7dfaef3670291c2c6940a4f73e45a8ad18a3f0b75eaafd" exitCode=0
Oct 30 21:11:14.423846 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:14.423689    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9926f7-8ldbl" event={"ID":"0c7c5226-001b-4e5f-90ee-a0392597425d","Type":"ContainerDied","Data":"a01279cbb593eac7fc7dfaef3670291c2c6940a4f73e45a8ad18a3f0b75eaafd"}
Oct 30 21:11:14.628010 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:14.627905    5713 patch_prober.go:29] interesting pod/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Oct 30 21:11:14.628010 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:14.627975    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck" podUID="0da8c670-dd6c-4a19-856e-0241a7494dbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Oct 30 21:11:15.308106 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:15.308059    5713 patch_prober.go:29] interesting pod/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Oct 30 21:11:15.308299 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:15.308128    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf" podUID="c59144f6-c298-4a9d-a3c3-7a2e81bb8967" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Oct 30 21:11:15.628092 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:15.627900    5713 patch_prober.go:29] interesting pod/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Oct 30 21:11:15.628092 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:15.627990    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck" podUID="0da8c670-dd6c-4a19-856e-0241a7494dbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Oct 30 21:11:15.804411 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:15.804096    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9926f7-8ldbl"
Oct 30 21:11:15.930960 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:15.930931    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c7c5226-001b-4e5f-90ee-a0392597425d-kube-api-access-hhxgn\" (UniqueName: \"kubernetes.io/projected/0c7c5226-001b-4e5f-90ee-a0392597425d-kube-api-access-hhxgn\") pod \"0c7c5226-001b-4e5f-90ee-a0392597425d\" (UID: \"0c7c5226-001b-4e5f-90ee-a0392597425d\") "
Oct 30 21:11:15.931119 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:15.931092    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c7c5226-001b-4e5f-90ee-a0392597425d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c7c5226-001b-4e5f-90ee-a0392597425d-maas-api-xks-ca\") pod \"0c7c5226-001b-4e5f-90ee-a0392597425d\" (UID: \"0c7c5226-001b-4e5f-90ee-a0392597425d\") "
Oct 30 21:11:15.931194 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:15.931177    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c7c5226-001b-4e5f-90ee-a0392597425d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c7c5226-001b-4e5f-90ee-a0392597425d-maas-api-openshift-ca\") pod \"0c7c5226-001b-4e5f-90ee-a0392597425d\" (UID: \"0c7c5226-001b-4e5f-90ee-a0392597425d\") "
Oct 30 21:11:15.931505 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:15.931479    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c7c5226-001b-4e5f-90ee-a0392597425d-maas-api-openshift-ca" pod "0c7c5226-001b-4e5f-90ee-a0392597425d" (UID: "0c7c5226-001b-4e5f-90ee-a0392597425d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:11:15.931881 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:15.931864    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c7c5226-001b-4e5f-90ee-a0392597425d-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:11:15.933670 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:15.933647    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c7c5226-001b-4e5f-90ee-a0392597425d-kube-api-access-hhxgn" pod "0c7c5226-001b-4e5f-90ee-a0392597425d" (UID: "0c7c5226-001b-4e5f-90ee-a0392597425d"). InnerVolumeSpecName "kube-api-access-hhxgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:11:15.950609 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:15.949903    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c7c5226-001b-4e5f-90ee-a0392597425d-maas-api-xks-ca" pod "0c7c5226-001b-4e5f-90ee-a0392597425d" (UID: "0c7c5226-001b-4e5f-90ee-a0392597425d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:11:16.092444 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:16.092152    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhxgn\" (UniqueName: \"kubernetes.io/projected/0c7c5226-001b-4e5f-90ee-a0392597425d-kube-api-access-hhxgn\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:11:16.095074 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:16.092675    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c7c5226-001b-4e5f-90ee-a0392597425d-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:11:16.307961 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:16.307777    5713 patch_prober.go:29] interesting pod/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Oct 30 21:11:16.307961 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:16.307866    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf" podUID="c59144f6-c298-4a9d-a3c3-7a2e81bb8967" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Oct 30 21:11:16.456833 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:16.455977    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9926f7-8ldbl"
Oct 30 21:11:16.456833 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:16.456018    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9926f7-8ldbl" event={"ID":"0c7c5226-001b-4e5f-90ee-a0392597425d","Type":"ContainerDied","Data":"1ea53ed136ffc28f3f6398f1800d107316c4368bfa7c885ea4580302ad229065"}
Oct 30 21:11:16.456833 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:16.456069    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ea53ed136ffc28f3f6398f1800d107316c4368bfa7c885ea4580302ad229065"
Oct 30 21:11:16.631877 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:16.631019    5713 patch_prober.go:29] interesting pod/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Oct 30 21:11:16.631877 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:16.631084    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck" podUID="0da8c670-dd6c-4a19-856e-0241a7494dbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Oct 30 21:11:17.312648 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:17.312618    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf"
Oct 30 21:11:17.313091 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:17.313073    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf"
Oct 30 21:11:17.313718 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:17.313696    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf"
Oct 30 21:11:17.634867 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:17.633979    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck"
Oct 30 21:11:17.634867 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:17.634690    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck"
Oct 30 21:11:17.665863 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:17.637626    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck"
Oct 30 21:11:24.463733 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:11:24.463692    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:11:26.298187 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:26.297038    5713 patch_prober.go:29] interesting pod/e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": read tcp 10.134.0.2:57592->10.134.0.150:15021: read: connection reset by peer" start-of-body=
Oct 30 21:11:26.298187 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:26.297100    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q" podUID="ec28cb7b-925f-4b32-9ff8-d75caf2f3d14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": read tcp 10.134.0.2:57592->10.134.0.150:15021: read: connection reset by peer"
Oct 30 21:11:26.548737 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:26.548638    5713 generic.go:356] "Generic (PLEG): container finished" podID="ec28cb7b-925f-4b32-9ff8-d75caf2f3d14" containerID="00705fe9f3bf9f96294b219bba02b81f3ba219ce426d7b3825bfee73c1b9f2d7" exitCode=0
Oct 30 21:11:26.548737 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:26.548715    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q" event={"ID":"ec28cb7b-925f-4b32-9ff8-d75caf2f3d14","Type":"ContainerDied","Data":"00705fe9f3bf9f96294b219bba02b81f3ba219ce426d7b3825bfee73c1b9f2d7"}
Oct 30 21:11:26.549230 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:26.549213    5713 scope.go:122] "RemoveContainer" containerID="00705fe9f3bf9f96294b219bba02b81f3ba219ce426d7b3825bfee73c1b9f2d7"
Oct 30 21:11:26.550962 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:26.550934    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:11:26.551022 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:26.551001    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:11:26.551057 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:26.551033    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:11:27.569399 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:27.569363    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q" event={"ID":"ec28cb7b-925f-4b32-9ff8-d75caf2f3d14","Type":"ContainerStarted","Data":"38a6d001c0c0e7a02be45091eb6e21fc7f7b4f8e5e66e31c875c39424edb5f7e"}
Oct 30 21:11:27.602488 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:27.602460    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q"
Oct 30 21:11:27.602648 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:27.602616    5713 patch_prober.go:29] interesting pod/e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body=
Oct 30 21:11:27.602696 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:27.602657    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q" podUID="ec28cb7b-925f-4b32-9ff8-d75caf2f3d14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused"
Oct 30 21:11:28.602610 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:28.602568    5713 patch_prober.go:29] interesting pod/e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body=
Oct 30 21:11:28.603076 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:28.602638    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q" podUID="ec28cb7b-925f-4b32-9ff8-d75caf2f3d14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused"
Oct 30 21:11:29.602973 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:29.602938    5713 patch_prober.go:29] interesting pod/e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body=
Oct 30 21:11:29.603387 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:29.603009    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q" podUID="ec28cb7b-925f-4b32-9ff8-d75caf2f3d14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused"
Oct 30 21:11:30.610000 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:30.609947    5713 patch_prober.go:29] interesting pod/e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body=
Oct 30 21:11:30.610634 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:30.610583    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q" podUID="ec28cb7b-925f-4b32-9ff8-d75caf2f3d14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused"
Oct 30 21:11:31.603406 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:31.603370    5713 patch_prober.go:29] interesting pod/e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body=
Oct 30 21:11:31.603612 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:31.603430    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q" podUID="ec28cb7b-925f-4b32-9ff8-d75caf2f3d14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused"
Oct 30 21:11:32.603255 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:32.603220    5713 patch_prober.go:29] interesting pod/e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body=
Oct 30 21:11:32.603743 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:32.603297    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q" podUID="ec28cb7b-925f-4b32-9ff8-d75caf2f3d14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused"
Oct 30 21:11:33.607243 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:33.607210    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q"
Oct 30 21:11:33.607747 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:33.607615    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q"
Oct 30 21:11:33.608263 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:33.608243    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q"
Oct 30 21:11:38.628865 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:38.628832    5713 scope.go:122] "RemoveContainer" containerID="1dcae5b4d62b3b70ef7e3b26b089786d1f2fd0aba75fc3da23173c4ffaff3b20"
Oct 30 21:11:38.639451 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:38.639429    5713 scope.go:122] "RemoveContainer" containerID="7cc5a6e50d2524ac9ca47aa75cde48fe8031aea6a1c398b8a70592914995a270"
Oct 30 21:11:39.211835 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:39.211801    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz"]
Oct 30 21:11:39.212576 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:39.212560    5713 cpu_manager.go:414] "containerMap: removing container" podUID="0c7c5226-001b-4e5f-90ee-a0392597425d" containerName="revoke-keys"
Oct 30 21:11:39.212623 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:39.212578    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c7c5226-001b-4e5f-90ee-a0392597425d" containerName="revoke-keys"
Oct 30 21:11:39.212727 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:39.212718    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c7c5226-001b-4e5f-90ee-a0392597425d" containerName="revoke-keys"
Oct 30 21:11:39.217953 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:39.217935    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz"
Oct 30 21:11:39.261485 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:39.261450    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4"]
Oct 30 21:11:39.266019 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:39.266002    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4"
Oct 30 21:11:39.308743 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:39.308700    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b85e68b-c6a9-4362-a6c0-93af13623e67-istio-data\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4\" (UID: \"0b85e68b-c6a9-4362-a6c0-93af13623e67\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4"
Oct 30 21:11:39.308956 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:39.308753    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b85e68b-c6a9-4362-a6c0-93af13623e67-credential-socket\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4\" (UID: \"0b85e68b-c6a9-4362-a6c0-93af13623e67\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4"
Oct 30 21:11:39.308956 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:39.308880    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b85e68b-c6a9-4362-a6c0-93af13623e67-workload-socket\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4\" (UID: \"0b85e68b-c6a9-4362-a6c0-93af13623e67\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4"
Oct 30 21:11:39.308956 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:39.308935    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f-istio-envoy\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz\" (UID: \"d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz"
Oct 30 21:11:39.309085 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:39.309036    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f-credential-socket\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz\" (UID: \"d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz"
Oct 30 21:11:39.309128 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:39.309102    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f-workload-certs\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz\" (UID: \"d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz"
Oct 30 21:11:39.309179 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:39.309162    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdc2n\" (UniqueName: \"kubernetes.io/projected/0b85e68b-c6a9-4362-a6c0-93af13623e67-kube-api-access-tdc2n\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4\" (UID: \"0b85e68b-c6a9-4362-a6c0-93af13623e67\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4"
Oct 30 21:11:39.309275 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:39.309260    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f-istio-data\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz\" (UID: \"d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz"
Oct 30 21:11:39.309338 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:39.309293    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f-istio-token\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz\" (UID: \"d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz"
Oct 30 21:11:39.309425 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:39.309406    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f-istio-podinfo\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz\" (UID: \"d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz"
Oct 30 21:11:39.309463 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:39.309445    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b85e68b-c6a9-4362-a6c0-93af13623e67-workload-certs\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4\" (UID: \"0b85e68b-c6a9-4362-a6c0-93af13623e67\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4"
Oct 30 21:11:39.309526 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:39.309515    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b85e68b-c6a9-4362-a6c0-93af13623e67-istio-podinfo\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4\" (UID: \"0b85e68b-c6a9-4362-a6c0-93af13623e67\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4"
Oct 30 21:11:39.309610 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:39.309593    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b85e68b-c6a9-4362-a6c0-93af13623e67-istio-token\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4\" (UID: \"0b85e68b-c6a9-4362-a6c0-93af13623e67\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4"
Oct 30 21:11:39.309648 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:39.309628    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9m6b\" (UniqueName: \"kubernetes.io/projected/d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f-kube-api-access-v9m6b\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz\" (UID: \"d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz"
Oct 30 21:11:39.309724 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:39.309709    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f-workload-socket\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz\" (UID: \"d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz"
Oct 30 21:11:39.309881 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:39.309861    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b85e68b-c6a9-4362-a6c0-93af13623e67-istiod-ca-cert\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4\" (UID: \"0b85e68b-c6a9-4362-a6c0-93af13623e67\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4"
Oct 30 21:11:39.309941 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:39.309891    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f-istiod-ca-cert\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz\" (UID: \"d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz"
Oct 30 21:11:39.309941 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:39.309914    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b85e68b-c6a9-4362-a6c0-93af13623e67-istio-envoy\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4\" (UID: \"0b85e68b-c6a9-4362-a6c0-93af13623e67\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4"
Oct 30 21:11:39.315758 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:39.315735    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz"]
Oct 30 21:11:39.370644 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:39.370608    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4"]
Oct 30 21:11:39.411032 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:39.411001    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f-istio-token\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz\" (UID: \"d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz"
Oct 30 21:11:39.411242 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:39.411055    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f-istio-podinfo\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz\" (UID: \"d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz"
Oct 30 21:11:39.411321 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:39.411238    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b85e68b-c6a9-4362-a6c0-93af13623e67-workload-certs\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4\" (UID: \"0b85e68b-c6a9-4362-a6c0-93af13623e67\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4"
Oct 30 21:11:39.411383 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:39.411316    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b85e68b-c6a9-4362-a6c0-93af13623e67-istio-podinfo\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4\" (UID: \"0b85e68b-c6a9-4362-a6c0-93af13623e67\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4"
Oct 30 21:11:39.411434 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:39.411391    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b85e68b-c6a9-4362-a6c0-93af13623e67-istio-token\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4\" (UID: \"0b85e68b-c6a9-4362-a6c0-93af13623e67\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4"
Oct 30 21:11:39.411488 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:39.411433    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9m6b\" (UniqueName: \"kubernetes.io/projected/d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f-kube-api-access-v9m6b\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz\" (UID: \"d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz"
Oct 30 21:11:39.411547 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:39.411496    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f-workload-socket\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz\" (UID: \"d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz"
Oct 30 21:11:39.411649 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:39.411627    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b85e68b-c6a9-4362-a6c0-93af13623e67-workload-certs\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4\" (UID: \"0b85e68b-c6a9-4362-a6c0-93af13623e67\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4"
Oct 30 21:11:39.411715 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:39.411648    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b85e68b-c6a9-4362-a6c0-93af13623e67-istiod-ca-cert\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4\" (UID: \"0b85e68b-c6a9-4362-a6c0-93af13623e67\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4"
Oct 30 21:11:39.411715 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:39.411693    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f-istiod-ca-cert\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz\" (UID: \"d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz"
Oct 30 21:11:39.411848 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:39.411740    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b85e68b-c6a9-4362-a6c0-93af13623e67-istio-envoy\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4\" (UID: \"0b85e68b-c6a9-4362-a6c0-93af13623e67\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4"
Oct 30 21:11:39.412008 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:39.411989    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b85e68b-c6a9-4362-a6c0-93af13623e67-istio-data\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4\" (UID: \"0b85e68b-c6a9-4362-a6c0-93af13623e67\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4"
Oct 30 21:11:39.412100 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:39.412038    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b85e68b-c6a9-4362-a6c0-93af13623e67-credential-socket\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4\" (UID: \"0b85e68b-c6a9-4362-a6c0-93af13623e67\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4"
Oct 30 21:11:39.412305 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:39.412257    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b85e68b-c6a9-4362-a6c0-93af13623e67-workload-socket\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4\" (UID: \"0b85e68b-c6a9-4362-a6c0-93af13623e67\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4"
Oct 30 21:11:39.412390 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:39.412307    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f-istio-envoy\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz\" (UID: \"d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz"
Oct 30 21:11:39.412390 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:39.412363    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b85e68b-c6a9-4362-a6c0-93af13623e67-credential-socket\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4\" (UID: \"0b85e68b-c6a9-4362-a6c0-93af13623e67\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4"
Oct 30 21:11:39.412390 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:39.412377    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b85e68b-c6a9-4362-a6c0-93af13623e67-istiod-ca-cert\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4\" (UID: \"0b85e68b-c6a9-4362-a6c0-93af13623e67\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4"
Oct 30 21:11:39.412563 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:39.412378    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f-credential-socket\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz\" (UID: \"d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz"
Oct 30 21:11:39.412563 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:39.412464    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f-workload-certs\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz\" (UID: \"d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz"
Oct 30 21:11:39.412563 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:39.412532    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdc2n\" (UniqueName: \"kubernetes.io/projected/0b85e68b-c6a9-4362-a6c0-93af13623e67-kube-api-access-tdc2n\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4\" (UID: \"0b85e68b-c6a9-4362-a6c0-93af13623e67\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4"
Oct 30 21:11:39.412721 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:39.412610    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f-istio-data\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz\" (UID: \"d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz"
Oct 30 21:11:39.412780 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:39.412712    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f-credential-socket\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz\" (UID: \"d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz"
Oct 30 21:11:39.412780 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:39.412756    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f-workload-socket\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz\" (UID: \"d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz"
Oct 30 21:11:39.412944 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:39.412824    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b85e68b-c6a9-4362-a6c0-93af13623e67-workload-socket\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4\" (UID: \"0b85e68b-c6a9-4362-a6c0-93af13623e67\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4"
Oct 30 21:11:39.413042 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:39.413017    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b85e68b-c6a9-4362-a6c0-93af13623e67-istio-data\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4\" (UID: \"0b85e68b-c6a9-4362-a6c0-93af13623e67\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4"
Oct 30 21:11:39.413123 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:39.413102    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f-workload-certs\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz\" (UID: \"d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz"
Oct 30 21:11:39.413314 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:39.413292    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f-istio-data\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz\" (UID: \"d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz"
Oct 30 21:11:39.413634 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:39.413615    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f-istiod-ca-cert\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz\" (UID: \"d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz"
Oct 30 21:11:39.414582 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:39.414558    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f-istio-podinfo\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz\" (UID: \"d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz"
Oct 30 21:11:39.415605 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:39.415573    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f-istio-envoy\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz\" (UID: \"d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz"
Oct 30 21:11:39.415605 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:39.415589    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b85e68b-c6a9-4362-a6c0-93af13623e67-istio-envoy\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4\" (UID: \"0b85e68b-c6a9-4362-a6c0-93af13623e67\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4"
Oct 30 21:11:39.415722 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:39.415702    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b85e68b-c6a9-4362-a6c0-93af13623e67-istio-podinfo\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4\" (UID: \"0b85e68b-c6a9-4362-a6c0-93af13623e67\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4"
Oct 30 21:11:39.469548 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:39.469434    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9m6b\" (UniqueName: \"kubernetes.io/projected/d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f-kube-api-access-v9m6b\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz\" (UID: \"d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz"
Oct 30 21:11:39.469548 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:39.469471    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f-istio-token\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz\" (UID: \"d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz"
Oct 30 21:11:39.469826 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:39.469573    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdc2n\" (UniqueName: \"kubernetes.io/projected/0b85e68b-c6a9-4362-a6c0-93af13623e67-kube-api-access-tdc2n\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4\" (UID: \"0b85e68b-c6a9-4362-a6c0-93af13623e67\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4"
Oct 30 21:11:39.470319 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:39.470294    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b85e68b-c6a9-4362-a6c0-93af13623e67-istio-token\") pod \"e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4\" (UID: \"0b85e68b-c6a9-4362-a6c0-93af13623e67\") " pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4"
Oct 30 21:11:39.532635 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:39.532600    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz"
Oct 30 21:11:39.581528 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:39.580821    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4"
Oct 30 21:11:39.978693 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:39.978655    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz"]
Oct 30 21:11:40.014196 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:11:40.014167    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd34f59c3_1aa3_4673_a9ac_4a00b5bfd23f.slice/crio-8dc38cde0b3ba97470fbe62b504ee8e6d60d150e2a4c1b59d1609416d0ed94c1 WatchSource:0}: Error finding container 8dc38cde0b3ba97470fbe62b504ee8e6d60d150e2a4c1b59d1609416d0ed94c1: Status 404 returned error can't find the container with id 8dc38cde0b3ba97470fbe62b504ee8e6d60d150e2a4c1b59d1609416d0ed94c1
Oct 30 21:11:40.017461 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:40.017426    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:11:40.017571 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:40.017501    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:11:40.017571 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:40.017531    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:11:40.039003 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:40.038972    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4"]
Oct 30 21:11:40.039238 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:11:40.038995    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b85e68b_c6a9_4362_a6c0_93af13623e67.slice/crio-444e694acdb929f8b45a6ea551b99d32c34b77297857885aa1ac73368a5cfee0 WatchSource:0}: Error finding container 444e694acdb929f8b45a6ea551b99d32c34b77297857885aa1ac73368a5cfee0: Status 404 returned error can't find the container with id 444e694acdb929f8b45a6ea551b99d32c34b77297857885aa1ac73368a5cfee0
Oct 30 21:11:40.043226 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:40.043192    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:11:40.043325 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:40.043267    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:11:40.043325 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:40.043298    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:11:40.742677 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:40.742639    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz" event={"ID":"d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f","Type":"ContainerStarted","Data":"cdd26ac7c03c95d666f43e2e53a5b8e81a441eb72f86201ea2b8f3da1d35b591"}
Oct 30 21:11:40.742963 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:40.742696    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz" event={"ID":"d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f","Type":"ContainerStarted","Data":"8dc38cde0b3ba97470fbe62b504ee8e6d60d150e2a4c1b59d1609416d0ed94c1"}
Oct 30 21:11:40.744544 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:40.744518    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4" event={"ID":"0b85e68b-c6a9-4362-a6c0-93af13623e67","Type":"ContainerStarted","Data":"a79c0d0d0481c3988fda6012603ca6917b6fb98b676c2ae18668cddb5d157a2b"}
Oct 30 21:11:40.744645 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:40.744568    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4" event={"ID":"0b85e68b-c6a9-4362-a6c0-93af13623e67","Type":"ContainerStarted","Data":"444e694acdb929f8b45a6ea551b99d32c34b77297857885aa1ac73368a5cfee0"}
Oct 30 21:11:40.863445 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:40.863386    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz" podStartSLOduration=1.863370193 podStartE2EDuration="1.863370193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:11:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:40.859854876 +0000 UTC m=+1983.924631268" watchObservedRunningTime="2026-10-30 21:11:40.863370193 +0000 UTC m=+1983.928146590"
Oct 30 21:11:40.996874 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:40.989771    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4" podStartSLOduration=1.9897441740000001 podStartE2EDuration="1.989744174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:11:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:40.98036203 +0000 UTC m=+1984.045138429" watchObservedRunningTime="2026-10-30 21:11:40.989744174 +0000 UTC m=+1984.054520575"
Oct 30 21:11:41.538816 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:41.536173    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz"
Oct 30 21:11:41.559816 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:41.542240    5713 patch_prober.go:29] interesting pod/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Oct 30 21:11:41.559816 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:41.542323    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz" podUID="d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Oct 30 21:11:41.583815 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:41.583162    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4"
Oct 30 21:11:41.587814 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:41.585934    5713 patch_prober.go:29] interesting pod/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Oct 30 21:11:41.587814 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:41.585994    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4" podUID="0b85e68b-c6a9-4362-a6c0-93af13623e67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Oct 30 21:11:42.540818 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:42.540270    5713 patch_prober.go:29] interesting pod/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Oct 30 21:11:42.540818 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:42.540334    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz" podUID="d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Oct 30 21:11:42.586815 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:42.583282    5713 patch_prober.go:29] interesting pod/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Oct 30 21:11:42.586815 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:42.583360    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4" podUID="0b85e68b-c6a9-4362-a6c0-93af13623e67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Oct 30 21:11:43.533432 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:43.533390    5713 patch_prober.go:29] interesting pod/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Oct 30 21:11:43.533620 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:43.533447    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz" podUID="d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Oct 30 21:11:43.585813 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:43.585224    5713 patch_prober.go:29] interesting pod/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Oct 30 21:11:43.585813 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:43.585289    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4" podUID="0b85e68b-c6a9-4362-a6c0-93af13623e67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Oct 30 21:11:44.544501 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:44.544463    5713 patch_prober.go:29] interesting pod/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Oct 30 21:11:44.544829 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:44.544773    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz" podUID="d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Oct 30 21:11:44.588253 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:44.588218    5713 patch_prober.go:29] interesting pod/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Oct 30 21:11:44.588876 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:44.588840    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4" podUID="0b85e68b-c6a9-4362-a6c0-93af13623e67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Oct 30 21:11:45.533766 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:45.533726    5713 patch_prober.go:29] interesting pod/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Oct 30 21:11:45.534020 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:45.533780    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz" podUID="d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Oct 30 21:11:45.593813 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:45.585126    5713 patch_prober.go:29] interesting pod/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Oct 30 21:11:45.593813 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:45.585189    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4" podUID="0b85e68b-c6a9-4362-a6c0-93af13623e67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Oct 30 21:11:46.533088 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:46.533007    5713 patch_prober.go:29] interesting pod/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Oct 30 21:11:46.533279 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:46.533073    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz" podUID="d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Oct 30 21:11:46.595496 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:46.595460    5713 patch_prober.go:29] interesting pod/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Oct 30 21:11:46.596258 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:46.596224    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4" podUID="0b85e68b-c6a9-4362-a6c0-93af13623e67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Oct 30 21:11:47.678066 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:47.675455    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz"
Oct 30 21:11:47.678066 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:47.677033    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz"
Oct 30 21:11:47.678989 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:47.678960    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz"
Oct 30 21:11:48.029898 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:48.022145    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4"
Oct 30 21:11:48.029898 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:48.023745    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4"
Oct 30 21:11:48.029898 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:48.023997    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4"
Oct 30 21:11:51.051523 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:51.051489    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-388e5b76-4tqxt"]
Oct 30 21:11:51.102619 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:51.102582    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-388e5b76-4tqxt"]
Oct 30 21:11:52.850295 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:11:52.850257    5713 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad23561f_a24f_4c13_9df7_0ecce2596034.slice/crio-conmon-4157e3bb57ce1a7d4978d8fa5903723c81ad1f43ca0a805cd809861f4e54edcc.scope\": RecentStats: unable to find data in memory cache], [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@7-32784-0.service\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad23561f_a24f_4c13_9df7_0ecce2596034.slice/crio-4157e3bb57ce1a7d4978d8fa5903723c81ad1f43ca0a805cd809861f4e54edcc.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 21:11:53.066835 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:53.066729    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cbf9725-307f-4b4e-8119-dbd41f83be2a" path="/var/lib/kubelet/pods/2cbf9725-307f-4b4e-8119-dbd41f83be2a/volumes"
Oct 30 21:11:53.155198 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:53.155168    5713 generic.go:356] "Generic (PLEG): container finished" podID="ad23561f-a24f-4c13-9df7-0ecce2596034" containerID="4157e3bb57ce1a7d4978d8fa5903723c81ad1f43ca0a805cd809861f4e54edcc" exitCode=0
Oct 30 21:11:53.155374 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:53.155243    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk" event={"ID":"ad23561f-a24f-4c13-9df7-0ecce2596034","Type":"ContainerDied","Data":"4157e3bb57ce1a7d4978d8fa5903723c81ad1f43ca0a805cd809861f4e54edcc"}
Oct 30 21:11:53.155698 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:53.155674    5713 scope.go:122] "RemoveContainer" containerID="4157e3bb57ce1a7d4978d8fa5903723c81ad1f43ca0a805cd809861f4e54edcc"
Oct 30 21:11:53.157369 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:53.157341    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:11:53.157462 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:53.157413    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:11:53.157462 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:53.157445    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:11:54.168020 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:54.167962    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk" event={"ID":"ad23561f-a24f-4c13-9df7-0ecce2596034","Type":"ContainerStarted","Data":"f43f383d2ee675de7ecee69c246a89878d85c124ccb234cbee9817a50f1fba8d"}
Oct 30 21:11:54.390870 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:54.390838    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk"
Oct 30 21:11:54.392579 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:54.392549    5713 patch_prober.go:29] interesting pod/e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Oct 30 21:11:54.392719 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:54.392610    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk" podUID="ad23561f-a24f-4c13-9df7-0ecce2596034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Oct 30 21:11:54.486118 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:11:54.486080    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:11:55.391543 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:55.391512    5713 patch_prober.go:29] interesting pod/e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Oct 30 21:11:55.392004 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:55.391577    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk" podUID="ad23561f-a24f-4c13-9df7-0ecce2596034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Oct 30 21:11:56.014489 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:56.014457    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk"
Oct 30 21:11:56.390845 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:56.390710    5713 patch_prober.go:29] interesting pod/e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Oct 30 21:11:56.390845 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:56.390807    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk" podUID="ad23561f-a24f-4c13-9df7-0ecce2596034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Oct 30 21:11:57.391221 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:57.391181    5713 patch_prober.go:29] interesting pod/e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Oct 30 21:11:57.391712 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:57.391251    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk" podUID="ad23561f-a24f-4c13-9df7-0ecce2596034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Oct 30 21:11:58.390898 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:58.390862    5713 patch_prober.go:29] interesting pod/e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Oct 30 21:11:58.391091 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:58.390916    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk" podUID="ad23561f-a24f-4c13-9df7-0ecce2596034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Oct 30 21:11:59.433537 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:59.433499    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk"
Oct 30 21:11:59.435170 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:11:59.435148    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk"
Oct 30 21:12:06.369298 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:06.369258    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h"]
Oct 30 21:12:06.369773 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:06.369562    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h" podUID="66df6c04-8987-4f07-9fab-4e80cf71f6a2" containerName="istio-proxy" containerID="cri-o://74a6e9e8f8edbf94c47086069fa1c7a56bd97f78cdff54c04ac77192491e2e6c" gracePeriod=30
Oct 30 21:12:07.281632 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:07.281601    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59"]
Oct 30 21:12:07.281885 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:07.281840    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59" podUID="eb252301-e703-4e46-b569-87b1039211d9" containerName="istio-proxy" containerID="cri-o://76ecd8330c15b09ee14ec2cc7140992dd8e7ec4896965380dc97f8943ea1a64a" gracePeriod=30
Oct 30 21:12:07.286309 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:07.286282    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb"]
Oct 30 21:12:07.286524 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:07.286501    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb" podUID="2c7e4408-b25d-44fa-9587-3a1288a8fc87" containerName="istio-proxy" containerID="cri-o://148708a70cf2d226013537be1c071f074cc1d2ac70302684097f722729bb6293" gracePeriod=30
Oct 30 21:12:07.302006 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:07.301970    5713 patch_prober.go:29] interesting pod/e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:12:07.302154 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:07.302038    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h" podUID="66df6c04-8987-4f07-9fab-4e80cf71f6a2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:12:07.351853 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:07.351815    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj"]
Oct 30 21:12:07.352314 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:07.352262    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj" podUID="4b4b9def-33e5-4d81-a159-1389d71ced98" containerName="istio-proxy" containerID="cri-o://f4a8e6c8b5d4bc22f229aea1dd9f16222b97e03f0028e172d56130e7d53ffe48" gracePeriod=30
Oct 30 21:12:11.704092 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:11.704059    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h"
Oct 30 21:12:11.745586 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:11.745549    5713 patch_prober.go:29] interesting pod/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:12:11.745752 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:11.745608    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59" podUID="eb252301-e703-4e46-b569-87b1039211d9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:12:11.799110 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:11.799075    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66df6c04-8987-4f07-9fab-4e80cf71f6a2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66df6c04-8987-4f07-9fab-4e80cf71f6a2-workload-socket\") pod \"66df6c04-8987-4f07-9fab-4e80cf71f6a2\" (UID: \"66df6c04-8987-4f07-9fab-4e80cf71f6a2\") "
Oct 30 21:12:11.799256 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:11.799160    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66df6c04-8987-4f07-9fab-4e80cf71f6a2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66df6c04-8987-4f07-9fab-4e80cf71f6a2-workload-certs\") pod \"66df6c04-8987-4f07-9fab-4e80cf71f6a2\" (UID: \"66df6c04-8987-4f07-9fab-4e80cf71f6a2\") "
Oct 30 21:12:11.799256 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:11.799195    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66df6c04-8987-4f07-9fab-4e80cf71f6a2-istio-token\" (UniqueName: \"kubernetes.io/projected/66df6c04-8987-4f07-9fab-4e80cf71f6a2-istio-token\") pod \"66df6c04-8987-4f07-9fab-4e80cf71f6a2\" (UID: \"66df6c04-8987-4f07-9fab-4e80cf71f6a2\") "
Oct 30 21:12:11.799347 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:11.799286    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66df6c04-8987-4f07-9fab-4e80cf71f6a2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66df6c04-8987-4f07-9fab-4e80cf71f6a2-istio-data\") pod \"66df6c04-8987-4f07-9fab-4e80cf71f6a2\" (UID: \"66df6c04-8987-4f07-9fab-4e80cf71f6a2\") "
Oct 30 21:12:11.799402 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:11.799369    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66df6c04-8987-4f07-9fab-4e80cf71f6a2-workload-socket" pod "66df6c04-8987-4f07-9fab-4e80cf71f6a2" (UID: "66df6c04-8987-4f07-9fab-4e80cf71f6a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:12:11.799525 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:11.799499    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66df6c04-8987-4f07-9fab-4e80cf71f6a2-workload-certs" pod "66df6c04-8987-4f07-9fab-4e80cf71f6a2" (UID: "66df6c04-8987-4f07-9fab-4e80cf71f6a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:12:11.799567 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:11.799541    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66df6c04-8987-4f07-9fab-4e80cf71f6a2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66df6c04-8987-4f07-9fab-4e80cf71f6a2-credential-socket\") pod \"66df6c04-8987-4f07-9fab-4e80cf71f6a2\" (UID: \"66df6c04-8987-4f07-9fab-4e80cf71f6a2\") "
Oct 30 21:12:11.799567 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:11.799536    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66df6c04-8987-4f07-9fab-4e80cf71f6a2-istio-data" pod "66df6c04-8987-4f07-9fab-4e80cf71f6a2" (UID: "66df6c04-8987-4f07-9fab-4e80cf71f6a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:12:11.799654 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:11.799566    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66df6c04-8987-4f07-9fab-4e80cf71f6a2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66df6c04-8987-4f07-9fab-4e80cf71f6a2-istiod-ca-cert\") pod \"66df6c04-8987-4f07-9fab-4e80cf71f6a2\" (UID: \"66df6c04-8987-4f07-9fab-4e80cf71f6a2\") "
Oct 30 21:12:11.799654 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:11.799648    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/66df6c04-8987-4f07-9fab-4e80cf71f6a2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66df6c04-8987-4f07-9fab-4e80cf71f6a2-istio-podinfo\") pod \"66df6c04-8987-4f07-9fab-4e80cf71f6a2\" (UID: \"66df6c04-8987-4f07-9fab-4e80cf71f6a2\") "
Oct 30 21:12:11.799766 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:11.799676    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66df6c04-8987-4f07-9fab-4e80cf71f6a2-kube-api-access-vw2k5\" (UniqueName: \"kubernetes.io/projected/66df6c04-8987-4f07-9fab-4e80cf71f6a2-kube-api-access-vw2k5\") pod \"66df6c04-8987-4f07-9fab-4e80cf71f6a2\" (UID: \"66df6c04-8987-4f07-9fab-4e80cf71f6a2\") "
Oct 30 21:12:11.799766 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:11.799704    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66df6c04-8987-4f07-9fab-4e80cf71f6a2-credential-socket" pod "66df6c04-8987-4f07-9fab-4e80cf71f6a2" (UID: "66df6c04-8987-4f07-9fab-4e80cf71f6a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:12:11.799766 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:11.799720    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66df6c04-8987-4f07-9fab-4e80cf71f6a2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66df6c04-8987-4f07-9fab-4e80cf71f6a2-istio-envoy\") pod \"66df6c04-8987-4f07-9fab-4e80cf71f6a2\" (UID: \"66df6c04-8987-4f07-9fab-4e80cf71f6a2\") "
Oct 30 21:12:11.799969 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:11.799900    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66df6c04-8987-4f07-9fab-4e80cf71f6a2-istiod-ca-cert" pod "66df6c04-8987-4f07-9fab-4e80cf71f6a2" (UID: "66df6c04-8987-4f07-9fab-4e80cf71f6a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:12:11.800720 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:11.800695    5713 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66df6c04-8987-4f07-9fab-4e80cf71f6a2-credential-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:11.800898 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:11.800727    5713 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66df6c04-8987-4f07-9fab-4e80cf71f6a2-istiod-ca-cert\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:11.800898 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:11.800759    5713 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66df6c04-8987-4f07-9fab-4e80cf71f6a2-workload-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:11.800898 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:11.800774    5713 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66df6c04-8987-4f07-9fab-4e80cf71f6a2-workload-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:11.800898 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:11.800808    5713 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66df6c04-8987-4f07-9fab-4e80cf71f6a2-istio-data\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:11.801894 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:11.801869    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66df6c04-8987-4f07-9fab-4e80cf71f6a2-istio-token" pod "66df6c04-8987-4f07-9fab-4e80cf71f6a2" (UID: "66df6c04-8987-4f07-9fab-4e80cf71f6a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:12:11.802118 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:11.802100    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/66df6c04-8987-4f07-9fab-4e80cf71f6a2-istio-podinfo" pod "66df6c04-8987-4f07-9fab-4e80cf71f6a2" (UID: "66df6c04-8987-4f07-9fab-4e80cf71f6a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:12:11.802180 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:11.802132    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66df6c04-8987-4f07-9fab-4e80cf71f6a2-kube-api-access-vw2k5" pod "66df6c04-8987-4f07-9fab-4e80cf71f6a2" (UID: "66df6c04-8987-4f07-9fab-4e80cf71f6a2"). InnerVolumeSpecName "kube-api-access-vw2k5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:12:11.802333 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:11.802315    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66df6c04-8987-4f07-9fab-4e80cf71f6a2-istio-envoy" pod "66df6c04-8987-4f07-9fab-4e80cf71f6a2" (UID: "66df6c04-8987-4f07-9fab-4e80cf71f6a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:12:11.902895 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:11.902862    5713 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66df6c04-8987-4f07-9fab-4e80cf71f6a2-istio-token\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:11.902895 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:11.902892    5713 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66df6c04-8987-4f07-9fab-4e80cf71f6a2-istio-podinfo\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:11.903087 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:11.902903    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw2k5\" (UniqueName: \"kubernetes.io/projected/66df6c04-8987-4f07-9fab-4e80cf71f6a2-kube-api-access-vw2k5\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:11.903087 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:11.902918    5713 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66df6c04-8987-4f07-9fab-4e80cf71f6a2-istio-envoy\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:12.297094 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.297062    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef7e2d-rqhqc"]
Oct 30 21:12:12.298324 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.298297    5713 cpu_manager.go:414] "containerMap: removing container" podUID="66df6c04-8987-4f07-9fab-4e80cf71f6a2" containerName="istio-proxy"
Oct 30 21:12:12.298324 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.298325    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66df6c04-8987-4f07-9fab-4e80cf71f6a2" containerName="istio-proxy"
Oct 30 21:12:12.298552 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.298540    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="66df6c04-8987-4f07-9fab-4e80cf71f6a2" containerName="istio-proxy"
Oct 30 21:12:12.303084 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.303062    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef7e2d-rqhqc"
Oct 30 21:12:12.308464 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.308438    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b61bd136-4b2d-4ab1-b263-6531234de46b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ef7e2d-rqhqc\" (UID: \"b61bd136-4b2d-4ab1-b263-6531234de46b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef7e2d-rqhqc"
Oct 30 21:12:12.308667 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.308645    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnp27\" (UniqueName: \"kubernetes.io/projected/b61bd136-4b2d-4ab1-b263-6531234de46b-kube-api-access-bnp27\") pod \"maas-api-revoke-keys-e2e-api-a-ef7e2d-rqhqc\" (UID: \"b61bd136-4b2d-4ab1-b263-6531234de46b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef7e2d-rqhqc"
Oct 30 21:12:12.308719 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.308708    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b61bd136-4b2d-4ab1-b263-6531234de46b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ef7e2d-rqhqc\" (UID: \"b61bd136-4b2d-4ab1-b263-6531234de46b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef7e2d-rqhqc"
Oct 30 21:12:12.323186 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.323159    5713 generic.go:356] "Generic (PLEG): container finished" podID="66df6c04-8987-4f07-9fab-4e80cf71f6a2" containerID="74a6e9e8f8edbf94c47086069fa1c7a56bd97f78cdff54c04ac77192491e2e6c" exitCode=0
Oct 30 21:12:12.323300 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.323204    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h"
Oct 30 21:12:12.323300 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.323235    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h" event={"ID":"66df6c04-8987-4f07-9fab-4e80cf71f6a2","Type":"ContainerDied","Data":"74a6e9e8f8edbf94c47086069fa1c7a56bd97f78cdff54c04ac77192491e2e6c"}
Oct 30 21:12:12.323300 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.323284    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h" event={"ID":"66df6c04-8987-4f07-9fab-4e80cf71f6a2","Type":"ContainerDied","Data":"cc0a69a853eaf8088d66fe777ab0b8771686120181ed0176992f26a9dfd18faf"}
Oct 30 21:12:12.323454 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.323304    5713 scope.go:122] "RemoveContainer" containerID="74a6e9e8f8edbf94c47086069fa1c7a56bd97f78cdff54c04ac77192491e2e6c"
Oct 30 21:12:12.357285 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.355702    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:12:12.357285 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.355769    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:12:12.357285 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.355941    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:12:12.357285 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.355990    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:12:12.360657 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.360631    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef7e2d-rqhqc"]
Oct 30 21:12:12.410554 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.410529    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnp27\" (UniqueName: \"kubernetes.io/projected/b61bd136-4b2d-4ab1-b263-6531234de46b-kube-api-access-bnp27\") pod \"maas-api-revoke-keys-e2e-api-a-ef7e2d-rqhqc\" (UID: \"b61bd136-4b2d-4ab1-b263-6531234de46b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef7e2d-rqhqc"
Oct 30 21:12:12.410668 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.410601    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b61bd136-4b2d-4ab1-b263-6531234de46b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ef7e2d-rqhqc\" (UID: \"b61bd136-4b2d-4ab1-b263-6531234de46b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef7e2d-rqhqc"
Oct 30 21:12:12.410732 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.410679    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b61bd136-4b2d-4ab1-b263-6531234de46b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ef7e2d-rqhqc\" (UID: \"b61bd136-4b2d-4ab1-b263-6531234de46b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef7e2d-rqhqc"
Oct 30 21:12:12.411386 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.411317    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b61bd136-4b2d-4ab1-b263-6531234de46b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ef7e2d-rqhqc\" (UID: \"b61bd136-4b2d-4ab1-b263-6531234de46b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef7e2d-rqhqc"
Oct 30 21:12:12.413061 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.413038    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b61bd136-4b2d-4ab1-b263-6531234de46b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ef7e2d-rqhqc\" (UID: \"b61bd136-4b2d-4ab1-b263-6531234de46b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef7e2d-rqhqc"
Oct 30 21:12:12.474649 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.468646    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnp27\" (UniqueName: \"kubernetes.io/projected/b61bd136-4b2d-4ab1-b263-6531234de46b-kube-api-access-bnp27\") pod \"maas-api-revoke-keys-e2e-api-a-ef7e2d-rqhqc\" (UID: \"b61bd136-4b2d-4ab1-b263-6531234de46b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef7e2d-rqhqc"
Oct 30 21:12:12.474950 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.474907    5713 scope.go:122] "RemoveContainer" containerID="74a6e9e8f8edbf94c47086069fa1c7a56bd97f78cdff54c04ac77192491e2e6c"
Oct 30 21:12:12.475339 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:12:12.475313    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74a6e9e8f8edbf94c47086069fa1c7a56bd97f78cdff54c04ac77192491e2e6c\": container with ID starting with 74a6e9e8f8edbf94c47086069fa1c7a56bd97f78cdff54c04ac77192491e2e6c not found: ID does not exist" containerID="74a6e9e8f8edbf94c47086069fa1c7a56bd97f78cdff54c04ac77192491e2e6c"
Oct 30 21:12:12.475416 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.475353    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74a6e9e8f8edbf94c47086069fa1c7a56bd97f78cdff54c04ac77192491e2e6c"} err="failed to get container status \"74a6e9e8f8edbf94c47086069fa1c7a56bd97f78cdff54c04ac77192491e2e6c\": rpc error: code = NotFound desc = could not find container \"74a6e9e8f8edbf94c47086069fa1c7a56bd97f78cdff54c04ac77192491e2e6c\": container with ID starting with 74a6e9e8f8edbf94c47086069fa1c7a56bd97f78cdff54c04ac77192491e2e6c not found: ID does not exist"
Oct 30 21:12:12.511305 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.511143    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef7e2d-rqhqc"
Oct 30 21:12:12.623544 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.622481    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h"]
Oct 30 21:12:12.631662 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.631630    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-f9f277-openshift-default-869d58fb77-8ff2h"]
Oct 30 21:12:12.645112 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.645086    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-dc6125-4n8cr"]
Oct 30 21:12:12.662570 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.660042    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-dc6125-4n8cr"
Oct 30 21:12:12.668590 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.668559    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59"
Oct 30 21:12:12.716071 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.716046    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb252301-e703-4e46-b569-87b1039211d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb252301-e703-4e46-b569-87b1039211d9-workload-certs\") pod \"eb252301-e703-4e46-b569-87b1039211d9\" (UID: \"eb252301-e703-4e46-b569-87b1039211d9\") "
Oct 30 21:12:12.716482 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.716171    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb252301-e703-4e46-b569-87b1039211d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb252301-e703-4e46-b569-87b1039211d9-istio-envoy\") pod \"eb252301-e703-4e46-b569-87b1039211d9\" (UID: \"eb252301-e703-4e46-b569-87b1039211d9\") "
Oct 30 21:12:12.716482 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.716236    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eb252301-e703-4e46-b569-87b1039211d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb252301-e703-4e46-b569-87b1039211d9-istio-podinfo\") pod \"eb252301-e703-4e46-b569-87b1039211d9\" (UID: \"eb252301-e703-4e46-b569-87b1039211d9\") "
Oct 30 21:12:12.716482 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.716344    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb252301-e703-4e46-b569-87b1039211d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb252301-e703-4e46-b569-87b1039211d9-istiod-ca-cert\") pod \"eb252301-e703-4e46-b569-87b1039211d9\" (UID: \"eb252301-e703-4e46-b569-87b1039211d9\") "
Oct 30 21:12:12.716482 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.716415    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb252301-e703-4e46-b569-87b1039211d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb252301-e703-4e46-b569-87b1039211d9-istio-data\") pod \"eb252301-e703-4e46-b569-87b1039211d9\" (UID: \"eb252301-e703-4e46-b569-87b1039211d9\") "
Oct 30 21:12:12.716754 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.716489    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb252301-e703-4e46-b569-87b1039211d9-istio-token\" (UniqueName: \"kubernetes.io/projected/eb252301-e703-4e46-b569-87b1039211d9-istio-token\") pod \"eb252301-e703-4e46-b569-87b1039211d9\" (UID: \"eb252301-e703-4e46-b569-87b1039211d9\") "
Oct 30 21:12:12.716754 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.716563    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb252301-e703-4e46-b569-87b1039211d9-kube-api-access-4q65n\" (UniqueName: \"kubernetes.io/projected/eb252301-e703-4e46-b569-87b1039211d9-kube-api-access-4q65n\") pod \"eb252301-e703-4e46-b569-87b1039211d9\" (UID: \"eb252301-e703-4e46-b569-87b1039211d9\") "
Oct 30 21:12:12.716754 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.716659    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb252301-e703-4e46-b569-87b1039211d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb252301-e703-4e46-b569-87b1039211d9-credential-socket\") pod \"eb252301-e703-4e46-b569-87b1039211d9\" (UID: \"eb252301-e703-4e46-b569-87b1039211d9\") "
Oct 30 21:12:12.716754 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.716723    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb252301-e703-4e46-b569-87b1039211d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb252301-e703-4e46-b569-87b1039211d9-workload-socket\") pod \"eb252301-e703-4e46-b569-87b1039211d9\" (UID: \"eb252301-e703-4e46-b569-87b1039211d9\") "
Oct 30 21:12:12.717299 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.717275    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/089ae08c-aa75-4c9b-ab51-70684e9ba0a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-dc6125-4n8cr\" (UID: \"089ae08c-aa75-4c9b-ab51-70684e9ba0a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-dc6125-4n8cr"
Oct 30 21:12:12.717416 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.717400    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/089ae08c-aa75-4c9b-ab51-70684e9ba0a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-dc6125-4n8cr\" (UID: \"089ae08c-aa75-4c9b-ab51-70684e9ba0a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-dc6125-4n8cr"
Oct 30 21:12:12.717782 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.717757    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtcs6\" (UniqueName: \"kubernetes.io/projected/089ae08c-aa75-4c9b-ab51-70684e9ba0a1-kube-api-access-mtcs6\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-dc6125-4n8cr\" (UID: \"089ae08c-aa75-4c9b-ab51-70684e9ba0a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-dc6125-4n8cr"
Oct 30 21:12:12.719415 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.719389    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb252301-e703-4e46-b569-87b1039211d9-workload-socket" pod "eb252301-e703-4e46-b569-87b1039211d9" (UID: "eb252301-e703-4e46-b569-87b1039211d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:12:12.719606 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.719580    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb252301-e703-4e46-b569-87b1039211d9-credential-socket" pod "eb252301-e703-4e46-b569-87b1039211d9" (UID: "eb252301-e703-4e46-b569-87b1039211d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:12:12.719806 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.719769    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb252301-e703-4e46-b569-87b1039211d9-istio-data" pod "eb252301-e703-4e46-b569-87b1039211d9" (UID: "eb252301-e703-4e46-b569-87b1039211d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:12:12.720033 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.720004    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb252301-e703-4e46-b569-87b1039211d9-istiod-ca-cert" pod "eb252301-e703-4e46-b569-87b1039211d9" (UID: "eb252301-e703-4e46-b569-87b1039211d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:12:12.720859 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.720763    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb252301-e703-4e46-b569-87b1039211d9-workload-certs" pod "eb252301-e703-4e46-b569-87b1039211d9" (UID: "eb252301-e703-4e46-b569-87b1039211d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:12:12.721327 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.721302    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb252301-e703-4e46-b569-87b1039211d9-kube-api-access-4q65n" pod "eb252301-e703-4e46-b569-87b1039211d9" (UID: "eb252301-e703-4e46-b569-87b1039211d9"). InnerVolumeSpecName "kube-api-access-4q65n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:12:12.727947 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.727925    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb"
Oct 30 21:12:12.729233 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.729202    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb252301-e703-4e46-b569-87b1039211d9-istio-token" pod "eb252301-e703-4e46-b569-87b1039211d9" (UID: "eb252301-e703-4e46-b569-87b1039211d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:12:12.731592 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.731562    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb252301-e703-4e46-b569-87b1039211d9-istio-envoy" pod "eb252301-e703-4e46-b569-87b1039211d9" (UID: "eb252301-e703-4e46-b569-87b1039211d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:12:12.733554 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.733528    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb252301-e703-4e46-b569-87b1039211d9-istio-podinfo" pod "eb252301-e703-4e46-b569-87b1039211d9" (UID: "eb252301-e703-4e46-b569-87b1039211d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:12:12.739464 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.739439    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-dc6125-4n8cr"]
Oct 30 21:12:12.745286 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.745262    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj"
Oct 30 21:12:12.819346 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.819310    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c7e4408-b25d-44fa-9587-3a1288a8fc87-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c7e4408-b25d-44fa-9587-3a1288a8fc87-istio-envoy\") pod \"2c7e4408-b25d-44fa-9587-3a1288a8fc87\" (UID: \"2c7e4408-b25d-44fa-9587-3a1288a8fc87\") "
Oct 30 21:12:12.819535 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.819365    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c7e4408-b25d-44fa-9587-3a1288a8fc87-istio-token\" (UniqueName: \"kubernetes.io/projected/2c7e4408-b25d-44fa-9587-3a1288a8fc87-istio-token\") pod \"2c7e4408-b25d-44fa-9587-3a1288a8fc87\" (UID: \"2c7e4408-b25d-44fa-9587-3a1288a8fc87\") "
Oct 30 21:12:12.819535 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.819401    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b4b9def-33e5-4d81-a159-1389d71ced98-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b4b9def-33e5-4d81-a159-1389d71ced98-workload-socket\") pod \"4b4b9def-33e5-4d81-a159-1389d71ced98\" (UID: \"4b4b9def-33e5-4d81-a159-1389d71ced98\") "
Oct 30 21:12:12.819535 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.819497    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b4b9def-33e5-4d81-a159-1389d71ced98-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b4b9def-33e5-4d81-a159-1389d71ced98-istio-data\") pod \"4b4b9def-33e5-4d81-a159-1389d71ced98\" (UID: \"4b4b9def-33e5-4d81-a159-1389d71ced98\") "
Oct 30 21:12:12.819712 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.819550    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b4b9def-33e5-4d81-a159-1389d71ced98-kube-api-access-kp2jw\" (UniqueName: \"kubernetes.io/projected/4b4b9def-33e5-4d81-a159-1389d71ced98-kube-api-access-kp2jw\") pod \"4b4b9def-33e5-4d81-a159-1389d71ced98\" (UID: \"4b4b9def-33e5-4d81-a159-1389d71ced98\") "
Oct 30 21:12:12.819712 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.819606    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b4b9def-33e5-4d81-a159-1389d71ced98-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b4b9def-33e5-4d81-a159-1389d71ced98-istio-envoy\") pod \"4b4b9def-33e5-4d81-a159-1389d71ced98\" (UID: \"4b4b9def-33e5-4d81-a159-1389d71ced98\") "
Oct 30 21:12:12.819712 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.819640    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b4b9def-33e5-4d81-a159-1389d71ced98-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b4b9def-33e5-4d81-a159-1389d71ced98-credential-socket\") pod \"4b4b9def-33e5-4d81-a159-1389d71ced98\" (UID: \"4b4b9def-33e5-4d81-a159-1389d71ced98\") "
Oct 30 21:12:12.819712 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.819689    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4b4b9def-33e5-4d81-a159-1389d71ced98-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b4b9def-33e5-4d81-a159-1389d71ced98-istio-podinfo\") pod \"4b4b9def-33e5-4d81-a159-1389d71ced98\" (UID: \"4b4b9def-33e5-4d81-a159-1389d71ced98\") "
Oct 30 21:12:12.819948 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.819718    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c7e4408-b25d-44fa-9587-3a1288a8fc87-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c7e4408-b25d-44fa-9587-3a1288a8fc87-istiod-ca-cert\") pod \"2c7e4408-b25d-44fa-9587-3a1288a8fc87\" (UID: \"2c7e4408-b25d-44fa-9587-3a1288a8fc87\") "
Oct 30 21:12:12.819948 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.819782    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b4b9def-33e5-4d81-a159-1389d71ced98-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b4b9def-33e5-4d81-a159-1389d71ced98-istiod-ca-cert\") pod \"4b4b9def-33e5-4d81-a159-1389d71ced98\" (UID: \"4b4b9def-33e5-4d81-a159-1389d71ced98\") "
Oct 30 21:12:12.819948 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.819847    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c7e4408-b25d-44fa-9587-3a1288a8fc87-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c7e4408-b25d-44fa-9587-3a1288a8fc87-workload-certs\") pod \"2c7e4408-b25d-44fa-9587-3a1288a8fc87\" (UID: \"2c7e4408-b25d-44fa-9587-3a1288a8fc87\") "
Oct 30 21:12:12.819948 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.819889    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c7e4408-b25d-44fa-9587-3a1288a8fc87-kube-api-access-8fcm2\" (UniqueName: \"kubernetes.io/projected/2c7e4408-b25d-44fa-9587-3a1288a8fc87-kube-api-access-8fcm2\") pod \"2c7e4408-b25d-44fa-9587-3a1288a8fc87\" (UID: \"2c7e4408-b25d-44fa-9587-3a1288a8fc87\") "
Oct 30 21:12:12.819948 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.819937    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b4b9def-33e5-4d81-a159-1389d71ced98-istio-token\" (UniqueName: \"kubernetes.io/projected/4b4b9def-33e5-4d81-a159-1389d71ced98-istio-token\") pod \"4b4b9def-33e5-4d81-a159-1389d71ced98\" (UID: \"4b4b9def-33e5-4d81-a159-1389d71ced98\") "
Oct 30 21:12:12.820286 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.820058    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c7e4408-b25d-44fa-9587-3a1288a8fc87-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c7e4408-b25d-44fa-9587-3a1288a8fc87-istio-data\") pod \"2c7e4408-b25d-44fa-9587-3a1288a8fc87\" (UID: \"2c7e4408-b25d-44fa-9587-3a1288a8fc87\") "
Oct 30 21:12:12.820286 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.820082    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2c7e4408-b25d-44fa-9587-3a1288a8fc87-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c7e4408-b25d-44fa-9587-3a1288a8fc87-istio-podinfo\") pod \"2c7e4408-b25d-44fa-9587-3a1288a8fc87\" (UID: \"2c7e4408-b25d-44fa-9587-3a1288a8fc87\") "
Oct 30 21:12:12.820286 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.820125    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b4b9def-33e5-4d81-a159-1389d71ced98-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b4b9def-33e5-4d81-a159-1389d71ced98-workload-certs\") pod \"4b4b9def-33e5-4d81-a159-1389d71ced98\" (UID: \"4b4b9def-33e5-4d81-a159-1389d71ced98\") "
Oct 30 21:12:12.820286 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.820183    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c7e4408-b25d-44fa-9587-3a1288a8fc87-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c7e4408-b25d-44fa-9587-3a1288a8fc87-workload-socket\") pod \"2c7e4408-b25d-44fa-9587-3a1288a8fc87\" (UID: \"2c7e4408-b25d-44fa-9587-3a1288a8fc87\") "
Oct 30 21:12:12.822856 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.819710    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b4b9def-33e5-4d81-a159-1389d71ced98-workload-socket" pod "4b4b9def-33e5-4d81-a159-1389d71ced98" (UID: "4b4b9def-33e5-4d81-a159-1389d71ced98"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:12:12.822984 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.819753    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b4b9def-33e5-4d81-a159-1389d71ced98-istio-data" pod "4b4b9def-33e5-4d81-a159-1389d71ced98" (UID: "4b4b9def-33e5-4d81-a159-1389d71ced98"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:12:12.822984 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.820491    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c7e4408-b25d-44fa-9587-3a1288a8fc87-workload-socket" pod "2c7e4408-b25d-44fa-9587-3a1288a8fc87" (UID: "2c7e4408-b25d-44fa-9587-3a1288a8fc87"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:12:12.822984 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.820758    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b4b9def-33e5-4d81-a159-1389d71ced98-credential-socket" pod "4b4b9def-33e5-4d81-a159-1389d71ced98" (UID: "4b4b9def-33e5-4d81-a159-1389d71ced98"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:12:12.822984 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.822342    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c7e4408-b25d-44fa-9587-3a1288a8fc87-istio-data" pod "2c7e4408-b25d-44fa-9587-3a1288a8fc87" (UID: "2c7e4408-b25d-44fa-9587-3a1288a8fc87"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:12:12.822984 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.822442    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c7e4408-b25d-44fa-9587-3a1288a8fc87-istio-token" pod "2c7e4408-b25d-44fa-9587-3a1288a8fc87" (UID: "2c7e4408-b25d-44fa-9587-3a1288a8fc87"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:12:12.822984 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.822666    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b4b9def-33e5-4d81-a159-1389d71ced98-kube-api-access-kp2jw" pod "4b4b9def-33e5-4d81-a159-1389d71ced98" (UID: "4b4b9def-33e5-4d81-a159-1389d71ced98"). InnerVolumeSpecName "kube-api-access-kp2jw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:12:12.822984 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.822739    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b4b9def-33e5-4d81-a159-1389d71ced98-istiod-ca-cert" pod "4b4b9def-33e5-4d81-a159-1389d71ced98" (UID: "4b4b9def-33e5-4d81-a159-1389d71ced98"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:12:12.822984 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.822937    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b4b9def-33e5-4d81-a159-1389d71ced98-workload-certs" pod "4b4b9def-33e5-4d81-a159-1389d71ced98" (UID: "4b4b9def-33e5-4d81-a159-1389d71ced98"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:12:12.822984 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.822781    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b4b9def-33e5-4d81-a159-1389d71ced98-istio-token" pod "4b4b9def-33e5-4d81-a159-1389d71ced98" (UID: "4b4b9def-33e5-4d81-a159-1389d71ced98"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:12:12.823537 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.823029    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c7e4408-b25d-44fa-9587-3a1288a8fc87-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c7e4408-b25d-44fa-9587-3a1288a8fc87-credential-socket\") pod \"2c7e4408-b25d-44fa-9587-3a1288a8fc87\" (UID: \"2c7e4408-b25d-44fa-9587-3a1288a8fc87\") "
Oct 30 21:12:12.823537 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.823204    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c7e4408-b25d-44fa-9587-3a1288a8fc87-istiod-ca-cert" pod "2c7e4408-b25d-44fa-9587-3a1288a8fc87" (UID: "2c7e4408-b25d-44fa-9587-3a1288a8fc87"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:12:12.823537 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.823217    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c7e4408-b25d-44fa-9587-3a1288a8fc87-workload-certs" pod "2c7e4408-b25d-44fa-9587-3a1288a8fc87" (UID: "2c7e4408-b25d-44fa-9587-3a1288a8fc87"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:12:12.823537 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.823490    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c7e4408-b25d-44fa-9587-3a1288a8fc87-credential-socket" pod "2c7e4408-b25d-44fa-9587-3a1288a8fc87" (UID: "2c7e4408-b25d-44fa-9587-3a1288a8fc87"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:12:12.823697 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.823583    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c7e4408-b25d-44fa-9587-3a1288a8fc87-istio-podinfo" pod "2c7e4408-b25d-44fa-9587-3a1288a8fc87" (UID: "2c7e4408-b25d-44fa-9587-3a1288a8fc87"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:12:12.823732 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.823692    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c7e4408-b25d-44fa-9587-3a1288a8fc87-kube-api-access-8fcm2" pod "2c7e4408-b25d-44fa-9587-3a1288a8fc87" (UID: "2c7e4408-b25d-44fa-9587-3a1288a8fc87"). InnerVolumeSpecName "kube-api-access-8fcm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:12:12.823765 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.823742    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/089ae08c-aa75-4c9b-ab51-70684e9ba0a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-dc6125-4n8cr\" (UID: \"089ae08c-aa75-4c9b-ab51-70684e9ba0a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-dc6125-4n8cr"
Oct 30 21:12:12.823910 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.823893    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/089ae08c-aa75-4c9b-ab51-70684e9ba0a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-dc6125-4n8cr\" (UID: \"089ae08c-aa75-4c9b-ab51-70684e9ba0a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-dc6125-4n8cr"
Oct 30 21:12:12.823989 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.823974    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtcs6\" (UniqueName: \"kubernetes.io/projected/089ae08c-aa75-4c9b-ab51-70684e9ba0a1-kube-api-access-mtcs6\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-dc6125-4n8cr\" (UID: \"089ae08c-aa75-4c9b-ab51-70684e9ba0a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-dc6125-4n8cr"
Oct 30 21:12:12.824223 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.824207    5713 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c7e4408-b25d-44fa-9587-3a1288a8fc87-credential-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:12.824297 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.824229    5713 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb252301-e703-4e46-b569-87b1039211d9-credential-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:12.824297 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.824227    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b4b9def-33e5-4d81-a159-1389d71ced98-istio-envoy" pod "4b4b9def-33e5-4d81-a159-1389d71ced98" (UID: "4b4b9def-33e5-4d81-a159-1389d71ced98"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:12:12.824297 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.824245    5713 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c7e4408-b25d-44fa-9587-3a1288a8fc87-istio-token\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:12.824297 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.824262    5713 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb252301-e703-4e46-b569-87b1039211d9-workload-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:12.824297 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.824275    5713 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b4b9def-33e5-4d81-a159-1389d71ced98-workload-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:12.824297 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.824289    5713 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b4b9def-33e5-4d81-a159-1389d71ced98-istio-data\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:12.824774 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.824304    5713 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb252301-e703-4e46-b569-87b1039211d9-workload-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:12.824774 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.824319    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kp2jw\" (UniqueName: \"kubernetes.io/projected/4b4b9def-33e5-4d81-a159-1389d71ced98-kube-api-access-kp2jw\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:12.824774 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.824333    5713 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b4b9def-33e5-4d81-a159-1389d71ced98-credential-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:12.824774 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.824347    5713 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c7e4408-b25d-44fa-9587-3a1288a8fc87-istiod-ca-cert\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:12.824774 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.824361    5713 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb252301-e703-4e46-b569-87b1039211d9-istio-envoy\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:12.824774 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.824374    5713 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b4b9def-33e5-4d81-a159-1389d71ced98-istiod-ca-cert\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:12.824774 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.824387    5713 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb252301-e703-4e46-b569-87b1039211d9-istio-podinfo\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:12.824774 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.824401    5713 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c7e4408-b25d-44fa-9587-3a1288a8fc87-workload-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:12.824774 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.824415    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fcm2\" (UniqueName: \"kubernetes.io/projected/2c7e4408-b25d-44fa-9587-3a1288a8fc87-kube-api-access-8fcm2\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:12.824774 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.824429    5713 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b4b9def-33e5-4d81-a159-1389d71ced98-istio-token\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:12.824774 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.824443    5713 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb252301-e703-4e46-b569-87b1039211d9-istiod-ca-cert\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:12.824774 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.824455    5713 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb252301-e703-4e46-b569-87b1039211d9-istio-data\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:12.824774 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.824468    5713 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c7e4408-b25d-44fa-9587-3a1288a8fc87-istio-data\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:12.824774 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.824473    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b4b9def-33e5-4d81-a159-1389d71ced98-istio-podinfo" pod "4b4b9def-33e5-4d81-a159-1389d71ced98" (UID: "4b4b9def-33e5-4d81-a159-1389d71ced98"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:12:12.824774 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.824481    5713 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c7e4408-b25d-44fa-9587-3a1288a8fc87-istio-podinfo\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:12.824774 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.824514    5713 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b4b9def-33e5-4d81-a159-1389d71ced98-workload-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:12.824774 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.824532    5713 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c7e4408-b25d-44fa-9587-3a1288a8fc87-workload-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:12.824774 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.824547    5713 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb252301-e703-4e46-b569-87b1039211d9-istio-token\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:12.824774 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.824562    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4q65n\" (UniqueName: \"kubernetes.io/projected/eb252301-e703-4e46-b569-87b1039211d9-kube-api-access-4q65n\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:12.824774 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.824740    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/089ae08c-aa75-4c9b-ab51-70684e9ba0a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-dc6125-4n8cr\" (UID: \"089ae08c-aa75-4c9b-ab51-70684e9ba0a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-dc6125-4n8cr"
Oct 30 21:12:12.825961 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.825566    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c7e4408-b25d-44fa-9587-3a1288a8fc87-istio-envoy" pod "2c7e4408-b25d-44fa-9587-3a1288a8fc87" (UID: "2c7e4408-b25d-44fa-9587-3a1288a8fc87"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:12:12.826510 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.826488    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/089ae08c-aa75-4c9b-ab51-70684e9ba0a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-dc6125-4n8cr\" (UID: \"089ae08c-aa75-4c9b-ab51-70684e9ba0a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-dc6125-4n8cr"
Oct 30 21:12:12.881357 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.881272    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtcs6\" (UniqueName: \"kubernetes.io/projected/089ae08c-aa75-4c9b-ab51-70684e9ba0a1-kube-api-access-mtcs6\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-dc6125-4n8cr\" (UID: \"089ae08c-aa75-4c9b-ab51-70684e9ba0a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-dc6125-4n8cr"
Oct 30 21:12:12.926010 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.925972    5713 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c7e4408-b25d-44fa-9587-3a1288a8fc87-istio-envoy\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:12.926010 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.926009    5713 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b4b9def-33e5-4d81-a159-1389d71ced98-istio-envoy\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:12.926210 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.926026    5713 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b4b9def-33e5-4d81-a159-1389d71ced98-istio-podinfo\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:12.940781 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.940751    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef7e2d-rqhqc"]
Oct 30 21:12:12.942513 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:12:12.942483    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb61bd136_4b2d_4ab1_b263_6531234de46b.slice/crio-045a66029ea3143e04e08697408064f819c8496169c3491ba500322cbaedee18 WatchSource:0}: Error finding container 045a66029ea3143e04e08697408064f819c8496169c3491ba500322cbaedee18: Status 404 returned error can't find the container with id 045a66029ea3143e04e08697408064f819c8496169c3491ba500322cbaedee18
Oct 30 21:12:12.984528 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:12.984498    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-dc6125-4n8cr"
Oct 30 21:12:13.071305 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:13.071262    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66df6c04-8987-4f07-9fab-4e80cf71f6a2" path="/var/lib/kubelet/pods/66df6c04-8987-4f07-9fab-4e80cf71f6a2/volumes"
Oct 30 21:12:13.335282 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:13.335252    5713 generic.go:356] "Generic (PLEG): container finished" podID="2c7e4408-b25d-44fa-9587-3a1288a8fc87" containerID="148708a70cf2d226013537be1c071f074cc1d2ac70302684097f722729bb6293" exitCode=0
Oct 30 21:12:13.335464 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:13.335301    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb"
Oct 30 21:12:13.335464 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:13.335340    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb" event={"ID":"2c7e4408-b25d-44fa-9587-3a1288a8fc87","Type":"ContainerDied","Data":"148708a70cf2d226013537be1c071f074cc1d2ac70302684097f722729bb6293"}
Oct 30 21:12:13.335464 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:13.335397    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb" event={"ID":"2c7e4408-b25d-44fa-9587-3a1288a8fc87","Type":"ContainerDied","Data":"0ebcf54c326c29900aa6d6e383bb983bd4021417be3127a93ac4378491463954"}
Oct 30 21:12:13.335464 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:13.335429    5713 scope.go:122] "RemoveContainer" containerID="148708a70cf2d226013537be1c071f074cc1d2ac70302684097f722729bb6293"
Oct 30 21:12:13.339057 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:13.339033    5713 generic.go:356] "Generic (PLEG): container finished" podID="4b4b9def-33e5-4d81-a159-1389d71ced98" containerID="f4a8e6c8b5d4bc22f229aea1dd9f16222b97e03f0028e172d56130e7d53ffe48" exitCode=0
Oct 30 21:12:13.339194 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:13.339069    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj" event={"ID":"4b4b9def-33e5-4d81-a159-1389d71ced98","Type":"ContainerDied","Data":"f4a8e6c8b5d4bc22f229aea1dd9f16222b97e03f0028e172d56130e7d53ffe48"}
Oct 30 21:12:13.339194 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:13.339124    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj" event={"ID":"4b4b9def-33e5-4d81-a159-1389d71ced98","Type":"ContainerDied","Data":"be792cf26fcfb01d8ba3a9b42146c24e105e6e9c07a4397e5c86840732a362fd"}
Oct 30 21:12:13.339194 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:13.339086    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj"
Oct 30 21:12:13.340934 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:13.340909    5713 generic.go:356] "Generic (PLEG): container finished" podID="eb252301-e703-4e46-b569-87b1039211d9" containerID="76ecd8330c15b09ee14ec2cc7140992dd8e7ec4896965380dc97f8943ea1a64a" exitCode=0
Oct 30 21:12:13.341037 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:13.341004    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59"
Oct 30 21:12:13.341105 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:13.341059    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59" event={"ID":"eb252301-e703-4e46-b569-87b1039211d9","Type":"ContainerDied","Data":"76ecd8330c15b09ee14ec2cc7140992dd8e7ec4896965380dc97f8943ea1a64a"}
Oct 30 21:12:13.341163 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:13.341121    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59" event={"ID":"eb252301-e703-4e46-b569-87b1039211d9","Type":"ContainerDied","Data":"6cdaaa54ae16b5a481a0e227569241a3cfb1e4c8d34f38257c28644a852ea900"}
Oct 30 21:12:13.342866 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:13.342844    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef7e2d-rqhqc" event={"ID":"b61bd136-4b2d-4ab1-b263-6531234de46b","Type":"ContainerStarted","Data":"045a66029ea3143e04e08697408064f819c8496169c3491ba500322cbaedee18"}
Oct 30 21:12:13.349570 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:13.349553    5713 scope.go:122] "RemoveContainer" containerID="148708a70cf2d226013537be1c071f074cc1d2ac70302684097f722729bb6293"
Oct 30 21:12:13.349883 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:12:13.349856    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"148708a70cf2d226013537be1c071f074cc1d2ac70302684097f722729bb6293\": container with ID starting with 148708a70cf2d226013537be1c071f074cc1d2ac70302684097f722729bb6293 not found: ID does not exist" containerID="148708a70cf2d226013537be1c071f074cc1d2ac70302684097f722729bb6293"
Oct 30 21:12:13.349940 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:13.349893    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"148708a70cf2d226013537be1c071f074cc1d2ac70302684097f722729bb6293"} err="failed to get container status \"148708a70cf2d226013537be1c071f074cc1d2ac70302684097f722729bb6293\": rpc error: code = NotFound desc = could not find container \"148708a70cf2d226013537be1c071f074cc1d2ac70302684097f722729bb6293\": container with ID starting with 148708a70cf2d226013537be1c071f074cc1d2ac70302684097f722729bb6293 not found: ID does not exist"
Oct 30 21:12:13.349940 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:13.349927    5713 scope.go:122] "RemoveContainer" containerID="f4a8e6c8b5d4bc22f229aea1dd9f16222b97e03f0028e172d56130e7d53ffe48"
Oct 30 21:12:13.360060 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:13.360046    5713 scope.go:122] "RemoveContainer" containerID="f4a8e6c8b5d4bc22f229aea1dd9f16222b97e03f0028e172d56130e7d53ffe48"
Oct 30 21:12:13.360303 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:12:13.360286    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4a8e6c8b5d4bc22f229aea1dd9f16222b97e03f0028e172d56130e7d53ffe48\": container with ID starting with f4a8e6c8b5d4bc22f229aea1dd9f16222b97e03f0028e172d56130e7d53ffe48 not found: ID does not exist" containerID="f4a8e6c8b5d4bc22f229aea1dd9f16222b97e03f0028e172d56130e7d53ffe48"
Oct 30 21:12:13.360345 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:13.360309    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4a8e6c8b5d4bc22f229aea1dd9f16222b97e03f0028e172d56130e7d53ffe48"} err="failed to get container status \"f4a8e6c8b5d4bc22f229aea1dd9f16222b97e03f0028e172d56130e7d53ffe48\": rpc error: code = NotFound desc = could not find container \"f4a8e6c8b5d4bc22f229aea1dd9f16222b97e03f0028e172d56130e7d53ffe48\": container with ID starting with f4a8e6c8b5d4bc22f229aea1dd9f16222b97e03f0028e172d56130e7d53ffe48 not found: ID does not exist"
Oct 30 21:12:13.360345 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:13.360324    5713 scope.go:122] "RemoveContainer" containerID="76ecd8330c15b09ee14ec2cc7140992dd8e7ec4896965380dc97f8943ea1a64a"
Oct 30 21:12:13.370295 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:13.370276    5713 scope.go:122] "RemoveContainer" containerID="76ecd8330c15b09ee14ec2cc7140992dd8e7ec4896965380dc97f8943ea1a64a"
Oct 30 21:12:13.370547 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:12:13.370529    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76ecd8330c15b09ee14ec2cc7140992dd8e7ec4896965380dc97f8943ea1a64a\": container with ID starting with 76ecd8330c15b09ee14ec2cc7140992dd8e7ec4896965380dc97f8943ea1a64a not found: ID does not exist" containerID="76ecd8330c15b09ee14ec2cc7140992dd8e7ec4896965380dc97f8943ea1a64a"
Oct 30 21:12:13.370609 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:13.370552    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76ecd8330c15b09ee14ec2cc7140992dd8e7ec4896965380dc97f8943ea1a64a"} err="failed to get container status \"76ecd8330c15b09ee14ec2cc7140992dd8e7ec4896965380dc97f8943ea1a64a\": rpc error: code = NotFound desc = could not find container \"76ecd8330c15b09ee14ec2cc7140992dd8e7ec4896965380dc97f8943ea1a64a\": container with ID starting with 76ecd8330c15b09ee14ec2cc7140992dd8e7ec4896965380dc97f8943ea1a64a not found: ID does not exist"
Oct 30 21:12:13.497777 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:13.497736    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-dc6125-4n8cr"]
Oct 30 21:12:13.498450 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:12:13.498416    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod089ae08c_aa75_4c9b_ab51_70684e9ba0a1.slice/crio-bc42c27f69a4495caddb630ba5a4ef17f807de288654848edf93b0312e50bb4e WatchSource:0}: Error finding container bc42c27f69a4495caddb630ba5a4ef17f807de288654848edf93b0312e50bb4e: Status 404 returned error can't find the container with id bc42c27f69a4495caddb630ba5a4ef17f807de288654848edf93b0312e50bb4e
Oct 30 21:12:13.503353 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:13.503331    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59"]
Oct 30 21:12:13.512503 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:13.512476    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-nkq59"]
Oct 30 21:12:13.616797 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:13.616757    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj"]
Oct 30 21:12:13.620941 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:13.620911    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-wz8pj"]
Oct 30 21:12:13.728641 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:13.728610    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb"]
Oct 30 21:12:13.740184 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:13.740154    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-vthlb"]
Oct 30 21:12:14.355174 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:14.355140    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-dc6125-4n8cr" event={"ID":"089ae08c-aa75-4c9b-ab51-70684e9ba0a1","Type":"ContainerStarted","Data":"f58c4f7fbb4b78ce6c3de1cc5dd1ff4b3327ea7a65c1574dfec5ad788c23a165"}
Oct 30 21:12:14.355391 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:14.355196    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-dc6125-4n8cr" event={"ID":"089ae08c-aa75-4c9b-ab51-70684e9ba0a1","Type":"ContainerStarted","Data":"bc42c27f69a4495caddb630ba5a4ef17f807de288654848edf93b0312e50bb4e"}
Oct 30 21:12:14.359056 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:14.359024    5713 generic.go:356] "Generic (PLEG): container finished" podID="b61bd136-4b2d-4ab1-b263-6531234de46b" containerID="e66f28c3bdda8ff572c884c025956f2845957102575d2bd249c5e2db7382c889" exitCode=0
Oct 30 21:12:14.359174 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:14.359099    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef7e2d-rqhqc" event={"ID":"b61bd136-4b2d-4ab1-b263-6531234de46b","Type":"ContainerDied","Data":"e66f28c3bdda8ff572c884c025956f2845957102575d2bd249c5e2db7382c889"}
Oct 30 21:12:14.518267 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:14.518208    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-dc6125-4n8cr" podStartSLOduration=2.518196794 podStartE2EDuration="2.518196794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:12:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:12:14.467384781 +0000 UTC m=+2017.532161181" watchObservedRunningTime="2026-10-30 21:12:14.518196794 +0000 UTC m=+2017.582973223"
Oct 30 21:12:15.069164 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:15.069132    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c7e4408-b25d-44fa-9587-3a1288a8fc87" path="/var/lib/kubelet/pods/2c7e4408-b25d-44fa-9587-3a1288a8fc87/volumes"
Oct 30 21:12:15.069835 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:15.069816    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b4b9def-33e5-4d81-a159-1389d71ced98" path="/var/lib/kubelet/pods/4b4b9def-33e5-4d81-a159-1389d71ced98/volumes"
Oct 30 21:12:15.070313 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:15.070298    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb252301-e703-4e46-b569-87b1039211d9" path="/var/lib/kubelet/pods/eb252301-e703-4e46-b569-87b1039211d9/volumes"
Oct 30 21:12:15.369606 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:15.369518    5713 generic.go:356] "Generic (PLEG): container finished" podID="089ae08c-aa75-4c9b-ab51-70684e9ba0a1" containerID="f58c4f7fbb4b78ce6c3de1cc5dd1ff4b3327ea7a65c1574dfec5ad788c23a165" exitCode=0
Oct 30 21:12:15.369782 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:15.369609    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-dc6125-4n8cr" event={"ID":"089ae08c-aa75-4c9b-ab51-70684e9ba0a1","Type":"ContainerDied","Data":"f58c4f7fbb4b78ce6c3de1cc5dd1ff4b3327ea7a65c1574dfec5ad788c23a165"}
Oct 30 21:12:15.545390 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:15.545356    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef7e2d-rqhqc"
Oct 30 21:12:15.586575 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:15.586548    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b61bd136-4b2d-4ab1-b263-6531234de46b-kube-api-access-bnp27\" (UniqueName: \"kubernetes.io/projected/b61bd136-4b2d-4ab1-b263-6531234de46b-kube-api-access-bnp27\") pod \"b61bd136-4b2d-4ab1-b263-6531234de46b\" (UID: \"b61bd136-4b2d-4ab1-b263-6531234de46b\") "
Oct 30 21:12:15.586750 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:15.586601    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b61bd136-4b2d-4ab1-b263-6531234de46b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b61bd136-4b2d-4ab1-b263-6531234de46b-maas-api-openshift-ca\") pod \"b61bd136-4b2d-4ab1-b263-6531234de46b\" (UID: \"b61bd136-4b2d-4ab1-b263-6531234de46b\") "
Oct 30 21:12:15.586839 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:15.586756    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b61bd136-4b2d-4ab1-b263-6531234de46b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b61bd136-4b2d-4ab1-b263-6531234de46b-maas-api-xks-ca\") pod \"b61bd136-4b2d-4ab1-b263-6531234de46b\" (UID: \"b61bd136-4b2d-4ab1-b263-6531234de46b\") "
Oct 30 21:12:15.587061 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:15.587026    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b61bd136-4b2d-4ab1-b263-6531234de46b-maas-api-openshift-ca" pod "b61bd136-4b2d-4ab1-b263-6531234de46b" (UID: "b61bd136-4b2d-4ab1-b263-6531234de46b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:12:15.587559 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:15.587540    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b61bd136-4b2d-4ab1-b263-6531234de46b-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:15.588840 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:15.588776    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b61bd136-4b2d-4ab1-b263-6531234de46b-kube-api-access-bnp27" pod "b61bd136-4b2d-4ab1-b263-6531234de46b" (UID: "b61bd136-4b2d-4ab1-b263-6531234de46b"). InnerVolumeSpecName "kube-api-access-bnp27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:12:15.588932 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:15.588905    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b61bd136-4b2d-4ab1-b263-6531234de46b-maas-api-xks-ca" pod "b61bd136-4b2d-4ab1-b263-6531234de46b" (UID: "b61bd136-4b2d-4ab1-b263-6531234de46b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:12:15.690009 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:15.689977    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b61bd136-4b2d-4ab1-b263-6531234de46b-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:15.690168 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:15.690013    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnp27\" (UniqueName: \"kubernetes.io/projected/b61bd136-4b2d-4ab1-b263-6531234de46b-kube-api-access-bnp27\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:16.384226 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:16.384183    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef7e2d-rqhqc"
Oct 30 21:12:16.384783 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:16.384219    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef7e2d-rqhqc" event={"ID":"b61bd136-4b2d-4ab1-b263-6531234de46b","Type":"ContainerDied","Data":"045a66029ea3143e04e08697408064f819c8496169c3491ba500322cbaedee18"}
Oct 30 21:12:16.384783 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:16.384272    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="045a66029ea3143e04e08697408064f819c8496169c3491ba500322cbaedee18"
Oct 30 21:12:16.559371 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:16.559345    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-dc6125-4n8cr"
Oct 30 21:12:16.603762 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:16.603736    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/089ae08c-aa75-4c9b-ab51-70684e9ba0a1-kube-api-access-mtcs6\" (UniqueName: \"kubernetes.io/projected/089ae08c-aa75-4c9b-ab51-70684e9ba0a1-kube-api-access-mtcs6\") pod \"089ae08c-aa75-4c9b-ab51-70684e9ba0a1\" (UID: \"089ae08c-aa75-4c9b-ab51-70684e9ba0a1\") "
Oct 30 21:12:16.603925 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:16.603776    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/089ae08c-aa75-4c9b-ab51-70684e9ba0a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/089ae08c-aa75-4c9b-ab51-70684e9ba0a1-maas-api-xks-ca\") pod \"089ae08c-aa75-4c9b-ab51-70684e9ba0a1\" (UID: \"089ae08c-aa75-4c9b-ab51-70684e9ba0a1\") "
Oct 30 21:12:16.603925 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:16.603876    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/089ae08c-aa75-4c9b-ab51-70684e9ba0a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/089ae08c-aa75-4c9b-ab51-70684e9ba0a1-maas-api-openshift-ca\") pod \"089ae08c-aa75-4c9b-ab51-70684e9ba0a1\" (UID: \"089ae08c-aa75-4c9b-ab51-70684e9ba0a1\") "
Oct 30 21:12:16.604254 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:16.604225    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/089ae08c-aa75-4c9b-ab51-70684e9ba0a1-maas-api-openshift-ca" pod "089ae08c-aa75-4c9b-ab51-70684e9ba0a1" (UID: "089ae08c-aa75-4c9b-ab51-70684e9ba0a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:12:16.604626 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:16.604608    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/089ae08c-aa75-4c9b-ab51-70684e9ba0a1-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:16.606054 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:16.606025    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/089ae08c-aa75-4c9b-ab51-70684e9ba0a1-maas-api-xks-ca" pod "089ae08c-aa75-4c9b-ab51-70684e9ba0a1" (UID: "089ae08c-aa75-4c9b-ab51-70684e9ba0a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:12:16.606054 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:16.606038    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/089ae08c-aa75-4c9b-ab51-70684e9ba0a1-kube-api-access-mtcs6" pod "089ae08c-aa75-4c9b-ab51-70684e9ba0a1" (UID: "089ae08c-aa75-4c9b-ab51-70684e9ba0a1"). InnerVolumeSpecName "kube-api-access-mtcs6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:12:16.705903 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:16.705866    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtcs6\" (UniqueName: \"kubernetes.io/projected/089ae08c-aa75-4c9b-ab51-70684e9ba0a1-kube-api-access-mtcs6\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:16.705903 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:16.705896    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/089ae08c-aa75-4c9b-ab51-70684e9ba0a1-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:17.394757 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:17.394667    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-dc6125-4n8cr"
Oct 30 21:12:17.394757 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:17.394717    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-dc6125-4n8cr" event={"ID":"089ae08c-aa75-4c9b-ab51-70684e9ba0a1","Type":"ContainerDied","Data":"bc42c27f69a4495caddb630ba5a4ef17f807de288654848edf93b0312e50bb4e"}
Oct 30 21:12:17.395253 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:17.394777    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc42c27f69a4495caddb630ba5a4ef17f807de288654848edf93b0312e50bb4e"
Oct 30 21:12:22.477439 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:22.477404    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm"]
Oct 30 21:12:22.477996 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:22.477604    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm" podUID="9709b238-b5a6-4928-afe5-2337662a1e4e" containerName="istio-proxy" containerID="cri-o://e47639f67d5c1a6cf67c24a49803bf129d584d81998c2b3ae44e8db550423eec" gracePeriod=30
Oct 30 21:12:24.461828 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:12:24.461775    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:12:24.583078 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:24.583035    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jg6f5"]
Oct 30 21:12:24.583299 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:24.583273    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jg6f5" podUID="4271c173-e39b-4207-b710-e2298c3f7b11" containerName="discovery" containerID="cri-o://fba8c41dcd2043058ffbd6f39f88270988bb445ce37710b4caab64de94b61e08" gracePeriod=30
Oct 30 21:12:24.658901 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:24.658867    5713 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-jg6f5 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.136:8080/ready\": dial tcp 10.134.0.136:8080: connect: connection refused" start-of-body=
Oct 30 21:12:24.659074 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:24.658935    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jg6f5" podUID="4271c173-e39b-4207-b710-e2298c3f7b11" containerName="discovery" probeResult="failure" output="Get \"http://10.134.0.136:8080/ready\": dial tcp 10.134.0.136:8080: connect: connection refused"
Oct 30 21:12:24.922192 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:24.922166    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jg6f5"
Oct 30 21:12:25.047986 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:25.047903    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4271c173-e39b-4207-b710-e2298c3f7b11-kube-api-access-76z6f\" (UniqueName: \"kubernetes.io/projected/4271c173-e39b-4207-b710-e2298c3f7b11-kube-api-access-76z6f\") pod \"4271c173-e39b-4207-b710-e2298c3f7b11\" (UID: \"4271c173-e39b-4207-b710-e2298c3f7b11\") "
Oct 30 21:12:25.048161 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:25.047997    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4271c173-e39b-4207-b710-e2298c3f7b11-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4271c173-e39b-4207-b710-e2298c3f7b11-istio-csr-dns-cert\") pod \"4271c173-e39b-4207-b710-e2298c3f7b11\" (UID: \"4271c173-e39b-4207-b710-e2298c3f7b11\") "
Oct 30 21:12:25.048768 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:25.048743    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4271c173-e39b-4207-b710-e2298c3f7b11-istio-token\" (UniqueName: \"kubernetes.io/projected/4271c173-e39b-4207-b710-e2298c3f7b11-istio-token\") pod \"4271c173-e39b-4207-b710-e2298c3f7b11\" (UID: \"4271c173-e39b-4207-b710-e2298c3f7b11\") "
Oct 30 21:12:25.049025 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:25.049001    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4271c173-e39b-4207-b710-e2298c3f7b11-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4271c173-e39b-4207-b710-e2298c3f7b11-local-certs\") pod \"4271c173-e39b-4207-b710-e2298c3f7b11\" (UID: \"4271c173-e39b-4207-b710-e2298c3f7b11\") "
Oct 30 21:12:25.049128 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:25.049111    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4271c173-e39b-4207-b710-e2298c3f7b11-cacerts\" (UniqueName: \"kubernetes.io/secret/4271c173-e39b-4207-b710-e2298c3f7b11-cacerts\") pod \"4271c173-e39b-4207-b710-e2298c3f7b11\" (UID: \"4271c173-e39b-4207-b710-e2298c3f7b11\") "
Oct 30 21:12:25.049240 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:25.049210    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4271c173-e39b-4207-b710-e2298c3f7b11-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4271c173-e39b-4207-b710-e2298c3f7b11-istio-csr-ca-configmap\") pod \"4271c173-e39b-4207-b710-e2298c3f7b11\" (UID: \"4271c173-e39b-4207-b710-e2298c3f7b11\") "
Oct 30 21:12:25.049305 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:25.049284    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4271c173-e39b-4207-b710-e2298c3f7b11-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4271c173-e39b-4207-b710-e2298c3f7b11-istio-kubeconfig\") pod \"4271c173-e39b-4207-b710-e2298c3f7b11\" (UID: \"4271c173-e39b-4207-b710-e2298c3f7b11\") "
Oct 30 21:12:25.050767 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:25.050736    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4271c173-e39b-4207-b710-e2298c3f7b11-istio-csr-ca-configmap" pod "4271c173-e39b-4207-b710-e2298c3f7b11" (UID: "4271c173-e39b-4207-b710-e2298c3f7b11"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:12:25.051486 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:25.051431    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4271c173-e39b-4207-b710-e2298c3f7b11-kube-api-access-76z6f" pod "4271c173-e39b-4207-b710-e2298c3f7b11" (UID: "4271c173-e39b-4207-b710-e2298c3f7b11"). InnerVolumeSpecName "kube-api-access-76z6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:12:25.052040 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:25.052015    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4271c173-e39b-4207-b710-e2298c3f7b11-local-certs" pod "4271c173-e39b-4207-b710-e2298c3f7b11" (UID: "4271c173-e39b-4207-b710-e2298c3f7b11"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:12:25.052175 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:25.052152    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4271c173-e39b-4207-b710-e2298c3f7b11-cacerts" pod "4271c173-e39b-4207-b710-e2298c3f7b11" (UID: "4271c173-e39b-4207-b710-e2298c3f7b11"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:12:25.053287 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:25.053260    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4271c173-e39b-4207-b710-e2298c3f7b11-istio-token" pod "4271c173-e39b-4207-b710-e2298c3f7b11" (UID: "4271c173-e39b-4207-b710-e2298c3f7b11"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:12:25.053376 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:25.053281    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4271c173-e39b-4207-b710-e2298c3f7b11-istio-kubeconfig" pod "4271c173-e39b-4207-b710-e2298c3f7b11" (UID: "4271c173-e39b-4207-b710-e2298c3f7b11"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:12:25.054225 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:25.054203    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4271c173-e39b-4207-b710-e2298c3f7b11-istio-csr-dns-cert" pod "4271c173-e39b-4207-b710-e2298c3f7b11" (UID: "4271c173-e39b-4207-b710-e2298c3f7b11"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:12:25.152147 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:25.152113    5713 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4271c173-e39b-4207-b710-e2298c3f7b11-local-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:25.152147 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:25.152140    5713 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4271c173-e39b-4207-b710-e2298c3f7b11-cacerts\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:25.152147 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:25.152150    5713 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4271c173-e39b-4207-b710-e2298c3f7b11-istio-csr-ca-configmap\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:25.152417 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:25.152163    5713 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4271c173-e39b-4207-b710-e2298c3f7b11-istio-kubeconfig\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:25.152417 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:25.152172    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76z6f\" (UniqueName: \"kubernetes.io/projected/4271c173-e39b-4207-b710-e2298c3f7b11-kube-api-access-76z6f\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:25.152417 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:25.152181    5713 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4271c173-e39b-4207-b710-e2298c3f7b11-istio-csr-dns-cert\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:25.152417 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:25.152189    5713 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4271c173-e39b-4207-b710-e2298c3f7b11-istio-token\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:25.520705 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:25.520668    5713 generic.go:356] "Generic (PLEG): container finished" podID="4271c173-e39b-4207-b710-e2298c3f7b11" containerID="fba8c41dcd2043058ffbd6f39f88270988bb445ce37710b4caab64de94b61e08" exitCode=0
Oct 30 21:12:25.521235 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:25.520734    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jg6f5"
Oct 30 21:12:25.521235 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:25.520758    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jg6f5" event={"ID":"4271c173-e39b-4207-b710-e2298c3f7b11","Type":"ContainerDied","Data":"fba8c41dcd2043058ffbd6f39f88270988bb445ce37710b4caab64de94b61e08"}
Oct 30 21:12:25.521235 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:25.520851    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jg6f5" event={"ID":"4271c173-e39b-4207-b710-e2298c3f7b11","Type":"ContainerDied","Data":"35c1fcde4270eece5f8b5eb53af80d881b768c31c88668d00ec47ea6f2318d82"}
Oct 30 21:12:25.521235 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:25.520879    5713 scope.go:122] "RemoveContainer" containerID="fba8c41dcd2043058ffbd6f39f88270988bb445ce37710b4caab64de94b61e08"
Oct 30 21:12:25.539203 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:25.539178    5713 scope.go:122] "RemoveContainer" containerID="fba8c41dcd2043058ffbd6f39f88270988bb445ce37710b4caab64de94b61e08"
Oct 30 21:12:25.539555 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:12:25.539528    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fba8c41dcd2043058ffbd6f39f88270988bb445ce37710b4caab64de94b61e08\": container with ID starting with fba8c41dcd2043058ffbd6f39f88270988bb445ce37710b4caab64de94b61e08 not found: ID does not exist" containerID="fba8c41dcd2043058ffbd6f39f88270988bb445ce37710b4caab64de94b61e08"
Oct 30 21:12:25.539644 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:25.539566    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fba8c41dcd2043058ffbd6f39f88270988bb445ce37710b4caab64de94b61e08"} err="failed to get container status \"fba8c41dcd2043058ffbd6f39f88270988bb445ce37710b4caab64de94b61e08\": rpc error: code = NotFound desc = could not find container \"fba8c41dcd2043058ffbd6f39f88270988bb445ce37710b4caab64de94b61e08\": container with ID starting with fba8c41dcd2043058ffbd6f39f88270988bb445ce37710b4caab64de94b61e08 not found: ID does not exist"
Oct 30 21:12:25.691489 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:25.691453    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jg6f5"]
Oct 30 21:12:25.740767 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:25.740734    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jg6f5"]
Oct 30 21:12:26.099058 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:26.099029    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-4d1110-25cpk"]
Oct 30 21:12:26.099724 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:26.099709    5713 cpu_manager.go:414] "containerMap: removing container" podUID="4271c173-e39b-4207-b710-e2298c3f7b11" containerName="discovery"
Oct 30 21:12:26.099772 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:26.099726    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4271c173-e39b-4207-b710-e2298c3f7b11" containerName="discovery"
Oct 30 21:12:26.099772 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:26.099741    5713 cpu_manager.go:414] "containerMap: removing container" podUID="4b4b9def-33e5-4d81-a159-1389d71ced98" containerName="istio-proxy"
Oct 30 21:12:26.099772 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:26.099747    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b4b9def-33e5-4d81-a159-1389d71ced98" containerName="istio-proxy"
Oct 30 21:12:26.099913 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:26.099782    5713 cpu_manager.go:414] "containerMap: removing container" podUID="089ae08c-aa75-4c9b-ab51-70684e9ba0a1" containerName="revoke-keys"
Oct 30 21:12:26.099913 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:26.099807    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="089ae08c-aa75-4c9b-ab51-70684e9ba0a1" containerName="revoke-keys"
Oct 30 21:12:26.099913 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:26.099831    5713 cpu_manager.go:414] "containerMap: removing container" podUID="eb252301-e703-4e46-b569-87b1039211d9" containerName="istio-proxy"
Oct 30 21:12:26.099913 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:26.099838    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb252301-e703-4e46-b569-87b1039211d9" containerName="istio-proxy"
Oct 30 21:12:26.099913 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:26.099851    5713 cpu_manager.go:414] "containerMap: removing container" podUID="2c7e4408-b25d-44fa-9587-3a1288a8fc87" containerName="istio-proxy"
Oct 30 21:12:26.099913 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:26.099855    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c7e4408-b25d-44fa-9587-3a1288a8fc87" containerName="istio-proxy"
Oct 30 21:12:26.099913 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:26.099869    5713 cpu_manager.go:414] "containerMap: removing container" podUID="b61bd136-4b2d-4ab1-b263-6531234de46b" containerName="revoke-keys"
Oct 30 21:12:26.099913 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:26.099873    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b61bd136-4b2d-4ab1-b263-6531234de46b" containerName="revoke-keys"
Oct 30 21:12:26.100153 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:26.100003    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="089ae08c-aa75-4c9b-ab51-70684e9ba0a1" containerName="revoke-keys"
Oct 30 21:12:26.100153 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:26.100015    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb252301-e703-4e46-b569-87b1039211d9" containerName="istio-proxy"
Oct 30 21:12:26.100153 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:26.100024    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c7e4408-b25d-44fa-9587-3a1288a8fc87" containerName="istio-proxy"
Oct 30 21:12:26.100153 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:26.100032    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="b61bd136-4b2d-4ab1-b263-6531234de46b" containerName="revoke-keys"
Oct 30 21:12:26.100153 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:26.100040    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="4271c173-e39b-4207-b710-e2298c3f7b11" containerName="discovery"
Oct 30 21:12:26.100153 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:26.100047    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b4b9def-33e5-4d81-a159-1389d71ced98" containerName="istio-proxy"
Oct 30 21:12:26.104780 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:26.104765    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-4d1110-25cpk"
Oct 30 21:12:26.158047 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:26.158018    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:12:26.158234 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:26.158098    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:12:26.161775 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:26.161748    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-4d1110-25cpk"]
Oct 30 21:12:26.170532 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:26.170501    5713 patch_prober.go:29] interesting pod/e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:12:26.170678 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:26.170553    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm" podUID="9709b238-b5a6-4928-afe5-2337662a1e4e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:12:26.176619 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:26.176587    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/223405e8-db69-41dc-8c2c-f1258995245e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-4d1110-25cpk\" (UID: \"223405e8-db69-41dc-8c2c-f1258995245e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-4d1110-25cpk"
Oct 30 21:12:26.176752 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:26.176655    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8d5c\" (UniqueName: \"kubernetes.io/projected/223405e8-db69-41dc-8c2c-f1258995245e-kube-api-access-h8d5c\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-4d1110-25cpk\" (UID: \"223405e8-db69-41dc-8c2c-f1258995245e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-4d1110-25cpk"
Oct 30 21:12:26.176948 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:26.176919    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/223405e8-db69-41dc-8c2c-f1258995245e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-4d1110-25cpk\" (UID: \"223405e8-db69-41dc-8c2c-f1258995245e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-4d1110-25cpk"
Oct 30 21:12:26.230839 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:26.230778    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:12:26.231124 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:26.231107    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:12:26.278176 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:26.278143    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/223405e8-db69-41dc-8c2c-f1258995245e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-4d1110-25cpk\" (UID: \"223405e8-db69-41dc-8c2c-f1258995245e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-4d1110-25cpk"
Oct 30 21:12:26.278311 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:26.278209    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8d5c\" (UniqueName: \"kubernetes.io/projected/223405e8-db69-41dc-8c2c-f1258995245e-kube-api-access-h8d5c\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-4d1110-25cpk\" (UID: \"223405e8-db69-41dc-8c2c-f1258995245e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-4d1110-25cpk"
Oct 30 21:12:26.278311 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:26.278302    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/223405e8-db69-41dc-8c2c-f1258995245e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-4d1110-25cpk\" (UID: \"223405e8-db69-41dc-8c2c-f1258995245e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-4d1110-25cpk"
Oct 30 21:12:26.278919 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:26.278894    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/223405e8-db69-41dc-8c2c-f1258995245e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-4d1110-25cpk\" (UID: \"223405e8-db69-41dc-8c2c-f1258995245e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-4d1110-25cpk"
Oct 30 21:12:26.280924 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:26.280904    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/223405e8-db69-41dc-8c2c-f1258995245e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-4d1110-25cpk\" (UID: \"223405e8-db69-41dc-8c2c-f1258995245e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-4d1110-25cpk"
Oct 30 21:12:26.336846 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:26.336809    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8d5c\" (UniqueName: \"kubernetes.io/projected/223405e8-db69-41dc-8c2c-f1258995245e-kube-api-access-h8d5c\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-4d1110-25cpk\" (UID: \"223405e8-db69-41dc-8c2c-f1258995245e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-4d1110-25cpk"
Oct 30 21:12:26.430519 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:26.430491    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-4d1110-25cpk"
Oct 30 21:12:26.915184 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:26.911995    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-4d1110-25cpk"]
Oct 30 21:12:27.069488 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:27.069446    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4271c173-e39b-4207-b710-e2298c3f7b11" path="/var/lib/kubelet/pods/4271c173-e39b-4207-b710-e2298c3f7b11/volumes"
Oct 30 21:12:27.544022 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:27.543988    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-4d1110-25cpk" event={"ID":"223405e8-db69-41dc-8c2c-f1258995245e","Type":"ContainerStarted","Data":"62f60bb9996c98bb3b8d6919834b8dd0c6b6e4e12c1fa96250e7bab7c77510e8"}
Oct 30 21:12:27.544235 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:27.544056    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-4d1110-25cpk" event={"ID":"223405e8-db69-41dc-8c2c-f1258995245e","Type":"ContainerStarted","Data":"f3973ec4c904d33268e33359c8e57573888f18508c1f0dd17482bdfab7fb1309"}
Oct 30 21:12:27.660847 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:27.660484    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-4d1110-25cpk" podStartSLOduration=1.660469021 podStartE2EDuration="1.660469021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:12:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:12:27.658751903 +0000 UTC m=+2030.723528304" watchObservedRunningTime="2026-10-30 21:12:27.660469021 +0000 UTC m=+2030.725245418"
Oct 30 21:12:27.769186 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:27.769122    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm"
Oct 30 21:12:27.808178 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:27.808148    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9709b238-b5a6-4928-afe5-2337662a1e4e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9709b238-b5a6-4928-afe5-2337662a1e4e-istio-envoy\") pod \"9709b238-b5a6-4928-afe5-2337662a1e4e\" (UID: \"9709b238-b5a6-4928-afe5-2337662a1e4e\") "
Oct 30 21:12:27.808178 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:27.808184    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9709b238-b5a6-4928-afe5-2337662a1e4e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9709b238-b5a6-4928-afe5-2337662a1e4e-credential-socket\") pod \"9709b238-b5a6-4928-afe5-2337662a1e4e\" (UID: \"9709b238-b5a6-4928-afe5-2337662a1e4e\") "
Oct 30 21:12:27.808371 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:27.808317    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9709b238-b5a6-4928-afe5-2337662a1e4e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9709b238-b5a6-4928-afe5-2337662a1e4e-workload-certs\") pod \"9709b238-b5a6-4928-afe5-2337662a1e4e\" (UID: \"9709b238-b5a6-4928-afe5-2337662a1e4e\") "
Oct 30 21:12:27.808416 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:27.808370    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9709b238-b5a6-4928-afe5-2337662a1e4e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9709b238-b5a6-4928-afe5-2337662a1e4e-workload-socket\") pod \"9709b238-b5a6-4928-afe5-2337662a1e4e\" (UID: \"9709b238-b5a6-4928-afe5-2337662a1e4e\") "
Oct 30 21:12:27.808416 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:27.808400    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9709b238-b5a6-4928-afe5-2337662a1e4e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9709b238-b5a6-4928-afe5-2337662a1e4e-istio-data\") pod \"9709b238-b5a6-4928-afe5-2337662a1e4e\" (UID: \"9709b238-b5a6-4928-afe5-2337662a1e4e\") "
Oct 30 21:12:27.808487 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:27.808424    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9709b238-b5a6-4928-afe5-2337662a1e4e-istio-token\" (UniqueName: \"kubernetes.io/projected/9709b238-b5a6-4928-afe5-2337662a1e4e-istio-token\") pod \"9709b238-b5a6-4928-afe5-2337662a1e4e\" (UID: \"9709b238-b5a6-4928-afe5-2337662a1e4e\") "
Oct 30 21:12:27.808487 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:27.808465    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9709b238-b5a6-4928-afe5-2337662a1e4e-kube-api-access-rljrv\" (UniqueName: \"kubernetes.io/projected/9709b238-b5a6-4928-afe5-2337662a1e4e-kube-api-access-rljrv\") pod \"9709b238-b5a6-4928-afe5-2337662a1e4e\" (UID: \"9709b238-b5a6-4928-afe5-2337662a1e4e\") "
Oct 30 21:12:27.808586 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:27.808498    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9709b238-b5a6-4928-afe5-2337662a1e4e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9709b238-b5a6-4928-afe5-2337662a1e4e-istiod-ca-cert\") pod \"9709b238-b5a6-4928-afe5-2337662a1e4e\" (UID: \"9709b238-b5a6-4928-afe5-2337662a1e4e\") "
Oct 30 21:12:27.808586 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:27.808540    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9709b238-b5a6-4928-afe5-2337662a1e4e-credential-socket" pod "9709b238-b5a6-4928-afe5-2337662a1e4e" (UID: "9709b238-b5a6-4928-afe5-2337662a1e4e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:12:27.808692 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:27.808575    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9709b238-b5a6-4928-afe5-2337662a1e4e-workload-certs" pod "9709b238-b5a6-4928-afe5-2337662a1e4e" (UID: "9709b238-b5a6-4928-afe5-2337662a1e4e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:12:27.808692 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:27.808678    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9709b238-b5a6-4928-afe5-2337662a1e4e-workload-socket" pod "9709b238-b5a6-4928-afe5-2337662a1e4e" (UID: "9709b238-b5a6-4928-afe5-2337662a1e4e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:12:27.808819 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:27.808703    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9709b238-b5a6-4928-afe5-2337662a1e4e-istio-data" pod "9709b238-b5a6-4928-afe5-2337662a1e4e" (UID: "9709b238-b5a6-4928-afe5-2337662a1e4e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:12:27.808819 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:27.808714    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9709b238-b5a6-4928-afe5-2337662a1e4e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9709b238-b5a6-4928-afe5-2337662a1e4e-istio-podinfo\") pod \"9709b238-b5a6-4928-afe5-2337662a1e4e\" (UID: \"9709b238-b5a6-4928-afe5-2337662a1e4e\") "
Oct 30 21:12:27.808959 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:27.808939    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9709b238-b5a6-4928-afe5-2337662a1e4e-istiod-ca-cert" pod "9709b238-b5a6-4928-afe5-2337662a1e4e" (UID: "9709b238-b5a6-4928-afe5-2337662a1e4e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:12:27.809504 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:27.809407    5713 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9709b238-b5a6-4928-afe5-2337662a1e4e-workload-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:27.809631 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:27.809593    5713 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9709b238-b5a6-4928-afe5-2337662a1e4e-workload-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:27.809631 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:27.809609    5713 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9709b238-b5a6-4928-afe5-2337662a1e4e-istio-data\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:27.809631 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:27.809623    5713 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9709b238-b5a6-4928-afe5-2337662a1e4e-istiod-ca-cert\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:27.809853 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:27.809636    5713 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9709b238-b5a6-4928-afe5-2337662a1e4e-credential-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:27.810588 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:27.810562    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9709b238-b5a6-4928-afe5-2337662a1e4e-istio-envoy" pod "9709b238-b5a6-4928-afe5-2337662a1e4e" (UID: "9709b238-b5a6-4928-afe5-2337662a1e4e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:12:27.810892 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:27.810871    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9709b238-b5a6-4928-afe5-2337662a1e4e-kube-api-access-rljrv" pod "9709b238-b5a6-4928-afe5-2337662a1e4e" (UID: "9709b238-b5a6-4928-afe5-2337662a1e4e"). InnerVolumeSpecName "kube-api-access-rljrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:12:27.810974 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:27.810952    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9709b238-b5a6-4928-afe5-2337662a1e4e-istio-token" pod "9709b238-b5a6-4928-afe5-2337662a1e4e" (UID: "9709b238-b5a6-4928-afe5-2337662a1e4e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:12:27.811052 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:27.811029    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9709b238-b5a6-4928-afe5-2337662a1e4e-istio-podinfo" pod "9709b238-b5a6-4928-afe5-2337662a1e4e" (UID: "9709b238-b5a6-4928-afe5-2337662a1e4e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:12:27.910824 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:27.910769    5713 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9709b238-b5a6-4928-afe5-2337662a1e4e-istio-token\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:27.910824 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:27.910814    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rljrv\" (UniqueName: \"kubernetes.io/projected/9709b238-b5a6-4928-afe5-2337662a1e4e-kube-api-access-rljrv\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:27.910824 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:27.910826    5713 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9709b238-b5a6-4928-afe5-2337662a1e4e-istio-podinfo\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:27.910824 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:27.910835    5713 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9709b238-b5a6-4928-afe5-2337662a1e4e-istio-envoy\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:28.555904 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:28.555869    5713 generic.go:356] "Generic (PLEG): container finished" podID="223405e8-db69-41dc-8c2c-f1258995245e" containerID="62f60bb9996c98bb3b8d6919834b8dd0c6b6e4e12c1fa96250e7bab7c77510e8" exitCode=0
Oct 30 21:12:28.556304 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:28.555950    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-4d1110-25cpk" event={"ID":"223405e8-db69-41dc-8c2c-f1258995245e","Type":"ContainerDied","Data":"62f60bb9996c98bb3b8d6919834b8dd0c6b6e4e12c1fa96250e7bab7c77510e8"}
Oct 30 21:12:28.557402 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:28.557384    5713 generic.go:356] "Generic (PLEG): container finished" podID="9709b238-b5a6-4928-afe5-2337662a1e4e" containerID="e47639f67d5c1a6cf67c24a49803bf129d584d81998c2b3ae44e8db550423eec" exitCode=0
Oct 30 21:12:28.557453 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:28.557430    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm"
Oct 30 21:12:28.557487 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:28.557467    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm" event={"ID":"9709b238-b5a6-4928-afe5-2337662a1e4e","Type":"ContainerDied","Data":"e47639f67d5c1a6cf67c24a49803bf129d584d81998c2b3ae44e8db550423eec"}
Oct 30 21:12:28.557531 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:28.557519    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm" event={"ID":"9709b238-b5a6-4928-afe5-2337662a1e4e","Type":"ContainerDied","Data":"dd201631197fecda22f6b2d3986cecdc70a14395f17775f45af239655e393f06"}
Oct 30 21:12:28.557564 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:28.557551    5713 scope.go:122] "RemoveContainer" containerID="e47639f67d5c1a6cf67c24a49803bf129d584d81998c2b3ae44e8db550423eec"
Oct 30 21:12:28.569223 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:28.569200    5713 scope.go:122] "RemoveContainer" containerID="e47639f67d5c1a6cf67c24a49803bf129d584d81998c2b3ae44e8db550423eec"
Oct 30 21:12:28.569532 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:12:28.569513    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e47639f67d5c1a6cf67c24a49803bf129d584d81998c2b3ae44e8db550423eec\": container with ID starting with e47639f67d5c1a6cf67c24a49803bf129d584d81998c2b3ae44e8db550423eec not found: ID does not exist" containerID="e47639f67d5c1a6cf67c24a49803bf129d584d81998c2b3ae44e8db550423eec"
Oct 30 21:12:28.569604 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:28.569540    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e47639f67d5c1a6cf67c24a49803bf129d584d81998c2b3ae44e8db550423eec"} err="failed to get container status \"e47639f67d5c1a6cf67c24a49803bf129d584d81998c2b3ae44e8db550423eec\": rpc error: code = NotFound desc = could not find container \"e47639f67d5c1a6cf67c24a49803bf129d584d81998c2b3ae44e8db550423eec\": container with ID starting with e47639f67d5c1a6cf67c24a49803bf129d584d81998c2b3ae44e8db550423eec not found: ID does not exist"
Oct 30 21:12:28.783554 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:28.783520    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm"]
Oct 30 21:12:28.832773 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:28.832682    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-hfvwm"]
Oct 30 21:12:29.074955 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:29.074920    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9709b238-b5a6-4928-afe5-2337662a1e4e" path="/var/lib/kubelet/pods/9709b238-b5a6-4928-afe5-2337662a1e4e/volumes"
Oct 30 21:12:29.215150 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:29.215113    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ef7e2d-77489c76c6-vf7cz"]
Oct 30 21:12:29.320376 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:29.320341    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ef7e2d-6784699ff9-mcvmc"]
Oct 30 21:12:29.816394 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:29.816367    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-4d1110-25cpk"
Oct 30 21:12:29.908759 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:29.908662    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/223405e8-db69-41dc-8c2c-f1258995245e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/223405e8-db69-41dc-8c2c-f1258995245e-maas-api-xks-ca\") pod \"223405e8-db69-41dc-8c2c-f1258995245e\" (UID: \"223405e8-db69-41dc-8c2c-f1258995245e\") "
Oct 30 21:12:29.908988 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:29.908883    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/223405e8-db69-41dc-8c2c-f1258995245e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/223405e8-db69-41dc-8c2c-f1258995245e-maas-api-openshift-ca\") pod \"223405e8-db69-41dc-8c2c-f1258995245e\" (UID: \"223405e8-db69-41dc-8c2c-f1258995245e\") "
Oct 30 21:12:29.908988 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:29.908960    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/223405e8-db69-41dc-8c2c-f1258995245e-kube-api-access-h8d5c\" (UniqueName: \"kubernetes.io/projected/223405e8-db69-41dc-8c2c-f1258995245e-kube-api-access-h8d5c\") pod \"223405e8-db69-41dc-8c2c-f1258995245e\" (UID: \"223405e8-db69-41dc-8c2c-f1258995245e\") "
Oct 30 21:12:29.909263 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:29.909241    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/223405e8-db69-41dc-8c2c-f1258995245e-maas-api-openshift-ca" pod "223405e8-db69-41dc-8c2c-f1258995245e" (UID: "223405e8-db69-41dc-8c2c-f1258995245e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:12:29.909655 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:29.909635    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/223405e8-db69-41dc-8c2c-f1258995245e-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:29.911311 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:29.911278    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/223405e8-db69-41dc-8c2c-f1258995245e-maas-api-xks-ca" pod "223405e8-db69-41dc-8c2c-f1258995245e" (UID: "223405e8-db69-41dc-8c2c-f1258995245e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:12:29.911443 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:29.911430    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/223405e8-db69-41dc-8c2c-f1258995245e-kube-api-access-h8d5c" pod "223405e8-db69-41dc-8c2c-f1258995245e" (UID: "223405e8-db69-41dc-8c2c-f1258995245e"). InnerVolumeSpecName "kube-api-access-h8d5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:12:30.011774 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:30.011735    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/223405e8-db69-41dc-8c2c-f1258995245e-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:30.011774 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:30.011768    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8d5c\" (UniqueName: \"kubernetes.io/projected/223405e8-db69-41dc-8c2c-f1258995245e-kube-api-access-h8d5c\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:30.579056 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:30.579021    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-4d1110-25cpk"
Oct 30 21:12:30.579056 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:30.579048    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-4d1110-25cpk" event={"ID":"223405e8-db69-41dc-8c2c-f1258995245e","Type":"ContainerDied","Data":"f3973ec4c904d33268e33359c8e57573888f18508c1f0dd17482bdfab7fb1309"}
Oct 30 21:12:30.579319 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:30.579100    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3973ec4c904d33268e33359c8e57573888f18508c1f0dd17482bdfab7fb1309"
Oct 30 21:12:33.860656 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:33.860622    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-390b71-7ff679864c-ghxkc"]
Oct 30 21:12:33.861368 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:33.861353    5713 cpu_manager.go:414] "containerMap: removing container" podUID="9709b238-b5a6-4928-afe5-2337662a1e4e" containerName="istio-proxy"
Oct 30 21:12:33.861414 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:33.861370    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9709b238-b5a6-4928-afe5-2337662a1e4e" containerName="istio-proxy"
Oct 30 21:12:33.861414 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:33.861399    5713 cpu_manager.go:414] "containerMap: removing container" podUID="223405e8-db69-41dc-8c2c-f1258995245e" containerName="revoke-keys"
Oct 30 21:12:33.861414 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:33.861404    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="223405e8-db69-41dc-8c2c-f1258995245e" containerName="revoke-keys"
Oct 30 21:12:33.861516 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:33.861506    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="223405e8-db69-41dc-8c2c-f1258995245e" containerName="revoke-keys"
Oct 30 21:12:33.861549 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:33.861525    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="9709b238-b5a6-4928-afe5-2337662a1e4e" containerName="istio-proxy"
Oct 30 21:12:33.865327 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:33.865310    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-390b71-7ff679864c-ghxkc"
Oct 30 21:12:33.921448 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:33.921410    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-390b71-7ff679864c-ghxkc"]
Oct 30 21:12:33.965419 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:33.965387    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-390b71\""
Oct 30 21:12:34.028675 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:34.028643    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-390b71-5698dc795b-2hp76"]
Oct 30 21:12:34.040190 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:34.034875    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-390b71-5698dc795b-2hp76"
Oct 30 21:12:34.050880 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:34.045358    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1dcf961-808f-43d1-a978-3f9a8762006c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-390b71-7ff679864c-ghxkc\" (UID: \"f1dcf961-808f-43d1-a978-3f9a8762006c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-390b71-7ff679864c-ghxkc"
Oct 30 21:12:34.053212 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:34.053177    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5zc5\" (UniqueName: \"kubernetes.io/projected/f1dcf961-808f-43d1-a978-3f9a8762006c-kube-api-access-b5zc5\") pod \"payload-pre-processing-e2e-inf-a-390b71-7ff679864c-ghxkc\" (UID: \"f1dcf961-808f-43d1-a978-3f9a8762006c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-390b71-7ff679864c-ghxkc"
Oct 30 21:12:34.098645 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:34.094690    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-390b71-dockercfg-9glh5\""
Oct 30 21:12:34.216900 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:34.215746    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptjfc\" (UniqueName: \"kubernetes.io/projected/4a4313b7-0413-4625-9907-e12d701bd8fd-kube-api-access-ptjfc\") pod \"payload-processing-e2e-inf-a-390b71-5698dc795b-2hp76\" (UID: \"4a4313b7-0413-4625-9907-e12d701bd8fd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-390b71-5698dc795b-2hp76"
Oct 30 21:12:34.217346 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:34.217324    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a4313b7-0413-4625-9907-e12d701bd8fd-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-390b71-5698dc795b-2hp76\" (UID: \"4a4313b7-0413-4625-9907-e12d701bd8fd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-390b71-5698dc795b-2hp76"
Oct 30 21:12:34.217516 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:34.217501    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1dcf961-808f-43d1-a978-3f9a8762006c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-390b71-7ff679864c-ghxkc\" (UID: \"f1dcf961-808f-43d1-a978-3f9a8762006c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-390b71-7ff679864c-ghxkc"
Oct 30 21:12:34.217639 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:34.217627    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5zc5\" (UniqueName: \"kubernetes.io/projected/f1dcf961-808f-43d1-a978-3f9a8762006c-kube-api-access-b5zc5\") pod \"payload-pre-processing-e2e-inf-a-390b71-7ff679864c-ghxkc\" (UID: \"f1dcf961-808f-43d1-a978-3f9a8762006c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-390b71-7ff679864c-ghxkc"
Oct 30 21:12:34.217740 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:34.217715    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-390b71-5698dc795b-2hp76"]
Oct 30 21:12:34.218813 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:34.218767    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-ef7e2d-77489c76c6-vf7cz" podUID="1afb9a0a-9a10-4f71-b66d-a7cc873ed48b" containerName="payload-processing" containerID="cri-o://1e64ef0131528605e58d15b41272cf5521a51caaa6017cb61da4e52d0a137107" gracePeriod=25
Oct 30 21:12:34.219839 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:34.218858    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1dcf961-808f-43d1-a978-3f9a8762006c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-390b71-7ff679864c-ghxkc\" (UID: \"f1dcf961-808f-43d1-a978-3f9a8762006c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-390b71-7ff679864c-ghxkc"
Oct 30 21:12:34.276422 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:34.276389    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5zc5\" (UniqueName: \"kubernetes.io/projected/f1dcf961-808f-43d1-a978-3f9a8762006c-kube-api-access-b5zc5\") pod \"payload-pre-processing-e2e-inf-a-390b71-7ff679864c-ghxkc\" (UID: \"f1dcf961-808f-43d1-a978-3f9a8762006c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-390b71-7ff679864c-ghxkc"
Oct 30 21:12:34.319731 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:34.319695    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a4313b7-0413-4625-9907-e12d701bd8fd-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-390b71-5698dc795b-2hp76\" (UID: \"4a4313b7-0413-4625-9907-e12d701bd8fd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-390b71-5698dc795b-2hp76"
Oct 30 21:12:34.319914 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:34.319902    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptjfc\" (UniqueName: \"kubernetes.io/projected/4a4313b7-0413-4625-9907-e12d701bd8fd-kube-api-access-ptjfc\") pod \"payload-processing-e2e-inf-a-390b71-5698dc795b-2hp76\" (UID: \"4a4313b7-0413-4625-9907-e12d701bd8fd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-390b71-5698dc795b-2hp76"
Oct 30 21:12:34.320483 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:34.320457    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a4313b7-0413-4625-9907-e12d701bd8fd-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-390b71-5698dc795b-2hp76\" (UID: \"4a4313b7-0413-4625-9907-e12d701bd8fd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-390b71-5698dc795b-2hp76"
Oct 30 21:12:34.320928 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:34.320885    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef7e2d-6784699ff9-mcvmc" podUID="e9200d55-faeb-4087-85cb-949f64353a55" containerName="payload-pre-processing" containerID="cri-o://3eff780b5ce92b1e268fbe5531fd500409f36519abc1c8c66dd97e1a44d90e24" gracePeriod=25
Oct 30 21:12:34.376771 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:34.376745    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptjfc\" (UniqueName: \"kubernetes.io/projected/4a4313b7-0413-4625-9907-e12d701bd8fd-kube-api-access-ptjfc\") pod \"payload-processing-e2e-inf-a-390b71-5698dc795b-2hp76\" (UID: \"4a4313b7-0413-4625-9907-e12d701bd8fd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-390b71-5698dc795b-2hp76"
Oct 30 21:12:34.406627 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:34.406603    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-390b71-5698dc795b-2hp76"
Oct 30 21:12:34.487385 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:34.487110    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-390b71-7ff679864c-ghxkc"
Oct 30 21:12:34.650808 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:34.650728    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ef7e2d-77489c76c6-vf7cz"
Oct 30 21:12:34.654995 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:34.654975    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef7e2d-6784699ff9-mcvmc"
Oct 30 21:12:34.663178 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:34.663148    5713 generic.go:356] "Generic (PLEG): container finished" podID="e9200d55-faeb-4087-85cb-949f64353a55" containerID="3eff780b5ce92b1e268fbe5531fd500409f36519abc1c8c66dd97e1a44d90e24" exitCode=0
Oct 30 21:12:34.663332 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:34.663241    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef7e2d-6784699ff9-mcvmc" event={"ID":"e9200d55-faeb-4087-85cb-949f64353a55","Type":"ContainerDied","Data":"3eff780b5ce92b1e268fbe5531fd500409f36519abc1c8c66dd97e1a44d90e24"}
Oct 30 21:12:34.663332 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:34.663278    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef7e2d-6784699ff9-mcvmc" event={"ID":"e9200d55-faeb-4087-85cb-949f64353a55","Type":"ContainerDied","Data":"bf8ad350c9a382b03c3ace48692854066b8f837b51a592284b09af8cbdce595f"}
Oct 30 21:12:34.663332 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:34.663300    5713 scope.go:122] "RemoveContainer" containerID="3eff780b5ce92b1e268fbe5531fd500409f36519abc1c8c66dd97e1a44d90e24"
Oct 30 21:12:34.663492 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:34.663421    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ef7e2d-6784699ff9-mcvmc"
Oct 30 21:12:34.665594 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:34.665566    5713 generic.go:356] "Generic (PLEG): container finished" podID="1afb9a0a-9a10-4f71-b66d-a7cc873ed48b" containerID="1e64ef0131528605e58d15b41272cf5521a51caaa6017cb61da4e52d0a137107" exitCode=0
Oct 30 21:12:34.665719 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:34.665701    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ef7e2d-77489c76c6-vf7cz" event={"ID":"1afb9a0a-9a10-4f71-b66d-a7cc873ed48b","Type":"ContainerDied","Data":"1e64ef0131528605e58d15b41272cf5521a51caaa6017cb61da4e52d0a137107"}
Oct 30 21:12:34.665804 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:34.665744    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ef7e2d-77489c76c6-vf7cz" event={"ID":"1afb9a0a-9a10-4f71-b66d-a7cc873ed48b","Type":"ContainerDied","Data":"60a7de002565b5a1d27966a0fdaeadd7f15ed0ab4527f0b936b4bd200efa40b1"}
Oct 30 21:12:34.665882 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:34.665867    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ef7e2d-77489c76c6-vf7cz"
Oct 30 21:12:34.680440 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:34.680411    5713 scope.go:122] "RemoveContainer" containerID="3eff780b5ce92b1e268fbe5531fd500409f36519abc1c8c66dd97e1a44d90e24"
Oct 30 21:12:34.680837 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:12:34.680811    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3eff780b5ce92b1e268fbe5531fd500409f36519abc1c8c66dd97e1a44d90e24\": container with ID starting with 3eff780b5ce92b1e268fbe5531fd500409f36519abc1c8c66dd97e1a44d90e24 not found: ID does not exist" containerID="3eff780b5ce92b1e268fbe5531fd500409f36519abc1c8c66dd97e1a44d90e24"
Oct 30 21:12:34.680933 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:34.680849    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3eff780b5ce92b1e268fbe5531fd500409f36519abc1c8c66dd97e1a44d90e24"} err="failed to get container status \"3eff780b5ce92b1e268fbe5531fd500409f36519abc1c8c66dd97e1a44d90e24\": rpc error: code = NotFound desc = could not find container \"3eff780b5ce92b1e268fbe5531fd500409f36519abc1c8c66dd97e1a44d90e24\": container with ID starting with 3eff780b5ce92b1e268fbe5531fd500409f36519abc1c8c66dd97e1a44d90e24 not found: ID does not exist"
Oct 30 21:12:34.680933 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:34.680874    5713 scope.go:122] "RemoveContainer" containerID="1e64ef0131528605e58d15b41272cf5521a51caaa6017cb61da4e52d0a137107"
Oct 30 21:12:34.695588 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:34.695566    5713 scope.go:122] "RemoveContainer" containerID="1e64ef0131528605e58d15b41272cf5521a51caaa6017cb61da4e52d0a137107"
Oct 30 21:12:34.698632 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:12:34.696781    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e64ef0131528605e58d15b41272cf5521a51caaa6017cb61da4e52d0a137107\": container with ID starting with 1e64ef0131528605e58d15b41272cf5521a51caaa6017cb61da4e52d0a137107 not found: ID does not exist" containerID="1e64ef0131528605e58d15b41272cf5521a51caaa6017cb61da4e52d0a137107"
Oct 30 21:12:34.698632 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:34.696842    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e64ef0131528605e58d15b41272cf5521a51caaa6017cb61da4e52d0a137107"} err="failed to get container status \"1e64ef0131528605e58d15b41272cf5521a51caaa6017cb61da4e52d0a137107\": rpc error: code = NotFound desc = could not find container \"1e64ef0131528605e58d15b41272cf5521a51caaa6017cb61da4e52d0a137107\": container with ID starting with 1e64ef0131528605e58d15b41272cf5521a51caaa6017cb61da4e52d0a137107 not found: ID does not exist"
Oct 30 21:12:34.830800 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:34.830690    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9200d55-faeb-4087-85cb-949f64353a55-kube-api-access-9r9qt\" (UniqueName: \"kubernetes.io/projected/e9200d55-faeb-4087-85cb-949f64353a55-kube-api-access-9r9qt\") pod \"e9200d55-faeb-4087-85cb-949f64353a55\" (UID: \"e9200d55-faeb-4087-85cb-949f64353a55\") "
Oct 30 21:12:34.830984 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:34.830833    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1afb9a0a-9a10-4f71-b66d-a7cc873ed48b-kube-api-access-zwxsr\" (UniqueName: \"kubernetes.io/projected/1afb9a0a-9a10-4f71-b66d-a7cc873ed48b-kube-api-access-zwxsr\") pod \"1afb9a0a-9a10-4f71-b66d-a7cc873ed48b\" (UID: \"1afb9a0a-9a10-4f71-b66d-a7cc873ed48b\") "
Oct 30 21:12:34.830984 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:34.830963    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1afb9a0a-9a10-4f71-b66d-a7cc873ed48b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1afb9a0a-9a10-4f71-b66d-a7cc873ed48b-plugins-config-volume\") pod \"1afb9a0a-9a10-4f71-b66d-a7cc873ed48b\" (UID: \"1afb9a0a-9a10-4f71-b66d-a7cc873ed48b\") "
Oct 30 21:12:34.831181 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:34.831138    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9200d55-faeb-4087-85cb-949f64353a55-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9200d55-faeb-4087-85cb-949f64353a55-plugins-config-volume\") pod \"e9200d55-faeb-4087-85cb-949f64353a55\" (UID: \"e9200d55-faeb-4087-85cb-949f64353a55\") "
Oct 30 21:12:34.831482 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:34.831454    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1afb9a0a-9a10-4f71-b66d-a7cc873ed48b-plugins-config-volume" pod "1afb9a0a-9a10-4f71-b66d-a7cc873ed48b" (UID: "1afb9a0a-9a10-4f71-b66d-a7cc873ed48b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:12:34.831701 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:34.831660    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9200d55-faeb-4087-85cb-949f64353a55-plugins-config-volume" pod "e9200d55-faeb-4087-85cb-949f64353a55" (UID: "e9200d55-faeb-4087-85cb-949f64353a55"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:12:34.832638 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:34.832612    5713 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9200d55-faeb-4087-85cb-949f64353a55-plugins-config-volume\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:34.833011 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:34.832778    5713 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1afb9a0a-9a10-4f71-b66d-a7cc873ed48b-plugins-config-volume\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:34.833491 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:34.833465    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9200d55-faeb-4087-85cb-949f64353a55-kube-api-access-9r9qt" pod "e9200d55-faeb-4087-85cb-949f64353a55" (UID: "e9200d55-faeb-4087-85cb-949f64353a55"). InnerVolumeSpecName "kube-api-access-9r9qt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:12:34.835993 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:34.835972    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1afb9a0a-9a10-4f71-b66d-a7cc873ed48b-kube-api-access-zwxsr" pod "1afb9a0a-9a10-4f71-b66d-a7cc873ed48b" (UID: "1afb9a0a-9a10-4f71-b66d-a7cc873ed48b"). InnerVolumeSpecName "kube-api-access-zwxsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:12:34.838439 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:12:34.838418    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a4313b7_0413_4625_9907_e12d701bd8fd.slice/crio-6ca761c9e7162594d5abd6d2f5a99af217c0aa4b2df6db3ad221ee67285aba38 WatchSource:0}: Error finding container 6ca761c9e7162594d5abd6d2f5a99af217c0aa4b2df6db3ad221ee67285aba38: Status 404 returned error can't find the container with id 6ca761c9e7162594d5abd6d2f5a99af217c0aa4b2df6db3ad221ee67285aba38
Oct 30 21:12:34.839078 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:34.839060    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-390b71-5698dc795b-2hp76"]
Oct 30 21:12:34.928167 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:34.928117    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-390b71-7ff679864c-ghxkc"]
Oct 30 21:12:34.928712 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:12:34.928684    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1dcf961_808f_43d1_a978_3f9a8762006c.slice/crio-58ccd16b064fbd72050fed86190f808c7fb1203c617e1911d712733ad5d46941 WatchSource:0}: Error finding container 58ccd16b064fbd72050fed86190f808c7fb1203c617e1911d712733ad5d46941: Status 404 returned error can't find the container with id 58ccd16b064fbd72050fed86190f808c7fb1203c617e1911d712733ad5d46941
Oct 30 21:12:34.934322 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:34.934300    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r9qt\" (UniqueName: \"kubernetes.io/projected/e9200d55-faeb-4087-85cb-949f64353a55-kube-api-access-9r9qt\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:34.934414 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:34.934326    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwxsr\" (UniqueName: \"kubernetes.io/projected/1afb9a0a-9a10-4f71-b66d-a7cc873ed48b-kube-api-access-zwxsr\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:35.107642 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:35.107572    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ef7e2d-77489c76c6-vf7cz"]
Oct 30 21:12:35.111398 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:35.111375    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ef7e2d-77489c76c6-vf7cz"]
Oct 30 21:12:35.218128 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:35.218096    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ef7e2d-6784699ff9-mcvmc"]
Oct 30 21:12:35.221887 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:35.221866    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ef7e2d-6784699ff9-mcvmc"]
Oct 30 21:12:35.676457 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:35.676413    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-390b71-7ff679864c-ghxkc" event={"ID":"f1dcf961-808f-43d1-a978-3f9a8762006c","Type":"ContainerStarted","Data":"c1b4ac90d78ab627f7a0a228774a118950b0622e2bed50396d9fd52669239ff1"}
Oct 30 21:12:35.676457 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:35.676461    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-390b71-7ff679864c-ghxkc" event={"ID":"f1dcf961-808f-43d1-a978-3f9a8762006c","Type":"ContainerStarted","Data":"58ccd16b064fbd72050fed86190f808c7fb1203c617e1911d712733ad5d46941"}
Oct 30 21:12:35.676721 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:35.676650    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-390b71-7ff679864c-ghxkc"
Oct 30 21:12:35.679490 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:35.679460    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-390b71-5698dc795b-2hp76" event={"ID":"4a4313b7-0413-4625-9907-e12d701bd8fd","Type":"ContainerStarted","Data":"c634c3ba4780c2513c5f992c3c41a9bb803071a75c5c71bf99eea35ddf376c5e"}
Oct 30 21:12:35.679598 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:35.679506    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-390b71-5698dc795b-2hp76" event={"ID":"4a4313b7-0413-4625-9907-e12d701bd8fd","Type":"ContainerStarted","Data":"6ca761c9e7162594d5abd6d2f5a99af217c0aa4b2df6db3ad221ee67285aba38"}
Oct 30 21:12:35.679598 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:35.679526    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-390b71-5698dc795b-2hp76"
Oct 30 21:12:35.792548 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:35.792491    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-390b71-7ff679864c-ghxkc" podStartSLOduration=2.792477971 podStartE2EDuration="2.792477971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:12:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:12:35.791129483 +0000 UTC m=+2038.855905884" watchObservedRunningTime="2026-10-30 21:12:35.792477971 +0000 UTC m=+2038.857254369"
Oct 30 21:12:37.065015 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:37.064978    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1afb9a0a-9a10-4f71-b66d-a7cc873ed48b" path="/var/lib/kubelet/pods/1afb9a0a-9a10-4f71-b66d-a7cc873ed48b/volumes"
Oct 30 21:12:37.065536 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:37.065513    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9200d55-faeb-4087-85cb-949f64353a55" path="/var/lib/kubelet/pods/e9200d55-faeb-4087-85cb-949f64353a55/volumes"
Oct 30 21:12:39.676473 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:39.676417    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-390b71-5698dc795b-2hp76" podStartSLOduration=6.676402694 podStartE2EDuration="6.676402694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:12:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:12:35.920653508 +0000 UTC m=+2038.985429908" watchObservedRunningTime="2026-10-30 21:12:39.676402694 +0000 UTC m=+2042.741179093"
Oct 30 21:12:39.680219 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:39.680000    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f3b172-x79t7"]
Oct 30 21:12:39.680831 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:39.680812    5713 cpu_manager.go:414] "containerMap: removing container" podUID="e9200d55-faeb-4087-85cb-949f64353a55" containerName="payload-pre-processing"
Oct 30 21:12:39.680831 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:39.680831    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9200d55-faeb-4087-85cb-949f64353a55" containerName="payload-pre-processing"
Oct 30 21:12:39.680979 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:39.680857    5713 cpu_manager.go:414] "containerMap: removing container" podUID="1afb9a0a-9a10-4f71-b66d-a7cc873ed48b" containerName="payload-processing"
Oct 30 21:12:39.680979 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:39.680863    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1afb9a0a-9a10-4f71-b66d-a7cc873ed48b" containerName="payload-processing"
Oct 30 21:12:39.681093 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:39.681016    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9200d55-faeb-4087-85cb-949f64353a55" containerName="payload-pre-processing"
Oct 30 21:12:39.681093 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:39.681034    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="1afb9a0a-9a10-4f71-b66d-a7cc873ed48b" containerName="payload-processing"
Oct 30 21:12:39.686637 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:39.686615    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f3b172-x79t7"
Oct 30 21:12:39.744328 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:39.744296    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f3b172-x79t7"]
Oct 30 21:12:39.785109 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:39.785084    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:12:39.785277 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:39.785138    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:12:39.785277 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:39.785185    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:12:39.788521 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:39.788500    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:12:39.811748 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:39.811705    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0aad42c1-6667-4c8f-bb7f-236eb9bc0898-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f3b172-x79t7\" (UID: \"0aad42c1-6667-4c8f-bb7f-236eb9bc0898\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f3b172-x79t7"
Oct 30 21:12:39.812020 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:39.811964    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0aad42c1-6667-4c8f-bb7f-236eb9bc0898-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f3b172-x79t7\" (UID: \"0aad42c1-6667-4c8f-bb7f-236eb9bc0898\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f3b172-x79t7"
Oct 30 21:12:39.812108 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:39.812053    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x225v\" (UniqueName: \"kubernetes.io/projected/0aad42c1-6667-4c8f-bb7f-236eb9bc0898-kube-api-access-x225v\") pod \"maas-api-revoke-keys-e2e-inf-a-f3b172-x79t7\" (UID: \"0aad42c1-6667-4c8f-bb7f-236eb9bc0898\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f3b172-x79t7"
Oct 30 21:12:39.914074 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:39.914035    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0aad42c1-6667-4c8f-bb7f-236eb9bc0898-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f3b172-x79t7\" (UID: \"0aad42c1-6667-4c8f-bb7f-236eb9bc0898\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f3b172-x79t7"
Oct 30 21:12:39.914242 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:39.914192    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0aad42c1-6667-4c8f-bb7f-236eb9bc0898-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f3b172-x79t7\" (UID: \"0aad42c1-6667-4c8f-bb7f-236eb9bc0898\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f3b172-x79t7"
Oct 30 21:12:39.914321 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:39.914240    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x225v\" (UniqueName: \"kubernetes.io/projected/0aad42c1-6667-4c8f-bb7f-236eb9bc0898-kube-api-access-x225v\") pod \"maas-api-revoke-keys-e2e-inf-a-f3b172-x79t7\" (UID: \"0aad42c1-6667-4c8f-bb7f-236eb9bc0898\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f3b172-x79t7"
Oct 30 21:12:39.914905 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:39.914875    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0aad42c1-6667-4c8f-bb7f-236eb9bc0898-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f3b172-x79t7\" (UID: \"0aad42c1-6667-4c8f-bb7f-236eb9bc0898\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f3b172-x79t7"
Oct 30 21:12:39.921841 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:39.918773    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0aad42c1-6667-4c8f-bb7f-236eb9bc0898-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f3b172-x79t7\" (UID: \"0aad42c1-6667-4c8f-bb7f-236eb9bc0898\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f3b172-x79t7"
Oct 30 21:12:39.973812 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:39.972284    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x225v\" (UniqueName: \"kubernetes.io/projected/0aad42c1-6667-4c8f-bb7f-236eb9bc0898-kube-api-access-x225v\") pod \"maas-api-revoke-keys-e2e-inf-a-f3b172-x79t7\" (UID: \"0aad42c1-6667-4c8f-bb7f-236eb9bc0898\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f3b172-x79t7"
Oct 30 21:12:40.008059 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:40.008001    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f3b172-x79t7"
Oct 30 21:12:40.401733 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:12:40.401701    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aad42c1_6667_4c8f_bb7f_236eb9bc0898.slice/crio-aa2eaadb5143eab19824ab6c0de684f045fd2d80f5bdb490233c6ba65af0c5e7 WatchSource:0}: Error finding container aa2eaadb5143eab19824ab6c0de684f045fd2d80f5bdb490233c6ba65af0c5e7: Status 404 returned error can't find the container with id aa2eaadb5143eab19824ab6c0de684f045fd2d80f5bdb490233c6ba65af0c5e7
Oct 30 21:12:40.401884 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:40.401836    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f3b172-x79t7"]
Oct 30 21:12:40.723693 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:40.723655    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f3b172-x79t7" event={"ID":"0aad42c1-6667-4c8f-bb7f-236eb9bc0898","Type":"ContainerStarted","Data":"aa2eaadb5143eab19824ab6c0de684f045fd2d80f5bdb490233c6ba65af0c5e7"}
Oct 30 21:12:41.733849 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:41.733770    5713 generic.go:356] "Generic (PLEG): container finished" podID="0aad42c1-6667-4c8f-bb7f-236eb9bc0898" containerID="daf7235867bc845f263697e98affeb8e91200330cbecfd0e7edc653b9d5c41ae" exitCode=0
Oct 30 21:12:41.733849 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:41.733822    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f3b172-x79t7" event={"ID":"0aad42c1-6667-4c8f-bb7f-236eb9bc0898","Type":"ContainerDied","Data":"daf7235867bc845f263697e98affeb8e91200330cbecfd0e7edc653b9d5c41ae"}
Oct 30 21:12:42.913680 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:42.913656    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f3b172-x79t7"
Oct 30 21:12:42.973510 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:42.973477    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0aad42c1-6667-4c8f-bb7f-236eb9bc0898-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0aad42c1-6667-4c8f-bb7f-236eb9bc0898-maas-api-openshift-ca\") pod \"0aad42c1-6667-4c8f-bb7f-236eb9bc0898\" (UID: \"0aad42c1-6667-4c8f-bb7f-236eb9bc0898\") "
Oct 30 21:12:42.973701 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:42.973551    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0aad42c1-6667-4c8f-bb7f-236eb9bc0898-kube-api-access-x225v\" (UniqueName: \"kubernetes.io/projected/0aad42c1-6667-4c8f-bb7f-236eb9bc0898-kube-api-access-x225v\") pod \"0aad42c1-6667-4c8f-bb7f-236eb9bc0898\" (UID: \"0aad42c1-6667-4c8f-bb7f-236eb9bc0898\") "
Oct 30 21:12:42.973916 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:42.973893    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0aad42c1-6667-4c8f-bb7f-236eb9bc0898-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0aad42c1-6667-4c8f-bb7f-236eb9bc0898-maas-api-xks-ca\") pod \"0aad42c1-6667-4c8f-bb7f-236eb9bc0898\" (UID: \"0aad42c1-6667-4c8f-bb7f-236eb9bc0898\") "
Oct 30 21:12:42.973989 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:42.973919    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0aad42c1-6667-4c8f-bb7f-236eb9bc0898-maas-api-openshift-ca" pod "0aad42c1-6667-4c8f-bb7f-236eb9bc0898" (UID: "0aad42c1-6667-4c8f-bb7f-236eb9bc0898"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:12:42.974597 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:42.974516    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0aad42c1-6667-4c8f-bb7f-236eb9bc0898-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:42.976006 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:42.975975    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aad42c1-6667-4c8f-bb7f-236eb9bc0898-kube-api-access-x225v" pod "0aad42c1-6667-4c8f-bb7f-236eb9bc0898" (UID: "0aad42c1-6667-4c8f-bb7f-236eb9bc0898"). InnerVolumeSpecName "kube-api-access-x225v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:12:42.976131 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:42.976072    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0aad42c1-6667-4c8f-bb7f-236eb9bc0898-maas-api-xks-ca" pod "0aad42c1-6667-4c8f-bb7f-236eb9bc0898" (UID: "0aad42c1-6667-4c8f-bb7f-236eb9bc0898"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:12:43.076006 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:43.075921    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x225v\" (UniqueName: \"kubernetes.io/projected/0aad42c1-6667-4c8f-bb7f-236eb9bc0898-kube-api-access-x225v\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:43.076006 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:43.075961    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0aad42c1-6667-4c8f-bb7f-236eb9bc0898-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:43.751817 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:43.751761    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f3b172-x79t7"
Oct 30 21:12:43.752019 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:43.751832    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f3b172-x79t7" event={"ID":"0aad42c1-6667-4c8f-bb7f-236eb9bc0898","Type":"ContainerDied","Data":"aa2eaadb5143eab19824ab6c0de684f045fd2d80f5bdb490233c6ba65af0c5e7"}
Oct 30 21:12:43.752019 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:43.751892    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa2eaadb5143eab19824ab6c0de684f045fd2d80f5bdb490233c6ba65af0c5e7"
Oct 30 21:12:46.690163 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:46.690135    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-390b71-7ff679864c-ghxkc"
Oct 30 21:12:46.690578 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:46.690200    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-390b71-5698dc795b-2hp76"
Oct 30 21:12:51.193273 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:51.193239    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f3b172-5c85644f98-5wgnl"]
Oct 30 21:12:51.303535 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:51.300462    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f3b172-79c58fc975-d9k5m"]
Oct 30 21:12:54.461473 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:12:54.461428    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:12:56.193838 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:56.193780    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-f3b172-5c85644f98-5wgnl" podUID="c9bb0c3d-cf42-4f2b-9dde-b16b890cddab" containerName="payload-processing" containerID="cri-o://a7dcbc295d693e3eff75509312e957e6f4e328b17bcd9550744516ddb9611e12" gracePeriod=25
Oct 30 21:12:56.301688 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:56.301639    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3b172-79c58fc975-d9k5m" podUID="6ed86aa9-530c-4122-99c9-23c7f28c2b08" containerName="payload-pre-processing" containerID="cri-o://6a355a95f6e2ff2347dc8601c9a28335b87bf791563f28d7d32a49f3f6353713" gracePeriod=25
Oct 30 21:12:56.476569 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:56.476542    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f3b172-5c85644f98-5wgnl"
Oct 30 21:12:56.601946 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:56.601920    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3b172-79c58fc975-d9k5m"
Oct 30 21:12:56.611538 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:56.611502    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9bb0c3d-cf42-4f2b-9dde-b16b890cddab-kube-api-access-s4ctd\" (UniqueName: \"kubernetes.io/projected/c9bb0c3d-cf42-4f2b-9dde-b16b890cddab-kube-api-access-s4ctd\") pod \"c9bb0c3d-cf42-4f2b-9dde-b16b890cddab\" (UID: \"c9bb0c3d-cf42-4f2b-9dde-b16b890cddab\") "
Oct 30 21:12:56.611723 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:56.611553    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9bb0c3d-cf42-4f2b-9dde-b16b890cddab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9bb0c3d-cf42-4f2b-9dde-b16b890cddab-plugins-config-volume\") pod \"c9bb0c3d-cf42-4f2b-9dde-b16b890cddab\" (UID: \"c9bb0c3d-cf42-4f2b-9dde-b16b890cddab\") "
Oct 30 21:12:56.611991 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:56.611962    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9bb0c3d-cf42-4f2b-9dde-b16b890cddab-plugins-config-volume" pod "c9bb0c3d-cf42-4f2b-9dde-b16b890cddab" (UID: "c9bb0c3d-cf42-4f2b-9dde-b16b890cddab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:12:56.612502 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:56.612476    5713 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9bb0c3d-cf42-4f2b-9dde-b16b890cddab-plugins-config-volume\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:56.614007 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:56.613985    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9bb0c3d-cf42-4f2b-9dde-b16b890cddab-kube-api-access-s4ctd" pod "c9bb0c3d-cf42-4f2b-9dde-b16b890cddab" (UID: "c9bb0c3d-cf42-4f2b-9dde-b16b890cddab"). InnerVolumeSpecName "kube-api-access-s4ctd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:12:56.713862 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:56.713723    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ed86aa9-530c-4122-99c9-23c7f28c2b08-kube-api-access-n98fn\" (UniqueName: \"kubernetes.io/projected/6ed86aa9-530c-4122-99c9-23c7f28c2b08-kube-api-access-n98fn\") pod \"6ed86aa9-530c-4122-99c9-23c7f28c2b08\" (UID: \"6ed86aa9-530c-4122-99c9-23c7f28c2b08\") "
Oct 30 21:12:56.714107 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:56.714074    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ed86aa9-530c-4122-99c9-23c7f28c2b08-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ed86aa9-530c-4122-99c9-23c7f28c2b08-plugins-config-volume\") pod \"6ed86aa9-530c-4122-99c9-23c7f28c2b08\" (UID: \"6ed86aa9-530c-4122-99c9-23c7f28c2b08\") "
Oct 30 21:12:56.715505 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:56.714705    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ed86aa9-530c-4122-99c9-23c7f28c2b08-plugins-config-volume" pod "6ed86aa9-530c-4122-99c9-23c7f28c2b08" (UID: "6ed86aa9-530c-4122-99c9-23c7f28c2b08"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:12:56.715505 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:56.715451    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4ctd\" (UniqueName: \"kubernetes.io/projected/c9bb0c3d-cf42-4f2b-9dde-b16b890cddab-kube-api-access-s4ctd\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:56.715505 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:56.715479    5713 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ed86aa9-530c-4122-99c9-23c7f28c2b08-plugins-config-volume\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:56.717483 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:56.717457    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ed86aa9-530c-4122-99c9-23c7f28c2b08-kube-api-access-n98fn" pod "6ed86aa9-530c-4122-99c9-23c7f28c2b08" (UID: "6ed86aa9-530c-4122-99c9-23c7f28c2b08"). InnerVolumeSpecName "kube-api-access-n98fn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:12:56.816669 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:56.816633    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n98fn\" (UniqueName: \"kubernetes.io/projected/6ed86aa9-530c-4122-99c9-23c7f28c2b08-kube-api-access-n98fn\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:12:56.894031 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:56.893997    5713 generic.go:356] "Generic (PLEG): container finished" podID="c9bb0c3d-cf42-4f2b-9dde-b16b890cddab" containerID="a7dcbc295d693e3eff75509312e957e6f4e328b17bcd9550744516ddb9611e12" exitCode=0
Oct 30 21:12:56.894235 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:56.894047    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f3b172-5c85644f98-5wgnl"
Oct 30 21:12:56.894235 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:56.894085    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f3b172-5c85644f98-5wgnl" event={"ID":"c9bb0c3d-cf42-4f2b-9dde-b16b890cddab","Type":"ContainerDied","Data":"a7dcbc295d693e3eff75509312e957e6f4e328b17bcd9550744516ddb9611e12"}
Oct 30 21:12:56.894235 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:56.894143    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f3b172-5c85644f98-5wgnl" event={"ID":"c9bb0c3d-cf42-4f2b-9dde-b16b890cddab","Type":"ContainerDied","Data":"df84c5f8038486244f71377a8bdf694657df828da733a35e971b3defe60cc4cc"}
Oct 30 21:12:56.894235 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:56.894176    5713 scope.go:122] "RemoveContainer" containerID="a7dcbc295d693e3eff75509312e957e6f4e328b17bcd9550744516ddb9611e12"
Oct 30 21:12:56.895749 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:56.895726    5713 generic.go:356] "Generic (PLEG): container finished" podID="6ed86aa9-530c-4122-99c9-23c7f28c2b08" containerID="6a355a95f6e2ff2347dc8601c9a28335b87bf791563f28d7d32a49f3f6353713" exitCode=0
Oct 30 21:12:56.895867 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:56.895777    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3b172-79c58fc975-d9k5m"
Oct 30 21:12:56.895938 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:56.895906    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3b172-79c58fc975-d9k5m" event={"ID":"6ed86aa9-530c-4122-99c9-23c7f28c2b08","Type":"ContainerDied","Data":"6a355a95f6e2ff2347dc8601c9a28335b87bf791563f28d7d32a49f3f6353713"}
Oct 30 21:12:56.895991 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:56.895938    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f3b172-79c58fc975-d9k5m" event={"ID":"6ed86aa9-530c-4122-99c9-23c7f28c2b08","Type":"ContainerDied","Data":"a6ef9d0d1bd8c96c0d0f54a78b7ef31d8852b0427c57dba3c0eec914bd93372e"}
Oct 30 21:12:56.906291 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:56.906269    5713 scope.go:122] "RemoveContainer" containerID="a7dcbc295d693e3eff75509312e957e6f4e328b17bcd9550744516ddb9611e12"
Oct 30 21:12:56.906637 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:12:56.906601    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7dcbc295d693e3eff75509312e957e6f4e328b17bcd9550744516ddb9611e12\": container with ID starting with a7dcbc295d693e3eff75509312e957e6f4e328b17bcd9550744516ddb9611e12 not found: ID does not exist" containerID="a7dcbc295d693e3eff75509312e957e6f4e328b17bcd9550744516ddb9611e12"
Oct 30 21:12:56.906700 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:56.906649    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7dcbc295d693e3eff75509312e957e6f4e328b17bcd9550744516ddb9611e12"} err="failed to get container status \"a7dcbc295d693e3eff75509312e957e6f4e328b17bcd9550744516ddb9611e12\": rpc error: code = NotFound desc = could not find container \"a7dcbc295d693e3eff75509312e957e6f4e328b17bcd9550744516ddb9611e12\": container with ID starting with a7dcbc295d693e3eff75509312e957e6f4e328b17bcd9550744516ddb9611e12 not found: ID does not exist"
Oct 30 21:12:56.906700 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:56.906675    5713 scope.go:122] "RemoveContainer" containerID="6a355a95f6e2ff2347dc8601c9a28335b87bf791563f28d7d32a49f3f6353713"
Oct 30 21:12:56.918726 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:56.918702    5713 scope.go:122] "RemoveContainer" containerID="6a355a95f6e2ff2347dc8601c9a28335b87bf791563f28d7d32a49f3f6353713"
Oct 30 21:12:56.919091 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:12:56.919071    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a355a95f6e2ff2347dc8601c9a28335b87bf791563f28d7d32a49f3f6353713\": container with ID starting with 6a355a95f6e2ff2347dc8601c9a28335b87bf791563f28d7d32a49f3f6353713 not found: ID does not exist" containerID="6a355a95f6e2ff2347dc8601c9a28335b87bf791563f28d7d32a49f3f6353713"
Oct 30 21:12:56.919157 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:56.919100    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a355a95f6e2ff2347dc8601c9a28335b87bf791563f28d7d32a49f3f6353713"} err="failed to get container status \"6a355a95f6e2ff2347dc8601c9a28335b87bf791563f28d7d32a49f3f6353713\": rpc error: code = NotFound desc = could not find container \"6a355a95f6e2ff2347dc8601c9a28335b87bf791563f28d7d32a49f3f6353713\": container with ID starting with 6a355a95f6e2ff2347dc8601c9a28335b87bf791563f28d7d32a49f3f6353713 not found: ID does not exist"
Oct 30 21:12:57.078439 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:57.078358    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f3b172-79c58fc975-d9k5m"]
Oct 30 21:12:57.081519 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:57.081491    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f3b172-79c58fc975-d9k5m"]
Oct 30 21:12:57.185399 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:57.185365    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f3b172-5c85644f98-5wgnl"]
Oct 30 21:12:57.190117 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:57.190084    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f3b172-5c85644f98-5wgnl"]
Oct 30 21:12:59.067305 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:59.067269    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ed86aa9-530c-4122-99c9-23c7f28c2b08" path="/var/lib/kubelet/pods/6ed86aa9-530c-4122-99c9-23c7f28c2b08/volumes"
Oct 30 21:12:59.067939 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:12:59.067918    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9bb0c3d-cf42-4f2b-9dde-b16b890cddab" path="/var/lib/kubelet/pods/c9bb0c3d-cf42-4f2b-9dde-b16b890cddab/volumes"
Oct 30 21:13:01.549281 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:01.549244    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9926f7-8ldbl"]
Oct 30 21:13:01.599755 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:01.599716    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9926f7-8ldbl"]
Oct 30 21:13:02.718286 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:02.718256    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f3b172-x79t7"]
Oct 30 21:13:02.767740 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:02.767702    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f3b172-x79t7"]
Oct 30 21:13:03.065846 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:03.065754    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0aad42c1-6667-4c8f-bb7f-236eb9bc0898" path="/var/lib/kubelet/pods/0aad42c1-6667-4c8f-bb7f-236eb9bc0898/volumes"
Oct 30 21:13:03.066383 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:03.066357    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c7c5226-001b-4e5f-90ee-a0392597425d" path="/var/lib/kubelet/pods/0c7c5226-001b-4e5f-90ee-a0392597425d/volumes"
Oct 30 21:13:08.368423 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:08.368387    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q"]
Oct 30 21:13:08.368917 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:08.368602    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q" podUID="ec28cb7b-925f-4b32-9ff8-d75caf2f3d14" containerName="istio-proxy" containerID="cri-o://38a6d001c0c0e7a02be45091eb6e21fc7f7b4f8e5e66e31c875c39424edb5f7e" gracePeriod=30
Oct 30 21:13:11.188041 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:11.188007    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g"]
Oct 30 21:13:11.188956 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:11.188933    5713 cpu_manager.go:414] "containerMap: removing container" podUID="c9bb0c3d-cf42-4f2b-9dde-b16b890cddab" containerName="payload-processing"
Oct 30 21:13:11.189078 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:11.188956    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9bb0c3d-cf42-4f2b-9dde-b16b890cddab" containerName="payload-processing"
Oct 30 21:13:11.189078 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:11.189002    5713 cpu_manager.go:414] "containerMap: removing container" podUID="0aad42c1-6667-4c8f-bb7f-236eb9bc0898" containerName="revoke-keys"
Oct 30 21:13:11.189078 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:11.189010    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0aad42c1-6667-4c8f-bb7f-236eb9bc0898" containerName="revoke-keys"
Oct 30 21:13:11.189078 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:11.189051    5713 cpu_manager.go:414] "containerMap: removing container" podUID="6ed86aa9-530c-4122-99c9-23c7f28c2b08" containerName="payload-pre-processing"
Oct 30 21:13:11.189078 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:11.189060    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ed86aa9-530c-4122-99c9-23c7f28c2b08" containerName="payload-pre-processing"
Oct 30 21:13:11.189358 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:11.189225    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ed86aa9-530c-4122-99c9-23c7f28c2b08" containerName="payload-pre-processing"
Oct 30 21:13:11.189358 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:11.189247    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9bb0c3d-cf42-4f2b-9dde-b16b890cddab" containerName="payload-processing"
Oct 30 21:13:11.189358 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:11.189263    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="0aad42c1-6667-4c8f-bb7f-236eb9bc0898" containerName="revoke-keys"
Oct 30 21:13:11.194503 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:11.194479    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g"
Oct 30 21:13:11.251987 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:11.251956    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-2f9eff-openshift-default-dockercfg-zcthz\""
Oct 30 21:13:11.285032 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:11.284992    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94f9b9e7-7097-43d9-870e-98b67f32d0e9-istio-podinfo\") pod \"e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g\" (UID: \"94f9b9e7-7097-43d9-870e-98b67f32d0e9\") " pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g"
Oct 30 21:13:11.285180 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:11.285074    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94f9b9e7-7097-43d9-870e-98b67f32d0e9-istio-envoy\") pod \"e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g\" (UID: \"94f9b9e7-7097-43d9-870e-98b67f32d0e9\") " pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g"
Oct 30 21:13:11.285265 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:11.285223    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f9b9e7-7097-43d9-870e-98b67f32d0e9-workload-socket\") pod \"e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g\" (UID: \"94f9b9e7-7097-43d9-870e-98b67f32d0e9\") " pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g"
Oct 30 21:13:11.285328 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:11.285266    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sdmw\" (UniqueName: \"kubernetes.io/projected/94f9b9e7-7097-43d9-870e-98b67f32d0e9-kube-api-access-8sdmw\") pod \"e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g\" (UID: \"94f9b9e7-7097-43d9-870e-98b67f32d0e9\") " pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g"
Oct 30 21:13:11.285386 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:11.285347    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94f9b9e7-7097-43d9-870e-98b67f32d0e9-istio-token\") pod \"e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g\" (UID: \"94f9b9e7-7097-43d9-870e-98b67f32d0e9\") " pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g"
Oct 30 21:13:11.285652 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:11.285606    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94f9b9e7-7097-43d9-870e-98b67f32d0e9-istiod-ca-cert\") pod \"e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g\" (UID: \"94f9b9e7-7097-43d9-870e-98b67f32d0e9\") " pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g"
Oct 30 21:13:11.286651 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:11.286625    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94f9b9e7-7097-43d9-870e-98b67f32d0e9-workload-certs\") pod \"e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g\" (UID: \"94f9b9e7-7097-43d9-870e-98b67f32d0e9\") " pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g"
Oct 30 21:13:11.286769 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:11.286751    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f9b9e7-7097-43d9-870e-98b67f32d0e9-credential-socket\") pod \"e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g\" (UID: \"94f9b9e7-7097-43d9-870e-98b67f32d0e9\") " pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g"
Oct 30 21:13:11.286873 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:11.286853    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94f9b9e7-7097-43d9-870e-98b67f32d0e9-istio-data\") pod \"e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g\" (UID: \"94f9b9e7-7097-43d9-870e-98b67f32d0e9\") " pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g"
Oct 30 21:13:11.424406 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:11.424364    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94f9b9e7-7097-43d9-870e-98b67f32d0e9-istiod-ca-cert\") pod \"e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g\" (UID: \"94f9b9e7-7097-43d9-870e-98b67f32d0e9\") " pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g"
Oct 30 21:13:11.476574 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:11.476490    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94f9b9e7-7097-43d9-870e-98b67f32d0e9-workload-certs\") pod \"e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g\" (UID: \"94f9b9e7-7097-43d9-870e-98b67f32d0e9\") " pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g"
Oct 30 21:13:11.477582 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:11.477553    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f9b9e7-7097-43d9-870e-98b67f32d0e9-credential-socket\") pod \"e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g\" (UID: \"94f9b9e7-7097-43d9-870e-98b67f32d0e9\") " pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g"
Oct 30 21:13:11.479057 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:11.479033    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94f9b9e7-7097-43d9-870e-98b67f32d0e9-istio-data\") pod \"e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g\" (UID: \"94f9b9e7-7097-43d9-870e-98b67f32d0e9\") " pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g"
Oct 30 21:13:11.479710 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:11.479690    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94f9b9e7-7097-43d9-870e-98b67f32d0e9-istio-podinfo\") pod \"e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g\" (UID: \"94f9b9e7-7097-43d9-870e-98b67f32d0e9\") " pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g"
Oct 30 21:13:11.480428 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:11.473677    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94f9b9e7-7097-43d9-870e-98b67f32d0e9-istiod-ca-cert\") pod \"e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g\" (UID: \"94f9b9e7-7097-43d9-870e-98b67f32d0e9\") " pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g"
Oct 30 21:13:11.480536 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:11.478931    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g"]
Oct 30 21:13:11.480536 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:11.477131    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94f9b9e7-7097-43d9-870e-98b67f32d0e9-workload-certs\") pod \"e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g\" (UID: \"94f9b9e7-7097-43d9-870e-98b67f32d0e9\") " pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g"
Oct 30 21:13:11.480536 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:11.479469    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94f9b9e7-7097-43d9-870e-98b67f32d0e9-istio-data\") pod \"e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g\" (UID: \"94f9b9e7-7097-43d9-870e-98b67f32d0e9\") " pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g"
Oct 30 21:13:11.480536 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:11.478088    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f9b9e7-7097-43d9-870e-98b67f32d0e9-credential-socket\") pod \"e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g\" (UID: \"94f9b9e7-7097-43d9-870e-98b67f32d0e9\") " pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g"
Oct 30 21:13:11.493895 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:11.493855    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94f9b9e7-7097-43d9-870e-98b67f32d0e9-istio-podinfo\") pod \"e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g\" (UID: \"94f9b9e7-7097-43d9-870e-98b67f32d0e9\") " pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g"
Oct 30 21:13:11.519851 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:11.519811    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94f9b9e7-7097-43d9-870e-98b67f32d0e9-istio-envoy\") pod \"e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g\" (UID: \"94f9b9e7-7097-43d9-870e-98b67f32d0e9\") " pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g"
Oct 30 21:13:11.520052 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:11.519864    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f9b9e7-7097-43d9-870e-98b67f32d0e9-workload-socket\") pod \"e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g\" (UID: \"94f9b9e7-7097-43d9-870e-98b67f32d0e9\") " pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g"
Oct 30 21:13:11.520052 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:11.519890    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sdmw\" (UniqueName: \"kubernetes.io/projected/94f9b9e7-7097-43d9-870e-98b67f32d0e9-kube-api-access-8sdmw\") pod \"e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g\" (UID: \"94f9b9e7-7097-43d9-870e-98b67f32d0e9\") " pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g"
Oct 30 21:13:11.520052 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:11.519917    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94f9b9e7-7097-43d9-870e-98b67f32d0e9-istio-token\") pod \"e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g\" (UID: \"94f9b9e7-7097-43d9-870e-98b67f32d0e9\") " pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g"
Oct 30 21:13:11.522227 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:11.521644    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f9b9e7-7097-43d9-870e-98b67f32d0e9-workload-socket\") pod \"e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g\" (UID: \"94f9b9e7-7097-43d9-870e-98b67f32d0e9\") " pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g"
Oct 30 21:13:11.525336 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:11.525304    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94f9b9e7-7097-43d9-870e-98b67f32d0e9-istio-envoy\") pod \"e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g\" (UID: \"94f9b9e7-7097-43d9-870e-98b67f32d0e9\") " pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g"
Oct 30 21:13:11.578334 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:11.578304    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94f9b9e7-7097-43d9-870e-98b67f32d0e9-istio-token\") pod \"e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g\" (UID: \"94f9b9e7-7097-43d9-870e-98b67f32d0e9\") " pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g"
Oct 30 21:13:11.579023 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:11.578997    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sdmw\" (UniqueName: \"kubernetes.io/projected/94f9b9e7-7097-43d9-870e-98b67f32d0e9-kube-api-access-8sdmw\") pod \"e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g\" (UID: \"94f9b9e7-7097-43d9-870e-98b67f32d0e9\") " pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g"
Oct 30 21:13:11.818210 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:11.818130    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g"
Oct 30 21:13:12.285225 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:12.285196    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g"]
Oct 30 21:13:12.286527 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:13:12.286500    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94f9b9e7_7097_43d9_870e_98b67f32d0e9.slice/crio-fc4f92a1df47f58de375cfca934ce992b8812af965c71c70e7a883fe5f199d25 WatchSource:0}: Error finding container fc4f92a1df47f58de375cfca934ce992b8812af965c71c70e7a883fe5f199d25: Status 404 returned error can't find the container with id fc4f92a1df47f58de375cfca934ce992b8812af965c71c70e7a883fe5f199d25
Oct 30 21:13:12.289242 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:12.289210    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:13:12.289328 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:12.289276    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:13:12.289328 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:12.289304    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:13:12.380363 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:12.380331    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-de53c2-kb4k4"]
Oct 30 21:13:12.385299 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:12.385278    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-de53c2-kb4k4"
Oct 30 21:13:12.439359 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:12.439283    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:13:12.439359 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:12.439322    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:13:12.443220 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:12.443193    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-de53c2-kb4k4"]
Oct 30 21:13:12.481074 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:12.481046    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:13:12.481272 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:12.481109    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:13:12.542677 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:12.542592    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l469\" (UniqueName: \"kubernetes.io/projected/2be77ed8-0e5a-4d1a-aef1-6675a140bdf0-kube-api-access-2l469\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-de53c2-kb4k4\" (UID: \"2be77ed8-0e5a-4d1a-aef1-6675a140bdf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-de53c2-kb4k4"
Oct 30 21:13:12.542856 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:12.542729    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2be77ed8-0e5a-4d1a-aef1-6675a140bdf0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-de53c2-kb4k4\" (UID: \"2be77ed8-0e5a-4d1a-aef1-6675a140bdf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-de53c2-kb4k4"
Oct 30 21:13:12.542923 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:12.542907    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2be77ed8-0e5a-4d1a-aef1-6675a140bdf0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-de53c2-kb4k4\" (UID: \"2be77ed8-0e5a-4d1a-aef1-6675a140bdf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-de53c2-kb4k4"
Oct 30 21:13:12.645143 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:12.644821    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2be77ed8-0e5a-4d1a-aef1-6675a140bdf0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-de53c2-kb4k4\" (UID: \"2be77ed8-0e5a-4d1a-aef1-6675a140bdf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-de53c2-kb4k4"
Oct 30 21:13:12.645143 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:12.645066    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2be77ed8-0e5a-4d1a-aef1-6675a140bdf0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-de53c2-kb4k4\" (UID: \"2be77ed8-0e5a-4d1a-aef1-6675a140bdf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-de53c2-kb4k4"
Oct 30 21:13:12.645412 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:12.645268    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l469\" (UniqueName: \"kubernetes.io/projected/2be77ed8-0e5a-4d1a-aef1-6675a140bdf0-kube-api-access-2l469\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-de53c2-kb4k4\" (UID: \"2be77ed8-0e5a-4d1a-aef1-6675a140bdf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-de53c2-kb4k4"
Oct 30 21:13:12.647425 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:12.646374    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2be77ed8-0e5a-4d1a-aef1-6675a140bdf0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-de53c2-kb4k4\" (UID: \"2be77ed8-0e5a-4d1a-aef1-6675a140bdf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-de53c2-kb4k4"
Oct 30 21:13:12.654711 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:12.654641    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2be77ed8-0e5a-4d1a-aef1-6675a140bdf0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-de53c2-kb4k4\" (UID: \"2be77ed8-0e5a-4d1a-aef1-6675a140bdf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-de53c2-kb4k4"
Oct 30 21:13:12.704482 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:12.704448    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l469\" (UniqueName: \"kubernetes.io/projected/2be77ed8-0e5a-4d1a-aef1-6675a140bdf0-kube-api-access-2l469\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-de53c2-kb4k4\" (UID: \"2be77ed8-0e5a-4d1a-aef1-6675a140bdf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-de53c2-kb4k4"
Oct 30 21:13:13.000346 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:13.000313    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-de53c2-kb4k4"
Oct 30 21:13:13.066323 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:13.066287    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g" event={"ID":"94f9b9e7-7097-43d9-870e-98b67f32d0e9","Type":"ContainerStarted","Data":"b38d889798e8ba005656c437fe5daa846aacd07bb171168a11444965343e6eab"}
Oct 30 21:13:13.066323 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:13.066326    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g" event={"ID":"94f9b9e7-7097-43d9-870e-98b67f32d0e9","Type":"ContainerStarted","Data":"fc4f92a1df47f58de375cfca934ce992b8812af965c71c70e7a883fe5f199d25"}
Oct 30 21:13:13.202390 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:13.202329    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g" podStartSLOduration=2.20231418 podStartE2EDuration="2.20231418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:13:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:13.197284482 +0000 UTC m=+2076.262060880" watchObservedRunningTime="2026-10-30 21:13:13.20231418 +0000 UTC m=+2076.267090655"
Oct 30 21:13:13.438251 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:13.438209    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-de53c2-kb4k4"]
Oct 30 21:13:13.818532 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:13.818497    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g"
Oct 30 21:13:13.824547 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:13.824518    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g"
Oct 30 21:13:14.074421 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:14.074322    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-de53c2-kb4k4" event={"ID":"2be77ed8-0e5a-4d1a-aef1-6675a140bdf0","Type":"ContainerStarted","Data":"e5fa43c9eb690b03c8703384dd6783b631004f0484a1852091ae327a18fd2c1a"}
Oct 30 21:13:14.074421 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:14.074395    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-de53c2-kb4k4" event={"ID":"2be77ed8-0e5a-4d1a-aef1-6675a140bdf0","Type":"ContainerStarted","Data":"7b3f9018051351d7c5ae47ce133d0b5e2b0ced7e7c7bd475418aaf7921079c2d"}
Oct 30 21:13:14.077106 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:14.077076    5713 generic.go:356] "Generic (PLEG): container finished" podID="ec28cb7b-925f-4b32-9ff8-d75caf2f3d14" containerID="38a6d001c0c0e7a02be45091eb6e21fc7f7b4f8e5e66e31c875c39424edb5f7e" exitCode=0
Oct 30 21:13:14.078263 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:14.078242    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q" event={"ID":"ec28cb7b-925f-4b32-9ff8-d75caf2f3d14","Type":"ContainerDied","Data":"38a6d001c0c0e7a02be45091eb6e21fc7f7b4f8e5e66e31c875c39424edb5f7e"}
Oct 30 21:13:14.078385 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:14.078288    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q" event={"ID":"ec28cb7b-925f-4b32-9ff8-d75caf2f3d14","Type":"ContainerDied","Data":"674d527d0732fa066e00a978fb6b289f3b5b3ebd50f28d16724b43c7129937be"}
Oct 30 21:13:14.078385 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:14.078304    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="674d527d0732fa066e00a978fb6b289f3b5b3ebd50f28d16724b43c7129937be"
Oct 30 21:13:14.078385 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:14.078317    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g"
Oct 30 21:13:14.078385 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:14.078337    5713 scope.go:122] "RemoveContainer" containerID="00705fe9f3bf9f96294b219bba02b81f3ba219ce426d7b3825bfee73c1b9f2d7"
Oct 30 21:13:14.079556 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:14.079534    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g"
Oct 30 21:13:14.091651 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:14.091628    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q"
Oct 30 21:13:14.190811 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:14.190581    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ec28cb7b-925f-4b32-9ff8-d75caf2f3d14-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec28cb7b-925f-4b32-9ff8-d75caf2f3d14-istio-podinfo\") pod \"ec28cb7b-925f-4b32-9ff8-d75caf2f3d14\" (UID: \"ec28cb7b-925f-4b32-9ff8-d75caf2f3d14\") "
Oct 30 21:13:14.190811 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:14.190629    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec28cb7b-925f-4b32-9ff8-d75caf2f3d14-kube-api-access-4gklj\" (UniqueName: \"kubernetes.io/projected/ec28cb7b-925f-4b32-9ff8-d75caf2f3d14-kube-api-access-4gklj\") pod \"ec28cb7b-925f-4b32-9ff8-d75caf2f3d14\" (UID: \"ec28cb7b-925f-4b32-9ff8-d75caf2f3d14\") "
Oct 30 21:13:14.190811 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:14.190669    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec28cb7b-925f-4b32-9ff8-d75caf2f3d14-istio-token\" (UniqueName: \"kubernetes.io/projected/ec28cb7b-925f-4b32-9ff8-d75caf2f3d14-istio-token\") pod \"ec28cb7b-925f-4b32-9ff8-d75caf2f3d14\" (UID: \"ec28cb7b-925f-4b32-9ff8-d75caf2f3d14\") "
Oct 30 21:13:14.190811 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:14.190735    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec28cb7b-925f-4b32-9ff8-d75caf2f3d14-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec28cb7b-925f-4b32-9ff8-d75caf2f3d14-workload-socket\") pod \"ec28cb7b-925f-4b32-9ff8-d75caf2f3d14\" (UID: \"ec28cb7b-925f-4b32-9ff8-d75caf2f3d14\") "
Oct 30 21:13:14.191247 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:14.190918    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec28cb7b-925f-4b32-9ff8-d75caf2f3d14-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec28cb7b-925f-4b32-9ff8-d75caf2f3d14-istio-data\") pod \"ec28cb7b-925f-4b32-9ff8-d75caf2f3d14\" (UID: \"ec28cb7b-925f-4b32-9ff8-d75caf2f3d14\") "
Oct 30 21:13:14.191247 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:14.190985    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec28cb7b-925f-4b32-9ff8-d75caf2f3d14-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec28cb7b-925f-4b32-9ff8-d75caf2f3d14-credential-socket\") pod \"ec28cb7b-925f-4b32-9ff8-d75caf2f3d14\" (UID: \"ec28cb7b-925f-4b32-9ff8-d75caf2f3d14\") "
Oct 30 21:13:14.191247 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:14.191089    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec28cb7b-925f-4b32-9ff8-d75caf2f3d14-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec28cb7b-925f-4b32-9ff8-d75caf2f3d14-istio-envoy\") pod \"ec28cb7b-925f-4b32-9ff8-d75caf2f3d14\" (UID: \"ec28cb7b-925f-4b32-9ff8-d75caf2f3d14\") "
Oct 30 21:13:14.191247 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:14.191193    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec28cb7b-925f-4b32-9ff8-d75caf2f3d14-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec28cb7b-925f-4b32-9ff8-d75caf2f3d14-istiod-ca-cert\") pod \"ec28cb7b-925f-4b32-9ff8-d75caf2f3d14\" (UID: \"ec28cb7b-925f-4b32-9ff8-d75caf2f3d14\") "
Oct 30 21:13:14.191481 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:14.191282    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec28cb7b-925f-4b32-9ff8-d75caf2f3d14-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec28cb7b-925f-4b32-9ff8-d75caf2f3d14-workload-certs\") pod \"ec28cb7b-925f-4b32-9ff8-d75caf2f3d14\" (UID: \"ec28cb7b-925f-4b32-9ff8-d75caf2f3d14\") "
Oct 30 21:13:14.209685 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:14.209648    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec28cb7b-925f-4b32-9ff8-d75caf2f3d14-credential-socket" pod "ec28cb7b-925f-4b32-9ff8-d75caf2f3d14" (UID: "ec28cb7b-925f-4b32-9ff8-d75caf2f3d14"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:13:14.212022 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:14.211580    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec28cb7b-925f-4b32-9ff8-d75caf2f3d14-kube-api-access-4gklj" pod "ec28cb7b-925f-4b32-9ff8-d75caf2f3d14" (UID: "ec28cb7b-925f-4b32-9ff8-d75caf2f3d14"). InnerVolumeSpecName "kube-api-access-4gklj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:13:14.230869 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:14.228254    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec28cb7b-925f-4b32-9ff8-d75caf2f3d14-istiod-ca-cert" pod "ec28cb7b-925f-4b32-9ff8-d75caf2f3d14" (UID: "ec28cb7b-925f-4b32-9ff8-d75caf2f3d14"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:13:14.230869 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:14.228498    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec28cb7b-925f-4b32-9ff8-d75caf2f3d14-workload-certs" pod "ec28cb7b-925f-4b32-9ff8-d75caf2f3d14" (UID: "ec28cb7b-925f-4b32-9ff8-d75caf2f3d14"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:13:14.230869 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:14.228657    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec28cb7b-925f-4b32-9ff8-d75caf2f3d14-workload-socket" pod "ec28cb7b-925f-4b32-9ff8-d75caf2f3d14" (UID: "ec28cb7b-925f-4b32-9ff8-d75caf2f3d14"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:13:14.230869 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:14.228864    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec28cb7b-925f-4b32-9ff8-d75caf2f3d14-istio-data" pod "ec28cb7b-925f-4b32-9ff8-d75caf2f3d14" (UID: "ec28cb7b-925f-4b32-9ff8-d75caf2f3d14"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:13:14.231438 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:14.231381    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-de53c2-kb4k4" podStartSLOduration=2.231363637 podStartE2EDuration="2.231363637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:13:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:14.228244388 +0000 UTC m=+2077.293020789" watchObservedRunningTime="2026-10-30 21:13:14.231363637 +0000 UTC m=+2077.296140036"
Oct 30 21:13:14.231936 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:14.231909    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec28cb7b-925f-4b32-9ff8-d75caf2f3d14-istio-token" pod "ec28cb7b-925f-4b32-9ff8-d75caf2f3d14" (UID: "ec28cb7b-925f-4b32-9ff8-d75caf2f3d14"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:13:14.232513 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:14.232471    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec28cb7b-925f-4b32-9ff8-d75caf2f3d14-istio-podinfo" pod "ec28cb7b-925f-4b32-9ff8-d75caf2f3d14" (UID: "ec28cb7b-925f-4b32-9ff8-d75caf2f3d14"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:13:14.250391 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:14.250347    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec28cb7b-925f-4b32-9ff8-d75caf2f3d14-istio-envoy" pod "ec28cb7b-925f-4b32-9ff8-d75caf2f3d14" (UID: "ec28cb7b-925f-4b32-9ff8-d75caf2f3d14"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:13:14.252621 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:14.252464    5713 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec28cb7b-925f-4b32-9ff8-d75caf2f3d14-istiod-ca-cert\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:13:14.252997 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:14.252973    5713 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec28cb7b-925f-4b32-9ff8-d75caf2f3d14-workload-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:13:14.253148 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:14.253133    5713 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec28cb7b-925f-4b32-9ff8-d75caf2f3d14-istio-podinfo\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:13:14.254744 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:14.253246    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gklj\" (UniqueName: \"kubernetes.io/projected/ec28cb7b-925f-4b32-9ff8-d75caf2f3d14-kube-api-access-4gklj\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:13:14.254942 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:14.254922    5713 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec28cb7b-925f-4b32-9ff8-d75caf2f3d14-istio-token\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:13:14.256677 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:14.256659    5713 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec28cb7b-925f-4b32-9ff8-d75caf2f3d14-workload-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:13:14.256842 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:14.256769    5713 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec28cb7b-925f-4b32-9ff8-d75caf2f3d14-istio-data\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:13:14.256842 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:14.256819    5713 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec28cb7b-925f-4b32-9ff8-d75caf2f3d14-credential-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:13:14.257048 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:14.256852    5713 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec28cb7b-925f-4b32-9ff8-d75caf2f3d14-istio-envoy\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:13:15.087091 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:15.087061    5713 generic.go:356] "Generic (PLEG): container finished" podID="2be77ed8-0e5a-4d1a-aef1-6675a140bdf0" containerID="e5fa43c9eb690b03c8703384dd6783b631004f0484a1852091ae327a18fd2c1a" exitCode=0
Oct 30 21:13:15.087510 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:15.087132    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-de53c2-kb4k4" event={"ID":"2be77ed8-0e5a-4d1a-aef1-6675a140bdf0","Type":"ContainerDied","Data":"e5fa43c9eb690b03c8703384dd6783b631004f0484a1852091ae327a18fd2c1a"}
Oct 30 21:13:15.088671 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:15.088652    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q"
Oct 30 21:13:15.402217 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:15.402125    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q"]
Oct 30 21:13:15.406399 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:15.406372    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-9926f7-openshift-default-665f488bff-qgl7q"]
Oct 30 21:13:16.265202 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:16.265169    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-de53c2-kb4k4"
Oct 30 21:13:16.388587 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:16.388504    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2be77ed8-0e5a-4d1a-aef1-6675a140bdf0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2be77ed8-0e5a-4d1a-aef1-6675a140bdf0-maas-api-xks-ca\") pod \"2be77ed8-0e5a-4d1a-aef1-6675a140bdf0\" (UID: \"2be77ed8-0e5a-4d1a-aef1-6675a140bdf0\") "
Oct 30 21:13:16.388587 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:16.388569    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2be77ed8-0e5a-4d1a-aef1-6675a140bdf0-kube-api-access-2l469\" (UniqueName: \"kubernetes.io/projected/2be77ed8-0e5a-4d1a-aef1-6675a140bdf0-kube-api-access-2l469\") pod \"2be77ed8-0e5a-4d1a-aef1-6675a140bdf0\" (UID: \"2be77ed8-0e5a-4d1a-aef1-6675a140bdf0\") "
Oct 30 21:13:16.388870 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:16.388710    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2be77ed8-0e5a-4d1a-aef1-6675a140bdf0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2be77ed8-0e5a-4d1a-aef1-6675a140bdf0-maas-api-openshift-ca\") pod \"2be77ed8-0e5a-4d1a-aef1-6675a140bdf0\" (UID: \"2be77ed8-0e5a-4d1a-aef1-6675a140bdf0\") "
Oct 30 21:13:16.389132 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:16.389106    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2be77ed8-0e5a-4d1a-aef1-6675a140bdf0-maas-api-openshift-ca" pod "2be77ed8-0e5a-4d1a-aef1-6675a140bdf0" (UID: "2be77ed8-0e5a-4d1a-aef1-6675a140bdf0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:13:16.389529 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:16.389509    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2be77ed8-0e5a-4d1a-aef1-6675a140bdf0-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:13:16.390823 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:16.390799    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2be77ed8-0e5a-4d1a-aef1-6675a140bdf0-maas-api-xks-ca" pod "2be77ed8-0e5a-4d1a-aef1-6675a140bdf0" (UID: "2be77ed8-0e5a-4d1a-aef1-6675a140bdf0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:13:16.390931 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:16.390879    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2be77ed8-0e5a-4d1a-aef1-6675a140bdf0-kube-api-access-2l469" pod "2be77ed8-0e5a-4d1a-aef1-6675a140bdf0" (UID: "2be77ed8-0e5a-4d1a-aef1-6675a140bdf0"). InnerVolumeSpecName "kube-api-access-2l469". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:13:16.490647 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:16.490576    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2be77ed8-0e5a-4d1a-aef1-6675a140bdf0-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:13:16.490647 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:16.490604    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2l469\" (UniqueName: \"kubernetes.io/projected/2be77ed8-0e5a-4d1a-aef1-6675a140bdf0-kube-api-access-2l469\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:13:17.066168 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:17.066131    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec28cb7b-925f-4b32-9ff8-d75caf2f3d14" path="/var/lib/kubelet/pods/ec28cb7b-925f-4b32-9ff8-d75caf2f3d14/volumes"
Oct 30 21:13:17.105895 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:17.105855    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-de53c2-kb4k4" event={"ID":"2be77ed8-0e5a-4d1a-aef1-6675a140bdf0","Type":"ContainerDied","Data":"7b3f9018051351d7c5ae47ce133d0b5e2b0ced7e7c7bd475418aaf7921079c2d"}
Oct 30 21:13:17.106079 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:17.105908    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b3f9018051351d7c5ae47ce133d0b5e2b0ced7e7c7bd475418aaf7921079c2d"
Oct 30 21:13:17.106079 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:17.105917    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-de53c2-kb4k4"
Oct 30 21:13:19.781747 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:19.781701    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2f9eff-7796bfd547-fkj5q"]
Oct 30 21:13:19.782600 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:19.782577    5713 cpu_manager.go:414] "containerMap: removing container" podUID="ec28cb7b-925f-4b32-9ff8-d75caf2f3d14" containerName="istio-proxy"
Oct 30 21:13:19.782600 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:19.782599    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec28cb7b-925f-4b32-9ff8-d75caf2f3d14" containerName="istio-proxy"
Oct 30 21:13:19.782764 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:19.782623    5713 cpu_manager.go:414] "containerMap: removing container" podUID="ec28cb7b-925f-4b32-9ff8-d75caf2f3d14" containerName="istio-proxy"
Oct 30 21:13:19.782764 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:19.782632    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec28cb7b-925f-4b32-9ff8-d75caf2f3d14" containerName="istio-proxy"
Oct 30 21:13:19.782764 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:19.782649    5713 cpu_manager.go:414] "containerMap: removing container" podUID="2be77ed8-0e5a-4d1a-aef1-6675a140bdf0" containerName="revoke-keys"
Oct 30 21:13:19.782764 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:19.782655    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2be77ed8-0e5a-4d1a-aef1-6675a140bdf0" containerName="revoke-keys"
Oct 30 21:13:19.783039 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:19.782776    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec28cb7b-925f-4b32-9ff8-d75caf2f3d14" containerName="istio-proxy"
Oct 30 21:13:19.783039 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:19.782809    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="2be77ed8-0e5a-4d1a-aef1-6675a140bdf0" containerName="revoke-keys"
Oct 30 21:13:19.787362 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:19.787336    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f9eff-7796bfd547-fkj5q"
Oct 30 21:13:19.844928 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:19.844895    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-2f9eff\""
Oct 30 21:13:19.916945 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:19.916908    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-2f9eff-dockercfg-8j92c\""
Oct 30 21:13:19.960821 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:19.954053    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51a91785-6750-4b1a-8174-4598816ea82d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2f9eff-7796bfd547-fkj5q\" (UID: \"51a91785-6750-4b1a-8174-4598816ea82d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f9eff-7796bfd547-fkj5q"
Oct 30 21:13:19.960821 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:19.954177    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttrv8\" (UniqueName: \"kubernetes.io/projected/51a91785-6750-4b1a-8174-4598816ea82d-kube-api-access-ttrv8\") pod \"payload-pre-processing-e2e-inf-a-2f9eff-7796bfd547-fkj5q\" (UID: \"51a91785-6750-4b1a-8174-4598816ea82d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f9eff-7796bfd547-fkj5q"
Oct 30 21:13:19.960821 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:19.955877    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2f9eff-7796bfd547-fkj5q"]
Oct 30 21:13:20.087904 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:20.083424    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2f9eff-689ff796b8-sf94t"]
Oct 30 21:13:20.087904 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:20.084708    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec28cb7b-925f-4b32-9ff8-d75caf2f3d14" containerName="istio-proxy"
Oct 30 21:13:20.098061 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:20.095074    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2f9eff-689ff796b8-sf94t"
Oct 30 21:13:20.110946 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:20.110658    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51a91785-6750-4b1a-8174-4598816ea82d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2f9eff-7796bfd547-fkj5q\" (UID: \"51a91785-6750-4b1a-8174-4598816ea82d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f9eff-7796bfd547-fkj5q"
Oct 30 21:13:20.110946 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:20.110819    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttrv8\" (UniqueName: \"kubernetes.io/projected/51a91785-6750-4b1a-8174-4598816ea82d-kube-api-access-ttrv8\") pod \"payload-pre-processing-e2e-inf-a-2f9eff-7796bfd547-fkj5q\" (UID: \"51a91785-6750-4b1a-8174-4598816ea82d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f9eff-7796bfd547-fkj5q"
Oct 30 21:13:20.120052 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:20.113695    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51a91785-6750-4b1a-8174-4598816ea82d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2f9eff-7796bfd547-fkj5q\" (UID: \"51a91785-6750-4b1a-8174-4598816ea82d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f9eff-7796bfd547-fkj5q"
Oct 30 21:13:20.124561 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:20.123751    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2f9eff-689ff796b8-sf94t"]
Oct 30 21:13:20.178130 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:20.178098    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttrv8\" (UniqueName: \"kubernetes.io/projected/51a91785-6750-4b1a-8174-4598816ea82d-kube-api-access-ttrv8\") pod \"payload-pre-processing-e2e-inf-a-2f9eff-7796bfd547-fkj5q\" (UID: \"51a91785-6750-4b1a-8174-4598816ea82d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f9eff-7796bfd547-fkj5q"
Oct 30 21:13:20.212477 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:20.212447    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/102176e5-2bda-4e24-b86a-0d50fba75ac3-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2f9eff-689ff796b8-sf94t\" (UID: \"102176e5-2bda-4e24-b86a-0d50fba75ac3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2f9eff-689ff796b8-sf94t"
Oct 30 21:13:20.212477 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:20.212482    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd5qq\" (UniqueName: \"kubernetes.io/projected/102176e5-2bda-4e24-b86a-0d50fba75ac3-kube-api-access-zd5qq\") pod \"payload-processing-e2e-inf-a-2f9eff-689ff796b8-sf94t\" (UID: \"102176e5-2bda-4e24-b86a-0d50fba75ac3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2f9eff-689ff796b8-sf94t"
Oct 30 21:13:20.313946 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:20.313914    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/102176e5-2bda-4e24-b86a-0d50fba75ac3-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2f9eff-689ff796b8-sf94t\" (UID: \"102176e5-2bda-4e24-b86a-0d50fba75ac3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2f9eff-689ff796b8-sf94t"
Oct 30 21:13:20.313946 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:20.313951    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd5qq\" (UniqueName: \"kubernetes.io/projected/102176e5-2bda-4e24-b86a-0d50fba75ac3-kube-api-access-zd5qq\") pod \"payload-processing-e2e-inf-a-2f9eff-689ff796b8-sf94t\" (UID: \"102176e5-2bda-4e24-b86a-0d50fba75ac3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2f9eff-689ff796b8-sf94t"
Oct 30 21:13:20.314692 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:20.314674    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/102176e5-2bda-4e24-b86a-0d50fba75ac3-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2f9eff-689ff796b8-sf94t\" (UID: \"102176e5-2bda-4e24-b86a-0d50fba75ac3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2f9eff-689ff796b8-sf94t"
Oct 30 21:13:20.377843 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:20.377736    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd5qq\" (UniqueName: \"kubernetes.io/projected/102176e5-2bda-4e24-b86a-0d50fba75ac3-kube-api-access-zd5qq\") pod \"payload-processing-e2e-inf-a-2f9eff-689ff796b8-sf94t\" (UID: \"102176e5-2bda-4e24-b86a-0d50fba75ac3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2f9eff-689ff796b8-sf94t"
Oct 30 21:13:20.411661 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:20.411626    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f9eff-7796bfd547-fkj5q"
Oct 30 21:13:20.435212 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:20.435181    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2f9eff-689ff796b8-sf94t"
Oct 30 21:13:20.808193 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:20.808160    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2f9eff-7796bfd547-fkj5q"]
Oct 30 21:13:20.851401 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:20.851376    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2f9eff-689ff796b8-sf94t"]
Oct 30 21:13:20.851558 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:13:20.851540    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod102176e5_2bda_4e24_b86a_0d50fba75ac3.slice/crio-2283a57982a9589e62767c12d0dd42433c9cf89231e50ab0a898c2e5e2965d34 WatchSource:0}: Error finding container 2283a57982a9589e62767c12d0dd42433c9cf89231e50ab0a898c2e5e2965d34: Status 404 returned error can't find the container with id 2283a57982a9589e62767c12d0dd42433c9cf89231e50ab0a898c2e5e2965d34
Oct 30 21:13:21.145630 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:21.145532    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2f9eff-689ff796b8-sf94t" event={"ID":"102176e5-2bda-4e24-b86a-0d50fba75ac3","Type":"ContainerStarted","Data":"14f252049ea00e1ae4dae8092ecc9b3f8a97b53841993edef17577a08d6ee1b2"}
Oct 30 21:13:21.145630 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:21.145597    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2f9eff-689ff796b8-sf94t" event={"ID":"102176e5-2bda-4e24-b86a-0d50fba75ac3","Type":"ContainerStarted","Data":"2283a57982a9589e62767c12d0dd42433c9cf89231e50ab0a898c2e5e2965d34"}
Oct 30 21:13:21.145630 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:21.145627    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-2f9eff-689ff796b8-sf94t"
Oct 30 21:13:21.147351 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:21.147324    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f9eff-7796bfd547-fkj5q" event={"ID":"51a91785-6750-4b1a-8174-4598816ea82d","Type":"ContainerStarted","Data":"0ca83bbf41038832964465ad63185b5410f84cabaf8611b8c8678222565da7f7"}
Oct 30 21:13:21.147482 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:21.147368    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f9eff-7796bfd547-fkj5q" event={"ID":"51a91785-6750-4b1a-8174-4598816ea82d","Type":"ContainerStarted","Data":"0d88c2b32b38dbab0703e640b63828d0f8034ba0ce357586eddcbf5a92ae0305"}
Oct 30 21:13:21.147482 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:21.147390    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f9eff-7796bfd547-fkj5q"
Oct 30 21:13:21.265223 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:21.265170    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-2f9eff-689ff796b8-sf94t" podStartSLOduration=2.265153939 podStartE2EDuration="2.265153939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:13:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:21.261215874 +0000 UTC m=+2084.325992273" watchObservedRunningTime="2026-10-30 21:13:21.265153939 +0000 UTC m=+2084.329930336"
Oct 30 21:13:21.378616 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:21.378556    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f9eff-7796bfd547-fkj5q" podStartSLOduration=2.378541666 podStartE2EDuration="2.378541666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:13:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:21.375694942 +0000 UTC m=+2084.440471355" watchObservedRunningTime="2026-10-30 21:13:21.378541666 +0000 UTC m=+2084.443318064"
Oct 30 21:13:24.459479 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:13:24.459445    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:13:32.160157 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:32.160123    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-2f9eff-689ff796b8-sf94t"
Oct 30 21:13:32.160673 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:32.160510    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f9eff-7796bfd547-fkj5q"
Oct 30 21:13:33.070239 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:33.070202    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40b7b51a-8dtkf"]
Oct 30 21:13:33.111843 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:33.111801    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40b7b51a-8dtkf"]
Oct 30 21:13:35.065641 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:35.065605    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3da62021-1436-4278-9434-26b7d9feea6b" path="/var/lib/kubelet/pods/3da62021-1436-4278-9434-26b7d9feea6b/volumes"
Oct 30 21:13:37.547541 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:37.547503    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d4597073-8ff0-4a4e-933c-c64e934fe977/prometheus/0.log"
Oct 30 21:13:37.553364 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:37.553340    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d4597073-8ff0-4a4e-933c-c64e934fe977/prometheus/0.log"
Oct 30 21:13:37.562928 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:37.562906    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r7lpf_764461d5-e0b2-4856-a97e-0c5d3d34a497/ovn-acl-logging/0.log"
Oct 30 21:13:37.568824 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:37.568799    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r7lpf_764461d5-e0b2-4856-a97e-0c5d3d34a497/ovn-acl-logging/0.log"
Oct 30 21:13:38.736509 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:38.736478    5713 scope.go:122] "RemoveContainer" containerID="44959020f09c0a00a00ef1a891c7de7e2c79e96055404ff92a99ec10c9f987f5"
Oct 30 21:13:38.747647 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:38.747622    5713 scope.go:122] "RemoveContainer" containerID="763a2733f60e8ca6ca8111e5ae1da1f6bdc71c6e49d31a67fc45e13bc6015eed"
Oct 30 21:13:39.256696 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:39.256659    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b888d0-trcw2"]
Oct 30 21:13:39.262875 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:39.262857    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b888d0-trcw2"
Oct 30 21:13:39.317914 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:39.317881    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b888d0-trcw2"]
Oct 30 21:13:39.349649 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:39.349610    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/059f5085-59af-4a89-97ac-890c70f071b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b888d0-trcw2\" (UID: \"059f5085-59af-4a89-97ac-890c70f071b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b888d0-trcw2"
Oct 30 21:13:39.349834 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:39.349668    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5jll\" (UniqueName: \"kubernetes.io/projected/059f5085-59af-4a89-97ac-890c70f071b1-kube-api-access-m5jll\") pod \"maas-api-revoke-keys-e2e-api-a-b888d0-trcw2\" (UID: \"059f5085-59af-4a89-97ac-890c70f071b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b888d0-trcw2"
Oct 30 21:13:39.349902 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:39.349844    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/059f5085-59af-4a89-97ac-890c70f071b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b888d0-trcw2\" (UID: \"059f5085-59af-4a89-97ac-890c70f071b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b888d0-trcw2"
Oct 30 21:13:39.362215 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:39.362156    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:13:39.362215 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:39.362178    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:13:39.362373 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:39.362247    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:13:39.366895 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:39.366880    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:13:39.451874 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:39.451835    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/059f5085-59af-4a89-97ac-890c70f071b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b888d0-trcw2\" (UID: \"059f5085-59af-4a89-97ac-890c70f071b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b888d0-trcw2"
Oct 30 21:13:39.452098 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:39.451891    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5jll\" (UniqueName: \"kubernetes.io/projected/059f5085-59af-4a89-97ac-890c70f071b1-kube-api-access-m5jll\") pod \"maas-api-revoke-keys-e2e-api-a-b888d0-trcw2\" (UID: \"059f5085-59af-4a89-97ac-890c70f071b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b888d0-trcw2"
Oct 30 21:13:39.452098 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:39.452023    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/059f5085-59af-4a89-97ac-890c70f071b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b888d0-trcw2\" (UID: \"059f5085-59af-4a89-97ac-890c70f071b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b888d0-trcw2"
Oct 30 21:13:39.452706 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:39.452686    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/059f5085-59af-4a89-97ac-890c70f071b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b888d0-trcw2\" (UID: \"059f5085-59af-4a89-97ac-890c70f071b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b888d0-trcw2"
Oct 30 21:13:39.454498 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:39.454474    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/059f5085-59af-4a89-97ac-890c70f071b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b888d0-trcw2\" (UID: \"059f5085-59af-4a89-97ac-890c70f071b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b888d0-trcw2"
Oct 30 21:13:39.509184 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:39.509118    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5jll\" (UniqueName: \"kubernetes.io/projected/059f5085-59af-4a89-97ac-890c70f071b1-kube-api-access-m5jll\") pod \"maas-api-revoke-keys-e2e-api-a-b888d0-trcw2\" (UID: \"059f5085-59af-4a89-97ac-890c70f071b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b888d0-trcw2"
Oct 30 21:13:39.577359 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:39.577318    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b888d0-trcw2"
Oct 30 21:13:39.965233 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:13:39.965197    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod059f5085_59af_4a89_97ac_890c70f071b1.slice/crio-9d6103f8f1f2a1c6917616ec2fc3aa5f22ad35b09a203342c26672cb91c64110 WatchSource:0}: Error finding container 9d6103f8f1f2a1c6917616ec2fc3aa5f22ad35b09a203342c26672cb91c64110: Status 404 returned error can't find the container with id 9d6103f8f1f2a1c6917616ec2fc3aa5f22ad35b09a203342c26672cb91c64110
Oct 30 21:13:39.966856 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:39.966773    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b888d0-trcw2"]
Oct 30 21:13:40.318845 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:40.318742    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b888d0-trcw2" event={"ID":"059f5085-59af-4a89-97ac-890c70f071b1","Type":"ContainerStarted","Data":"9d6103f8f1f2a1c6917616ec2fc3aa5f22ad35b09a203342c26672cb91c64110"}
Oct 30 21:13:41.328103 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:41.328060    5713 generic.go:356] "Generic (PLEG): container finished" podID="059f5085-59af-4a89-97ac-890c70f071b1" containerID="6605deeeed0b9baa20e27dd93a524617b9639d5e3ee1b7b2a16155e04ae80859" exitCode=0
Oct 30 21:13:41.328512 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:41.328139    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b888d0-trcw2" event={"ID":"059f5085-59af-4a89-97ac-890c70f071b1","Type":"ContainerDied","Data":"6605deeeed0b9baa20e27dd93a524617b9639d5e3ee1b7b2a16155e04ae80859"}
Oct 30 21:13:42.508148 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:42.508122    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b888d0-trcw2"
Oct 30 21:13:42.702153 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:42.702122    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/059f5085-59af-4a89-97ac-890c70f071b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/059f5085-59af-4a89-97ac-890c70f071b1-maas-api-openshift-ca\") pod \"059f5085-59af-4a89-97ac-890c70f071b1\" (UID: \"059f5085-59af-4a89-97ac-890c70f071b1\") "
Oct 30 21:13:42.702330 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:42.702205    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/059f5085-59af-4a89-97ac-890c70f071b1-kube-api-access-m5jll\" (UniqueName: \"kubernetes.io/projected/059f5085-59af-4a89-97ac-890c70f071b1-kube-api-access-m5jll\") pod \"059f5085-59af-4a89-97ac-890c70f071b1\" (UID: \"059f5085-59af-4a89-97ac-890c70f071b1\") "
Oct 30 21:13:42.702437 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:42.702415    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/059f5085-59af-4a89-97ac-890c70f071b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/059f5085-59af-4a89-97ac-890c70f071b1-maas-api-xks-ca\") pod \"059f5085-59af-4a89-97ac-890c70f071b1\" (UID: \"059f5085-59af-4a89-97ac-890c70f071b1\") "
Oct 30 21:13:42.702533 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:42.702513    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/059f5085-59af-4a89-97ac-890c70f071b1-maas-api-openshift-ca" pod "059f5085-59af-4a89-97ac-890c70f071b1" (UID: "059f5085-59af-4a89-97ac-890c70f071b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:13:42.703082 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:42.703061    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/059f5085-59af-4a89-97ac-890c70f071b1-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:13:42.704576 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:42.704552    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/059f5085-59af-4a89-97ac-890c70f071b1-maas-api-xks-ca" pod "059f5085-59af-4a89-97ac-890c70f071b1" (UID: "059f5085-59af-4a89-97ac-890c70f071b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:13:42.704658 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:42.704629    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/059f5085-59af-4a89-97ac-890c70f071b1-kube-api-access-m5jll" pod "059f5085-59af-4a89-97ac-890c70f071b1" (UID: "059f5085-59af-4a89-97ac-890c70f071b1"). InnerVolumeSpecName "kube-api-access-m5jll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:13:42.804981 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:42.804941    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5jll\" (UniqueName: \"kubernetes.io/projected/059f5085-59af-4a89-97ac-890c70f071b1-kube-api-access-m5jll\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:13:42.804981 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:42.804976    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/059f5085-59af-4a89-97ac-890c70f071b1-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:13:43.345117 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:43.345040    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b888d0-trcw2"
Oct 30 21:13:43.345117 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:43.345072    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b888d0-trcw2" event={"ID":"059f5085-59af-4a89-97ac-890c70f071b1","Type":"ContainerDied","Data":"9d6103f8f1f2a1c6917616ec2fc3aa5f22ad35b09a203342c26672cb91c64110"}
Oct 30 21:13:43.345300 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:43.345125    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d6103f8f1f2a1c6917616ec2fc3aa5f22ad35b09a203342c26672cb91c64110"
Oct 30 21:13:46.772737 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:46.772700    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-dc6125-4n8cr"]
Oct 30 21:13:46.822346 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:46.822311    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-dc6125-4n8cr"]
Oct 30 21:13:47.065855 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:47.065762    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="089ae08c-aa75-4c9b-ab51-70684e9ba0a1" path="/var/lib/kubelet/pods/089ae08c-aa75-4c9b-ab51-70684e9ba0a1/volumes"
Oct 30 21:13:51.503224 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:51.503165    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b888d0-79557999cd-rzz2r"]
Oct 30 21:13:51.616900 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:51.616865    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b888d0-75cbbbf7c5-rnfbj"]
Oct 30 21:13:53.906919 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:53.906885    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-de53c2-kb4k4"]
Oct 30 21:13:53.956290 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:53.956242    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-de53c2-kb4k4"]
Oct 30 21:13:54.460440 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:13:54.460402    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:13:55.066075 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:55.066045    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2be77ed8-0e5a-4d1a-aef1-6675a140bdf0" path="/var/lib/kubelet/pods/2be77ed8-0e5a-4d1a-aef1-6675a140bdf0/volumes"
Oct 30 21:13:55.210600 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:55.210565    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef7e2d-rqhqc"]
Oct 30 21:13:55.259267 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:55.259237    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef7e2d-rqhqc"]
Oct 30 21:13:56.511938 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:56.511901    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-b888d0-79557999cd-rzz2r" podUID="fbd6d24d-5227-4534-8fe8-1288fef79e88" containerName="payload-processing" containerID="cri-o://358ecb28cf28938ce27fbeae78493d6cc8dcac24f619f217e356572da4728a2c" gracePeriod=25
Oct 30 21:13:56.598888 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:56.598850    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-7eaf74-b6mtd"]
Oct 30 21:13:56.599576 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:56.599561    5713 cpu_manager.go:414] "containerMap: removing container" podUID="059f5085-59af-4a89-97ac-890c70f071b1" containerName="revoke-keys"
Oct 30 21:13:56.599625 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:56.599578    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="059f5085-59af-4a89-97ac-890c70f071b1" containerName="revoke-keys"
Oct 30 21:13:56.599713 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:56.599703    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="059f5085-59af-4a89-97ac-890c70f071b1" containerName="revoke-keys"
Oct 30 21:13:56.603538 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:56.603517    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-7eaf74-b6mtd"
Oct 30 21:13:56.617543 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:56.617506    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b888d0-75cbbbf7c5-rnfbj" podUID="8335fd09-7966-4952-8214-1e569d15f4fa" containerName="payload-pre-processing" containerID="cri-o://e260163d30b28d68dc8894d67faef810fc5c9f95afc79d7b92a1657ba94c01a7" gracePeriod=25
Oct 30 21:13:56.659115 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:56.659080    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:13:56.675763 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:56.675712    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjf5j\" (UniqueName: \"kubernetes.io/projected/9364652a-228c-4e15-ac17-6d26d75befad-kube-api-access-hjf5j\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-7eaf74-b6mtd\" (UID: \"9364652a-228c-4e15-ac17-6d26d75befad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-7eaf74-b6mtd"
Oct 30 21:13:56.677360 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:56.675857    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9364652a-228c-4e15-ac17-6d26d75befad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-7eaf74-b6mtd\" (UID: \"9364652a-228c-4e15-ac17-6d26d75befad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-7eaf74-b6mtd"
Oct 30 21:13:56.677360 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:56.676153    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9364652a-228c-4e15-ac17-6d26d75befad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-7eaf74-b6mtd\" (UID: \"9364652a-228c-4e15-ac17-6d26d75befad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-7eaf74-b6mtd"
Oct 30 21:13:56.709777 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:56.709740    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:13:56.709995 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:56.709865    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:13:56.717049 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:56.717017    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-7eaf74-b6mtd"]
Oct 30 21:13:56.717211 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:56.717193    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:13:56.778126 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:56.778040    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjf5j\" (UniqueName: \"kubernetes.io/projected/9364652a-228c-4e15-ac17-6d26d75befad-kube-api-access-hjf5j\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-7eaf74-b6mtd\" (UID: \"9364652a-228c-4e15-ac17-6d26d75befad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-7eaf74-b6mtd"
Oct 30 21:13:56.778126 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:56.778104    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9364652a-228c-4e15-ac17-6d26d75befad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-7eaf74-b6mtd\" (UID: \"9364652a-228c-4e15-ac17-6d26d75befad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-7eaf74-b6mtd"
Oct 30 21:13:56.778374 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:56.778321    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9364652a-228c-4e15-ac17-6d26d75befad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-7eaf74-b6mtd\" (UID: \"9364652a-228c-4e15-ac17-6d26d75befad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-7eaf74-b6mtd"
Oct 30 21:13:56.779647 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:56.779613    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9364652a-228c-4e15-ac17-6d26d75befad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-7eaf74-b6mtd\" (UID: \"9364652a-228c-4e15-ac17-6d26d75befad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-7eaf74-b6mtd"
Oct 30 21:13:56.781579 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:56.781546    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9364652a-228c-4e15-ac17-6d26d75befad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-7eaf74-b6mtd\" (UID: \"9364652a-228c-4e15-ac17-6d26d75befad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-7eaf74-b6mtd"
Oct 30 21:13:56.831814 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:56.829371    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b888d0-79557999cd-rzz2r"
Oct 30 21:13:56.840000 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:56.836638    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjf5j\" (UniqueName: \"kubernetes.io/projected/9364652a-228c-4e15-ac17-6d26d75befad-kube-api-access-hjf5j\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-7eaf74-b6mtd\" (UID: \"9364652a-228c-4e15-ac17-6d26d75befad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-7eaf74-b6mtd"
Oct 30 21:13:56.879527 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:56.879497    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbd6d24d-5227-4534-8fe8-1288fef79e88-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbd6d24d-5227-4534-8fe8-1288fef79e88-plugins-config-volume\") pod \"fbd6d24d-5227-4534-8fe8-1288fef79e88\" (UID: \"fbd6d24d-5227-4534-8fe8-1288fef79e88\") "
Oct 30 21:13:56.879658 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:56.879581    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbd6d24d-5227-4534-8fe8-1288fef79e88-kube-api-access-twdbl\" (UniqueName: \"kubernetes.io/projected/fbd6d24d-5227-4534-8fe8-1288fef79e88-kube-api-access-twdbl\") pod \"fbd6d24d-5227-4534-8fe8-1288fef79e88\" (UID: \"fbd6d24d-5227-4534-8fe8-1288fef79e88\") "
Oct 30 21:13:56.879967 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:56.879938    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbd6d24d-5227-4534-8fe8-1288fef79e88-plugins-config-volume" pod "fbd6d24d-5227-4534-8fe8-1288fef79e88" (UID: "fbd6d24d-5227-4534-8fe8-1288fef79e88"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:13:56.880742 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:56.880715    5713 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbd6d24d-5227-4534-8fe8-1288fef79e88-plugins-config-volume\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:13:56.882179 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:56.882146    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbd6d24d-5227-4534-8fe8-1288fef79e88-kube-api-access-twdbl" pod "fbd6d24d-5227-4534-8fe8-1288fef79e88" (UID: "fbd6d24d-5227-4534-8fe8-1288fef79e88"). InnerVolumeSpecName "kube-api-access-twdbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:13:56.890234 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:56.890213    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b888d0-75cbbbf7c5-rnfbj"
Oct 30 21:13:56.918454 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:56.918423    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-7eaf74-b6mtd"
Oct 30 21:13:56.988479 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:56.988446    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8335fd09-7966-4952-8214-1e569d15f4fa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8335fd09-7966-4952-8214-1e569d15f4fa-plugins-config-volume\") pod \"8335fd09-7966-4952-8214-1e569d15f4fa\" (UID: \"8335fd09-7966-4952-8214-1e569d15f4fa\") "
Oct 30 21:13:56.988614 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:56.988599    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8335fd09-7966-4952-8214-1e569d15f4fa-kube-api-access-ppjz9\" (UniqueName: \"kubernetes.io/projected/8335fd09-7966-4952-8214-1e569d15f4fa-kube-api-access-ppjz9\") pod \"8335fd09-7966-4952-8214-1e569d15f4fa\" (UID: \"8335fd09-7966-4952-8214-1e569d15f4fa\") "
Oct 30 21:13:56.989240 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:56.989221    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twdbl\" (UniqueName: \"kubernetes.io/projected/fbd6d24d-5227-4534-8fe8-1288fef79e88-kube-api-access-twdbl\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:13:56.989813 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:56.989773    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8335fd09-7966-4952-8214-1e569d15f4fa-plugins-config-volume" pod "8335fd09-7966-4952-8214-1e569d15f4fa" (UID: "8335fd09-7966-4952-8214-1e569d15f4fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:13:56.991783 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:56.991757    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8335fd09-7966-4952-8214-1e569d15f4fa-kube-api-access-ppjz9" pod "8335fd09-7966-4952-8214-1e569d15f4fa" (UID: "8335fd09-7966-4952-8214-1e569d15f4fa"). InnerVolumeSpecName "kube-api-access-ppjz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:13:57.070522 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:57.070410    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b61bd136-4b2d-4ab1-b263-6531234de46b" path="/var/lib/kubelet/pods/b61bd136-4b2d-4ab1-b263-6531234de46b/volumes"
Oct 30 21:13:57.092820 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:57.092764    5713 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8335fd09-7966-4952-8214-1e569d15f4fa-plugins-config-volume\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:13:57.093104 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:57.093083    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppjz9\" (UniqueName: \"kubernetes.io/projected/8335fd09-7966-4952-8214-1e569d15f4fa-kube-api-access-ppjz9\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:13:57.347738 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:57.347698    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-7eaf74-b6mtd"]
Oct 30 21:13:57.347934 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:13:57.347911    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9364652a_228c_4e15_ac17_6d26d75befad.slice/crio-2e0e23fa0a9c4ef38740797b1993b87f78c2222c597ec708a9fe0805f3f77800 WatchSource:0}: Error finding container 2e0e23fa0a9c4ef38740797b1993b87f78c2222c597ec708a9fe0805f3f77800: Status 404 returned error can't find the container with id 2e0e23fa0a9c4ef38740797b1993b87f78c2222c597ec708a9fe0805f3f77800
Oct 30 21:13:57.469319 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:57.469283    5713 generic.go:356] "Generic (PLEG): container finished" podID="fbd6d24d-5227-4534-8fe8-1288fef79e88" containerID="358ecb28cf28938ce27fbeae78493d6cc8dcac24f619f217e356572da4728a2c" exitCode=0
Oct 30 21:13:57.469489 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:57.469328    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b888d0-79557999cd-rzz2r"
Oct 30 21:13:57.469489 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:57.469375    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b888d0-79557999cd-rzz2r" event={"ID":"fbd6d24d-5227-4534-8fe8-1288fef79e88","Type":"ContainerDied","Data":"358ecb28cf28938ce27fbeae78493d6cc8dcac24f619f217e356572da4728a2c"}
Oct 30 21:13:57.469489 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:57.469427    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b888d0-79557999cd-rzz2r" event={"ID":"fbd6d24d-5227-4534-8fe8-1288fef79e88","Type":"ContainerDied","Data":"7c84ee175b3e4d58a86759147d5fb8a1c18ada0fa8e595a0501f11d64aea6336"}
Oct 30 21:13:57.469489 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:57.469454    5713 scope.go:122] "RemoveContainer" containerID="358ecb28cf28938ce27fbeae78493d6cc8dcac24f619f217e356572da4728a2c"
Oct 30 21:13:57.471271 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:57.471240    5713 generic.go:356] "Generic (PLEG): container finished" podID="8335fd09-7966-4952-8214-1e569d15f4fa" containerID="e260163d30b28d68dc8894d67faef810fc5c9f95afc79d7b92a1657ba94c01a7" exitCode=0
Oct 30 21:13:57.471397 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:57.471276    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b888d0-75cbbbf7c5-rnfbj" event={"ID":"8335fd09-7966-4952-8214-1e569d15f4fa","Type":"ContainerDied","Data":"e260163d30b28d68dc8894d67faef810fc5c9f95afc79d7b92a1657ba94c01a7"}
Oct 30 21:13:57.471397 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:57.471335    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b888d0-75cbbbf7c5-rnfbj" event={"ID":"8335fd09-7966-4952-8214-1e569d15f4fa","Type":"ContainerDied","Data":"6b9a1b966d9288db770f0b9b757c2d0ee1e89d110d126afa57bdc22f0c237836"}
Oct 30 21:13:57.471397 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:57.471287    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b888d0-75cbbbf7c5-rnfbj"
Oct 30 21:13:57.473149 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:57.473119    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-7eaf74-b6mtd" event={"ID":"9364652a-228c-4e15-ac17-6d26d75befad","Type":"ContainerStarted","Data":"2e0e23fa0a9c4ef38740797b1993b87f78c2222c597ec708a9fe0805f3f77800"}
Oct 30 21:13:57.481568 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:57.481549    5713 scope.go:122] "RemoveContainer" containerID="358ecb28cf28938ce27fbeae78493d6cc8dcac24f619f217e356572da4728a2c"
Oct 30 21:13:57.481917 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:13:57.481890    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"358ecb28cf28938ce27fbeae78493d6cc8dcac24f619f217e356572da4728a2c\": container with ID starting with 358ecb28cf28938ce27fbeae78493d6cc8dcac24f619f217e356572da4728a2c not found: ID does not exist" containerID="358ecb28cf28938ce27fbeae78493d6cc8dcac24f619f217e356572da4728a2c"
Oct 30 21:13:57.482017 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:57.481921    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"358ecb28cf28938ce27fbeae78493d6cc8dcac24f619f217e356572da4728a2c"} err="failed to get container status \"358ecb28cf28938ce27fbeae78493d6cc8dcac24f619f217e356572da4728a2c\": rpc error: code = NotFound desc = could not find container \"358ecb28cf28938ce27fbeae78493d6cc8dcac24f619f217e356572da4728a2c\": container with ID starting with 358ecb28cf28938ce27fbeae78493d6cc8dcac24f619f217e356572da4728a2c not found: ID does not exist"
Oct 30 21:13:57.482017 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:57.481939    5713 scope.go:122] "RemoveContainer" containerID="e260163d30b28d68dc8894d67faef810fc5c9f95afc79d7b92a1657ba94c01a7"
Oct 30 21:13:57.493448 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:57.493427    5713 scope.go:122] "RemoveContainer" containerID="e260163d30b28d68dc8894d67faef810fc5c9f95afc79d7b92a1657ba94c01a7"
Oct 30 21:13:57.493714 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:13:57.493692    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e260163d30b28d68dc8894d67faef810fc5c9f95afc79d7b92a1657ba94c01a7\": container with ID starting with e260163d30b28d68dc8894d67faef810fc5c9f95afc79d7b92a1657ba94c01a7 not found: ID does not exist" containerID="e260163d30b28d68dc8894d67faef810fc5c9f95afc79d7b92a1657ba94c01a7"
Oct 30 21:13:57.493766 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:57.493723    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e260163d30b28d68dc8894d67faef810fc5c9f95afc79d7b92a1657ba94c01a7"} err="failed to get container status \"e260163d30b28d68dc8894d67faef810fc5c9f95afc79d7b92a1657ba94c01a7\": rpc error: code = NotFound desc = could not find container \"e260163d30b28d68dc8894d67faef810fc5c9f95afc79d7b92a1657ba94c01a7\": container with ID starting with e260163d30b28d68dc8894d67faef810fc5c9f95afc79d7b92a1657ba94c01a7 not found: ID does not exist"
Oct 30 21:13:57.656286 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:57.656185    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b888d0-79557999cd-rzz2r"]
Oct 30 21:13:57.666262 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:57.666226    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b888d0-79557999cd-rzz2r"]
Oct 30 21:13:57.768489 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:57.768460    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b888d0-75cbbbf7c5-rnfbj"]
Oct 30 21:13:57.773756 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:57.773728    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b888d0-75cbbbf7c5-rnfbj"]
Oct 30 21:13:58.484587 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:58.484551    5713 generic.go:356] "Generic (PLEG): container finished" podID="9364652a-228c-4e15-ac17-6d26d75befad" containerID="b890ca3381b72c7b6cf673273b1c0d6dbc918c9dd3b781a080beb6ec7fa41658" exitCode=0
Oct 30 21:13:58.484825 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:58.484638    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-7eaf74-b6mtd" event={"ID":"9364652a-228c-4e15-ac17-6d26d75befad","Type":"ContainerDied","Data":"b890ca3381b72c7b6cf673273b1c0d6dbc918c9dd3b781a080beb6ec7fa41658"}
Oct 30 21:13:59.065202 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:59.065170    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8335fd09-7966-4952-8214-1e569d15f4fa" path="/var/lib/kubelet/pods/8335fd09-7966-4952-8214-1e569d15f4fa/volumes"
Oct 30 21:13:59.065716 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:59.065673    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbd6d24d-5227-4534-8fe8-1288fef79e88" path="/var/lib/kubelet/pods/fbd6d24d-5227-4534-8fe8-1288fef79e88/volumes"
Oct 30 21:13:59.666011 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:59.665984    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-7eaf74-b6mtd"
Oct 30 21:13:59.734161 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:59.734126    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9364652a-228c-4e15-ac17-6d26d75befad-kube-api-access-hjf5j\" (UniqueName: \"kubernetes.io/projected/9364652a-228c-4e15-ac17-6d26d75befad-kube-api-access-hjf5j\") pod \"9364652a-228c-4e15-ac17-6d26d75befad\" (UID: \"9364652a-228c-4e15-ac17-6d26d75befad\") "
Oct 30 21:13:59.734161 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:59.734170    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9364652a-228c-4e15-ac17-6d26d75befad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9364652a-228c-4e15-ac17-6d26d75befad-maas-api-openshift-ca\") pod \"9364652a-228c-4e15-ac17-6d26d75befad\" (UID: \"9364652a-228c-4e15-ac17-6d26d75befad\") "
Oct 30 21:13:59.734433 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:59.734411    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9364652a-228c-4e15-ac17-6d26d75befad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9364652a-228c-4e15-ac17-6d26d75befad-maas-api-xks-ca\") pod \"9364652a-228c-4e15-ac17-6d26d75befad\" (UID: \"9364652a-228c-4e15-ac17-6d26d75befad\") "
Oct 30 21:13:59.734629 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:59.734607    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9364652a-228c-4e15-ac17-6d26d75befad-maas-api-openshift-ca" pod "9364652a-228c-4e15-ac17-6d26d75befad" (UID: "9364652a-228c-4e15-ac17-6d26d75befad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:13:59.735323 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:59.735297    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9364652a-228c-4e15-ac17-6d26d75befad-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:13:59.736559 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:59.736536    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9364652a-228c-4e15-ac17-6d26d75befad-kube-api-access-hjf5j" pod "9364652a-228c-4e15-ac17-6d26d75befad" (UID: "9364652a-228c-4e15-ac17-6d26d75befad"). InnerVolumeSpecName "kube-api-access-hjf5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:13:59.736722 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:59.736699    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9364652a-228c-4e15-ac17-6d26d75befad-maas-api-xks-ca" pod "9364652a-228c-4e15-ac17-6d26d75befad" (UID: "9364652a-228c-4e15-ac17-6d26d75befad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:13:59.836992 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:59.836906    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9364652a-228c-4e15-ac17-6d26d75befad-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:13:59.836992 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:13:59.836942    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjf5j\" (UniqueName: \"kubernetes.io/projected/9364652a-228c-4e15-ac17-6d26d75befad-kube-api-access-hjf5j\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:14:00.526600 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:00.526275    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-7eaf74-b6mtd" event={"ID":"9364652a-228c-4e15-ac17-6d26d75befad","Type":"ContainerDied","Data":"2e0e23fa0a9c4ef38740797b1993b87f78c2222c597ec708a9fe0805f3f77800"}
Oct 30 21:14:00.526600 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:00.526344    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e0e23fa0a9c4ef38740797b1993b87f78c2222c597ec708a9fe0805f3f77800"
Oct 30 21:14:00.526600 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:00.526419    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-7eaf74-b6mtd"
Oct 30 21:14:01.690231 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:01.690195    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-255f05-jvw7k"]
Oct 30 21:14:01.690922 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:01.690908    5713 cpu_manager.go:414] "containerMap: removing container" podUID="fbd6d24d-5227-4534-8fe8-1288fef79e88" containerName="payload-processing"
Oct 30 21:14:01.690979 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:01.690924    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbd6d24d-5227-4534-8fe8-1288fef79e88" containerName="payload-processing"
Oct 30 21:14:01.690979 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:01.690958    5713 cpu_manager.go:414] "containerMap: removing container" podUID="9364652a-228c-4e15-ac17-6d26d75befad" containerName="revoke-keys"
Oct 30 21:14:01.690979 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:01.690964    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9364652a-228c-4e15-ac17-6d26d75befad" containerName="revoke-keys"
Oct 30 21:14:01.690979 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:01.690972    5713 cpu_manager.go:414] "containerMap: removing container" podUID="8335fd09-7966-4952-8214-1e569d15f4fa" containerName="payload-pre-processing"
Oct 30 21:14:01.690979 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:01.690978    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8335fd09-7966-4952-8214-1e569d15f4fa" containerName="payload-pre-processing"
Oct 30 21:14:01.691135 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:01.691098    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="9364652a-228c-4e15-ac17-6d26d75befad" containerName="revoke-keys"
Oct 30 21:14:01.691135 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:01.691108    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="8335fd09-7966-4952-8214-1e569d15f4fa" containerName="payload-pre-processing"
Oct 30 21:14:01.691135 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:01.691122    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbd6d24d-5227-4534-8fe8-1288fef79e88" containerName="payload-processing"
Oct 30 21:14:01.696460 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:01.696430    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-255f05-jvw7k"
Oct 30 21:14:01.751994 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:01.751964    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-255f05-jvw7k"]
Oct 30 21:14:01.771453 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:01.771408    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66ce7582-3ef0-47b6-9a48-99a988749de4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-255f05-jvw7k\" (UID: \"66ce7582-3ef0-47b6-9a48-99a988749de4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-255f05-jvw7k"
Oct 30 21:14:01.771633 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:01.771485    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66ce7582-3ef0-47b6-9a48-99a988749de4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-255f05-jvw7k\" (UID: \"66ce7582-3ef0-47b6-9a48-99a988749de4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-255f05-jvw7k"
Oct 30 21:14:01.771923 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:01.771895    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xth9g\" (UniqueName: \"kubernetes.io/projected/66ce7582-3ef0-47b6-9a48-99a988749de4-kube-api-access-xth9g\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-255f05-jvw7k\" (UID: \"66ce7582-3ef0-47b6-9a48-99a988749de4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-255f05-jvw7k"
Oct 30 21:14:01.796748 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:01.796727    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:14:01.796748 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:01.796730    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:14:01.798522 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:01.798506    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:14:01.798621 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:01.798586    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:14:01.873531 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:01.873500    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xth9g\" (UniqueName: \"kubernetes.io/projected/66ce7582-3ef0-47b6-9a48-99a988749de4-kube-api-access-xth9g\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-255f05-jvw7k\" (UID: \"66ce7582-3ef0-47b6-9a48-99a988749de4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-255f05-jvw7k"
Oct 30 21:14:01.873700 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:01.873597    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66ce7582-3ef0-47b6-9a48-99a988749de4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-255f05-jvw7k\" (UID: \"66ce7582-3ef0-47b6-9a48-99a988749de4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-255f05-jvw7k"
Oct 30 21:14:01.873700 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:01.873635    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66ce7582-3ef0-47b6-9a48-99a988749de4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-255f05-jvw7k\" (UID: \"66ce7582-3ef0-47b6-9a48-99a988749de4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-255f05-jvw7k"
Oct 30 21:14:01.874392 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:01.874370    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66ce7582-3ef0-47b6-9a48-99a988749de4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-255f05-jvw7k\" (UID: \"66ce7582-3ef0-47b6-9a48-99a988749de4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-255f05-jvw7k"
Oct 30 21:14:01.876242 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:01.876222    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66ce7582-3ef0-47b6-9a48-99a988749de4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-255f05-jvw7k\" (UID: \"66ce7582-3ef0-47b6-9a48-99a988749de4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-255f05-jvw7k"
Oct 30 21:14:01.930599 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:01.930565    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xth9g\" (UniqueName: \"kubernetes.io/projected/66ce7582-3ef0-47b6-9a48-99a988749de4-kube-api-access-xth9g\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-255f05-jvw7k\" (UID: \"66ce7582-3ef0-47b6-9a48-99a988749de4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-255f05-jvw7k"
Oct 30 21:14:02.016286 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:02.016194    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-255f05-jvw7k"
Oct 30 21:14:02.408428 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:02.408400    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-255f05-jvw7k"]
Oct 30 21:14:02.408695 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:14:02.408670    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66ce7582_3ef0_47b6_9a48_99a988749de4.slice/crio-bbad7c191ec3ffb4d91490f4c5d7a439458525715ddbec4f6d0dcb5c4eb53d42 WatchSource:0}: Error finding container bbad7c191ec3ffb4d91490f4c5d7a439458525715ddbec4f6d0dcb5c4eb53d42: Status 404 returned error can't find the container with id bbad7c191ec3ffb4d91490f4c5d7a439458525715ddbec4f6d0dcb5c4eb53d42
Oct 30 21:14:02.545478 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:02.545446    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-255f05-jvw7k" event={"ID":"66ce7582-3ef0-47b6-9a48-99a988749de4","Type":"ContainerStarted","Data":"bbad7c191ec3ffb4d91490f4c5d7a439458525715ddbec4f6d0dcb5c4eb53d42"}
Oct 30 21:14:02.907481 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:02.907449    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck"]
Oct 30 21:14:02.907892 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:02.907659    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck" podUID="0da8c670-dd6c-4a19-856e-0241a7494dbb" containerName="istio-proxy" containerID="cri-o://5976c2a9326676d051cc805f8fd5bc59ab319b0c58f895444242fc6d2ed8c2bc" gracePeriod=30
Oct 30 21:14:03.559333 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:03.559301    5713 generic.go:356] "Generic (PLEG): container finished" podID="66ce7582-3ef0-47b6-9a48-99a988749de4" containerID="7d2615e052e0dea06f756a93c1d16a61956e062ec4b1a632bc312e9b4be9fd64" exitCode=0
Oct 30 21:14:03.559483 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:03.559375    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-255f05-jvw7k" event={"ID":"66ce7582-3ef0-47b6-9a48-99a988749de4","Type":"ContainerDied","Data":"7d2615e052e0dea06f756a93c1d16a61956e062ec4b1a632bc312e9b4be9fd64"}
Oct 30 21:14:04.733550 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:04.733517    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-255f05-jvw7k"
Oct 30 21:14:04.837511 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:04.837408    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66ce7582-3ef0-47b6-9a48-99a988749de4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66ce7582-3ef0-47b6-9a48-99a988749de4-maas-api-xks-ca\") pod \"66ce7582-3ef0-47b6-9a48-99a988749de4\" (UID: \"66ce7582-3ef0-47b6-9a48-99a988749de4\") "
Oct 30 21:14:04.837707 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:04.837508    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66ce7582-3ef0-47b6-9a48-99a988749de4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66ce7582-3ef0-47b6-9a48-99a988749de4-maas-api-openshift-ca\") pod \"66ce7582-3ef0-47b6-9a48-99a988749de4\" (UID: \"66ce7582-3ef0-47b6-9a48-99a988749de4\") "
Oct 30 21:14:04.837707 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:04.837639    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66ce7582-3ef0-47b6-9a48-99a988749de4-kube-api-access-xth9g\" (UniqueName: \"kubernetes.io/projected/66ce7582-3ef0-47b6-9a48-99a988749de4-kube-api-access-xth9g\") pod \"66ce7582-3ef0-47b6-9a48-99a988749de4\" (UID: \"66ce7582-3ef0-47b6-9a48-99a988749de4\") "
Oct 30 21:14:04.837927 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:04.837884    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66ce7582-3ef0-47b6-9a48-99a988749de4-maas-api-openshift-ca" pod "66ce7582-3ef0-47b6-9a48-99a988749de4" (UID: "66ce7582-3ef0-47b6-9a48-99a988749de4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:14:04.838665 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:04.838639    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66ce7582-3ef0-47b6-9a48-99a988749de4-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:14:04.840025 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:04.839997    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66ce7582-3ef0-47b6-9a48-99a988749de4-kube-api-access-xth9g" pod "66ce7582-3ef0-47b6-9a48-99a988749de4" (UID: "66ce7582-3ef0-47b6-9a48-99a988749de4"). InnerVolumeSpecName "kube-api-access-xth9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:14:04.840139 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:04.840029    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66ce7582-3ef0-47b6-9a48-99a988749de4-maas-api-xks-ca" pod "66ce7582-3ef0-47b6-9a48-99a988749de4" (UID: "66ce7582-3ef0-47b6-9a48-99a988749de4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:14:04.940605 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:04.940568    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66ce7582-3ef0-47b6-9a48-99a988749de4-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:14:04.940605 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:04.940599    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xth9g\" (UniqueName: \"kubernetes.io/projected/66ce7582-3ef0-47b6-9a48-99a988749de4-kube-api-access-xth9g\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:14:05.577490 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:05.577456    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-255f05-jvw7k"
Oct 30 21:14:05.577704 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:05.577493    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-255f05-jvw7k" event={"ID":"66ce7582-3ef0-47b6-9a48-99a988749de4","Type":"ContainerDied","Data":"bbad7c191ec3ffb4d91490f4c5d7a439458525715ddbec4f6d0dcb5c4eb53d42"}
Oct 30 21:14:05.577704 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:05.577574    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bbad7c191ec3ffb4d91490f4c5d7a439458525715ddbec4f6d0dcb5c4eb53d42"
Oct 30 21:14:05.765511 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:05.763179    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg"]
Oct 30 21:14:05.765511 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:05.764354    5713 cpu_manager.go:414] "containerMap: removing container" podUID="66ce7582-3ef0-47b6-9a48-99a988749de4" containerName="revoke-keys"
Oct 30 21:14:05.765511 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:05.764379    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66ce7582-3ef0-47b6-9a48-99a988749de4" containerName="revoke-keys"
Oct 30 21:14:05.765511 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:05.764552    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="66ce7582-3ef0-47b6-9a48-99a988749de4" containerName="revoke-keys"
Oct 30 21:14:05.805843 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:05.790258    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg"
Oct 30 21:14:05.869901 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:05.853239    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-f0fe13-openshift-default-dockercfg-jgl2n\""
Oct 30 21:14:05.977826 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:05.896535    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0dff592-d5a1-4832-8104-46cc5fd973c5-workload-socket\") pod \"e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg\" (UID: \"d0dff592-d5a1-4832-8104-46cc5fd973c5\") " pod="openshift-ingress/e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg"
Oct 30 21:14:05.978225 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:05.978190    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0dff592-d5a1-4832-8104-46cc5fd973c5-workload-certs\") pod \"e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg\" (UID: \"d0dff592-d5a1-4832-8104-46cc5fd973c5\") " pod="openshift-ingress/e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg"
Oct 30 21:14:05.978394 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:05.978379    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0dff592-d5a1-4832-8104-46cc5fd973c5-istio-podinfo\") pod \"e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg\" (UID: \"d0dff592-d5a1-4832-8104-46cc5fd973c5\") " pod="openshift-ingress/e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg"
Oct 30 21:14:05.978647 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:05.978621    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0dff592-d5a1-4832-8104-46cc5fd973c5-credential-socket\") pod \"e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg\" (UID: \"d0dff592-d5a1-4832-8104-46cc5fd973c5\") " pod="openshift-ingress/e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg"
Oct 30 21:14:05.978895 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:05.978876    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0dff592-d5a1-4832-8104-46cc5fd973c5-istiod-ca-cert\") pod \"e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg\" (UID: \"d0dff592-d5a1-4832-8104-46cc5fd973c5\") " pod="openshift-ingress/e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg"
Oct 30 21:14:05.979221 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:05.979202    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0dff592-d5a1-4832-8104-46cc5fd973c5-istio-data\") pod \"e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg\" (UID: \"d0dff592-d5a1-4832-8104-46cc5fd973c5\") " pod="openshift-ingress/e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg"
Oct 30 21:14:05.979405 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:05.979390    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0dff592-d5a1-4832-8104-46cc5fd973c5-istio-envoy\") pod \"e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg\" (UID: \"d0dff592-d5a1-4832-8104-46cc5fd973c5\") " pod="openshift-ingress/e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg"
Oct 30 21:14:05.979509 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:05.979495    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tcj4\" (UniqueName: \"kubernetes.io/projected/d0dff592-d5a1-4832-8104-46cc5fd973c5-kube-api-access-2tcj4\") pod \"e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg\" (UID: \"d0dff592-d5a1-4832-8104-46cc5fd973c5\") " pod="openshift-ingress/e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg"
Oct 30 21:14:05.979646 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:05.979633    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0dff592-d5a1-4832-8104-46cc5fd973c5-istio-token\") pod \"e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg\" (UID: \"d0dff592-d5a1-4832-8104-46cc5fd973c5\") " pod="openshift-ingress/e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg"
Oct 30 21:14:06.022523 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:06.021845    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg"]
Oct 30 21:14:06.083222 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:06.083177    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0dff592-d5a1-4832-8104-46cc5fd973c5-istiod-ca-cert\") pod \"e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg\" (UID: \"d0dff592-d5a1-4832-8104-46cc5fd973c5\") " pod="openshift-ingress/e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg"
Oct 30 21:14:06.084365 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:06.084241    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0dff592-d5a1-4832-8104-46cc5fd973c5-istiod-ca-cert\") pod \"e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg\" (UID: \"d0dff592-d5a1-4832-8104-46cc5fd973c5\") " pod="openshift-ingress/e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg"
Oct 30 21:14:06.084658 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:06.084627    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0dff592-d5a1-4832-8104-46cc5fd973c5-istio-data\") pod \"e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg\" (UID: \"d0dff592-d5a1-4832-8104-46cc5fd973c5\") " pod="openshift-ingress/e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg"
Oct 30 21:14:06.085297 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:06.085191    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0dff592-d5a1-4832-8104-46cc5fd973c5-istio-data\") pod \"e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg\" (UID: \"d0dff592-d5a1-4832-8104-46cc5fd973c5\") " pod="openshift-ingress/e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg"
Oct 30 21:14:06.085976 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:06.085957    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0dff592-d5a1-4832-8104-46cc5fd973c5-istio-envoy\") pod \"e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg\" (UID: \"d0dff592-d5a1-4832-8104-46cc5fd973c5\") " pod="openshift-ingress/e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg"
Oct 30 21:14:06.087969 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:06.087947    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tcj4\" (UniqueName: \"kubernetes.io/projected/d0dff592-d5a1-4832-8104-46cc5fd973c5-kube-api-access-2tcj4\") pod \"e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg\" (UID: \"d0dff592-d5a1-4832-8104-46cc5fd973c5\") " pod="openshift-ingress/e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg"
Oct 30 21:14:06.088685 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:06.088665    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0dff592-d5a1-4832-8104-46cc5fd973c5-istio-envoy\") pod \"e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg\" (UID: \"d0dff592-d5a1-4832-8104-46cc5fd973c5\") " pod="openshift-ingress/e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg"
Oct 30 21:14:06.088892 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:06.088874    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0dff592-d5a1-4832-8104-46cc5fd973c5-istio-token\") pod \"e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg\" (UID: \"d0dff592-d5a1-4832-8104-46cc5fd973c5\") " pod="openshift-ingress/e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg"
Oct 30 21:14:06.089406 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:06.089386    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0dff592-d5a1-4832-8104-46cc5fd973c5-workload-socket\") pod \"e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg\" (UID: \"d0dff592-d5a1-4832-8104-46cc5fd973c5\") " pod="openshift-ingress/e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg"
Oct 30 21:14:06.090099 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:06.090082    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0dff592-d5a1-4832-8104-46cc5fd973c5-workload-certs\") pod \"e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg\" (UID: \"d0dff592-d5a1-4832-8104-46cc5fd973c5\") " pod="openshift-ingress/e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg"
Oct 30 21:14:06.090772 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:06.090756    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0dff592-d5a1-4832-8104-46cc5fd973c5-istio-podinfo\") pod \"e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg\" (UID: \"d0dff592-d5a1-4832-8104-46cc5fd973c5\") " pod="openshift-ingress/e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg"
Oct 30 21:14:06.091083 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:06.091065    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0dff592-d5a1-4832-8104-46cc5fd973c5-credential-socket\") pod \"e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg\" (UID: \"d0dff592-d5a1-4832-8104-46cc5fd973c5\") " pod="openshift-ingress/e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg"
Oct 30 21:14:06.092384 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:06.089980    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0dff592-d5a1-4832-8104-46cc5fd973c5-workload-socket\") pod \"e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg\" (UID: \"d0dff592-d5a1-4832-8104-46cc5fd973c5\") " pod="openshift-ingress/e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg"
Oct 30 21:14:06.092384 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:06.092344    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0dff592-d5a1-4832-8104-46cc5fd973c5-credential-socket\") pod \"e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg\" (UID: \"d0dff592-d5a1-4832-8104-46cc5fd973c5\") " pod="openshift-ingress/e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg"
Oct 30 21:14:06.092384 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:06.090689    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0dff592-d5a1-4832-8104-46cc5fd973c5-workload-certs\") pod \"e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg\" (UID: \"d0dff592-d5a1-4832-8104-46cc5fd973c5\") " pod="openshift-ingress/e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg"
Oct 30 21:14:06.097196 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:06.097163    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0dff592-d5a1-4832-8104-46cc5fd973c5-istio-podinfo\") pod \"e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg\" (UID: \"d0dff592-d5a1-4832-8104-46cc5fd973c5\") " pod="openshift-ingress/e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg"
Oct 30 21:14:06.163872 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:06.162288    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tcj4\" (UniqueName: \"kubernetes.io/projected/d0dff592-d5a1-4832-8104-46cc5fd973c5-kube-api-access-2tcj4\") pod \"e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg\" (UID: \"d0dff592-d5a1-4832-8104-46cc5fd973c5\") " pod="openshift-ingress/e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg"
Oct 30 21:14:06.164336 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:06.164304    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0dff592-d5a1-4832-8104-46cc5fd973c5-istio-token\") pod \"e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg\" (UID: \"d0dff592-d5a1-4832-8104-46cc5fd973c5\") " pod="openshift-ingress/e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg"
Oct 30 21:14:06.361824 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:06.358114    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg"
Oct 30 21:14:06.863111 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:06.863081    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg"]
Oct 30 21:14:06.866363 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:14:06.866331    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0dff592_d5a1_4832_8104_46cc5fd973c5.slice/crio-7bf3846ed31e973bed68487e7622297e831b68bbf74eac744cb981a22ff4c18f WatchSource:0}: Error finding container 7bf3846ed31e973bed68487e7622297e831b68bbf74eac744cb981a22ff4c18f: Status 404 returned error can't find the container with id 7bf3846ed31e973bed68487e7622297e831b68bbf74eac744cb981a22ff4c18f
Oct 30 21:14:06.869218 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:06.869188    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:14:06.869304 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:06.869252    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:14:06.869304 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:06.869284    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:14:07.598508 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:07.598477    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg" event={"ID":"d0dff592-d5a1-4832-8104-46cc5fd973c5","Type":"ContainerStarted","Data":"607a33906cd32389d6ea58d710f74b7d5db210e30b2d21241e6e873832713ff8"}
Oct 30 21:14:07.598674 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:07.598527    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg" event={"ID":"d0dff592-d5a1-4832-8104-46cc5fd973c5","Type":"ContainerStarted","Data":"7bf3846ed31e973bed68487e7622297e831b68bbf74eac744cb981a22ff4c18f"}
Oct 30 21:14:07.773527 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:07.773460    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg" podStartSLOduration=2.773439965 podStartE2EDuration="2.773439965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:14:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:14:07.72011024 +0000 UTC m=+2130.784886633" watchObservedRunningTime="2026-10-30 21:14:07.773439965 +0000 UTC m=+2130.838216363"
Oct 30 21:14:08.359948 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:08.359916    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg"
Oct 30 21:14:08.367386 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:08.367357    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg"
Oct 30 21:14:08.370883 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:08.370852    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck"
Oct 30 21:14:08.488877 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:08.488848    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0da8c670-dd6c-4a19-856e-0241a7494dbb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0da8c670-dd6c-4a19-856e-0241a7494dbb-istio-data\") pod \"0da8c670-dd6c-4a19-856e-0241a7494dbb\" (UID: \"0da8c670-dd6c-4a19-856e-0241a7494dbb\") "
Oct 30 21:14:08.489037 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:08.489008    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0da8c670-dd6c-4a19-856e-0241a7494dbb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da8c670-dd6c-4a19-856e-0241a7494dbb-workload-socket\") pod \"0da8c670-dd6c-4a19-856e-0241a7494dbb\" (UID: \"0da8c670-dd6c-4a19-856e-0241a7494dbb\") "
Oct 30 21:14:08.489095 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:08.489069    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0da8c670-dd6c-4a19-856e-0241a7494dbb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da8c670-dd6c-4a19-856e-0241a7494dbb-credential-socket\") pod \"0da8c670-dd6c-4a19-856e-0241a7494dbb\" (UID: \"0da8c670-dd6c-4a19-856e-0241a7494dbb\") "
Oct 30 21:14:08.489137 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:08.489095    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0da8c670-dd6c-4a19-856e-0241a7494dbb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0da8c670-dd6c-4a19-856e-0241a7494dbb-istio-podinfo\") pod \"0da8c670-dd6c-4a19-856e-0241a7494dbb\" (UID: \"0da8c670-dd6c-4a19-856e-0241a7494dbb\") "
Oct 30 21:14:08.489182 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:08.489160    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0da8c670-dd6c-4a19-856e-0241a7494dbb-kube-api-access-8jjx8\" (UniqueName: \"kubernetes.io/projected/0da8c670-dd6c-4a19-856e-0241a7494dbb-kube-api-access-8jjx8\") pod \"0da8c670-dd6c-4a19-856e-0241a7494dbb\" (UID: \"0da8c670-dd6c-4a19-856e-0241a7494dbb\") "
Oct 30 21:14:08.489242 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:08.489154    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da8c670-dd6c-4a19-856e-0241a7494dbb-istio-data" pod "0da8c670-dd6c-4a19-856e-0241a7494dbb" (UID: "0da8c670-dd6c-4a19-856e-0241a7494dbb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:14:08.489294 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:08.489246    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da8c670-dd6c-4a19-856e-0241a7494dbb-workload-socket" pod "0da8c670-dd6c-4a19-856e-0241a7494dbb" (UID: "0da8c670-dd6c-4a19-856e-0241a7494dbb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:14:08.489294 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:08.489286    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0da8c670-dd6c-4a19-856e-0241a7494dbb-istio-token\" (UniqueName: \"kubernetes.io/projected/0da8c670-dd6c-4a19-856e-0241a7494dbb-istio-token\") pod \"0da8c670-dd6c-4a19-856e-0241a7494dbb\" (UID: \"0da8c670-dd6c-4a19-856e-0241a7494dbb\") "
Oct 30 21:14:08.489403 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:08.489347    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0da8c670-dd6c-4a19-856e-0241a7494dbb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0da8c670-dd6c-4a19-856e-0241a7494dbb-istiod-ca-cert\") pod \"0da8c670-dd6c-4a19-856e-0241a7494dbb\" (UID: \"0da8c670-dd6c-4a19-856e-0241a7494dbb\") "
Oct 30 21:14:08.489403 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:08.489367    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da8c670-dd6c-4a19-856e-0241a7494dbb-credential-socket" pod "0da8c670-dd6c-4a19-856e-0241a7494dbb" (UID: "0da8c670-dd6c-4a19-856e-0241a7494dbb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:14:08.489513 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:08.489413    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0da8c670-dd6c-4a19-856e-0241a7494dbb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0da8c670-dd6c-4a19-856e-0241a7494dbb-istio-envoy\") pod \"0da8c670-dd6c-4a19-856e-0241a7494dbb\" (UID: \"0da8c670-dd6c-4a19-856e-0241a7494dbb\") "
Oct 30 21:14:08.489513 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:08.489462    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0da8c670-dd6c-4a19-856e-0241a7494dbb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0da8c670-dd6c-4a19-856e-0241a7494dbb-workload-certs\") pod \"0da8c670-dd6c-4a19-856e-0241a7494dbb\" (UID: \"0da8c670-dd6c-4a19-856e-0241a7494dbb\") "
Oct 30 21:14:08.489731 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:08.489701    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0da8c670-dd6c-4a19-856e-0241a7494dbb-istiod-ca-cert" pod "0da8c670-dd6c-4a19-856e-0241a7494dbb" (UID: "0da8c670-dd6c-4a19-856e-0241a7494dbb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:14:08.489989 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:08.489917    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da8c670-dd6c-4a19-856e-0241a7494dbb-workload-certs" pod "0da8c670-dd6c-4a19-856e-0241a7494dbb" (UID: "0da8c670-dd6c-4a19-856e-0241a7494dbb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:14:08.490562 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:08.490536    5713 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0da8c670-dd6c-4a19-856e-0241a7494dbb-istiod-ca-cert\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:14:08.490694 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:08.490567    5713 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0da8c670-dd6c-4a19-856e-0241a7494dbb-workload-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:14:08.490694 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:08.490583    5713 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0da8c670-dd6c-4a19-856e-0241a7494dbb-istio-data\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:14:08.490694 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:08.490598    5713 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da8c670-dd6c-4a19-856e-0241a7494dbb-workload-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:14:08.490694 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:08.490612    5713 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da8c670-dd6c-4a19-856e-0241a7494dbb-credential-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:14:08.491673 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:08.491647    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0da8c670-dd6c-4a19-856e-0241a7494dbb-kube-api-access-8jjx8" pod "0da8c670-dd6c-4a19-856e-0241a7494dbb" (UID: "0da8c670-dd6c-4a19-856e-0241a7494dbb"). InnerVolumeSpecName "kube-api-access-8jjx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:14:08.491673 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:08.491663    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0da8c670-dd6c-4a19-856e-0241a7494dbb-istio-token" pod "0da8c670-dd6c-4a19-856e-0241a7494dbb" (UID: "0da8c670-dd6c-4a19-856e-0241a7494dbb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:14:08.491812 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:08.491691    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0da8c670-dd6c-4a19-856e-0241a7494dbb-istio-podinfo" pod "0da8c670-dd6c-4a19-856e-0241a7494dbb" (UID: "0da8c670-dd6c-4a19-856e-0241a7494dbb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:14:08.491940 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:08.491925    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da8c670-dd6c-4a19-856e-0241a7494dbb-istio-envoy" pod "0da8c670-dd6c-4a19-856e-0241a7494dbb" (UID: "0da8c670-dd6c-4a19-856e-0241a7494dbb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:14:08.592596 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:08.592514    5713 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0da8c670-dd6c-4a19-856e-0241a7494dbb-istio-podinfo\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:14:08.592596 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:08.592545    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jjx8\" (UniqueName: \"kubernetes.io/projected/0da8c670-dd6c-4a19-856e-0241a7494dbb-kube-api-access-8jjx8\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:14:08.592596 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:08.592555    5713 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0da8c670-dd6c-4a19-856e-0241a7494dbb-istio-token\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:14:08.592596 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:08.592565    5713 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0da8c670-dd6c-4a19-856e-0241a7494dbb-istio-envoy\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:14:08.608026 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:08.607997    5713 generic.go:356] "Generic (PLEG): container finished" podID="0da8c670-dd6c-4a19-856e-0241a7494dbb" containerID="5976c2a9326676d051cc805f8fd5bc59ab319b0c58f895444242fc6d2ed8c2bc" exitCode=0
Oct 30 21:14:08.608173 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:08.608047    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck"
Oct 30 21:14:08.608173 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:08.608081    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck" event={"ID":"0da8c670-dd6c-4a19-856e-0241a7494dbb","Type":"ContainerDied","Data":"5976c2a9326676d051cc805f8fd5bc59ab319b0c58f895444242fc6d2ed8c2bc"}
Oct 30 21:14:08.608173 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:08.608148    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck" event={"ID":"0da8c670-dd6c-4a19-856e-0241a7494dbb","Type":"ContainerDied","Data":"1a7d823eb45db6e55829818f4cd233185aa41ee695da34da583cf670d4b5f08e"}
Oct 30 21:14:08.608283 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:08.608178    5713 scope.go:122] "RemoveContainer" containerID="5976c2a9326676d051cc805f8fd5bc59ab319b0c58f895444242fc6d2ed8c2bc"
Oct 30 21:14:08.608661 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:08.608590    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg"
Oct 30 21:14:08.609915 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:08.609895    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg"
Oct 30 21:14:08.621191 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:08.621170    5713 scope.go:122] "RemoveContainer" containerID="5976c2a9326676d051cc805f8fd5bc59ab319b0c58f895444242fc6d2ed8c2bc"
Oct 30 21:14:08.621476 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:14:08.621459    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5976c2a9326676d051cc805f8fd5bc59ab319b0c58f895444242fc6d2ed8c2bc\": container with ID starting with 5976c2a9326676d051cc805f8fd5bc59ab319b0c58f895444242fc6d2ed8c2bc not found: ID does not exist" containerID="5976c2a9326676d051cc805f8fd5bc59ab319b0c58f895444242fc6d2ed8c2bc"
Oct 30 21:14:08.621534 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:08.621482    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5976c2a9326676d051cc805f8fd5bc59ab319b0c58f895444242fc6d2ed8c2bc"} err="failed to get container status \"5976c2a9326676d051cc805f8fd5bc59ab319b0c58f895444242fc6d2ed8c2bc\": rpc error: code = NotFound desc = could not find container \"5976c2a9326676d051cc805f8fd5bc59ab319b0c58f895444242fc6d2ed8c2bc\": container with ID starting with 5976c2a9326676d051cc805f8fd5bc59ab319b0c58f895444242fc6d2ed8c2bc not found: ID does not exist"
Oct 30 21:14:08.795763 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:08.795724    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck"]
Oct 30 21:14:08.801638 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:08.801606    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-ef7e2d-openshift-default-6b6dbc765f-76tck"]
Oct 30 21:14:09.074938 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:09.074910    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0da8c670-dd6c-4a19-856e-0241a7494dbb" path="/var/lib/kubelet/pods/0da8c670-dd6c-4a19-856e-0241a7494dbb/volumes"
Oct 30 21:14:13.763484 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:13.763450    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-255f05-677cd4895b-ncgcl"]
Oct 30 21:14:13.875483 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:13.875446    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-255f05-85b6fd8bf9-wdzgm"]
Oct 30 21:14:14.472536 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:14.472503    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-4d1110-25cpk"]
Oct 30 21:14:14.476206 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:14.476185    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-4d1110-25cpk"]
Oct 30 21:14:15.066505 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:15.066465    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="223405e8-db69-41dc-8c2c-f1258995245e" path="/var/lib/kubelet/pods/223405e8-db69-41dc-8c2c-f1258995245e/volumes"
Oct 30 21:14:15.679807 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:15.679766    5713 generic.go:356] "Generic (PLEG): container finished" podID="237d222b-50be-4b97-b53d-49187434c09b" containerID="05163563985b8d23bde1a51105a5b2d6ba5a9dd9196c54c6ca0a059993cd8339" exitCode=0
Oct 30 21:14:15.680009 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:15.679838    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk" event={"ID":"237d222b-50be-4b97-b53d-49187434c09b","Type":"ContainerDied","Data":"05163563985b8d23bde1a51105a5b2d6ba5a9dd9196c54c6ca0a059993cd8339"}
Oct 30 21:14:15.680327 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:15.680312    5713 scope.go:122] "RemoveContainer" containerID="05163563985b8d23bde1a51105a5b2d6ba5a9dd9196c54c6ca0a059993cd8339"
Oct 30 21:14:15.682130 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:15.682102    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:14:15.682254 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:15.682169    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:14:15.682254 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:15.682201    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:14:16.690356 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:16.690323    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk" event={"ID":"237d222b-50be-4b97-b53d-49187434c09b","Type":"ContainerStarted","Data":"332957b8ee59b70dc1edf253b8a8fe242676318b11019866f8f09563dbb41904"}
Oct 30 21:14:17.090667 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:17.090592    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk"
Oct 30 21:14:17.090827 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:17.090710    5713 patch_prober.go:29] interesting pod/e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Oct 30 21:14:17.090827 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:17.090755    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk" podUID="237d222b-50be-4b97-b53d-49187434c09b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Oct 30 21:14:18.141975 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:18.107387    5713 patch_prober.go:29] interesting pod/e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Oct 30 21:14:18.141975 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:18.107453    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk" podUID="237d222b-50be-4b97-b53d-49187434c09b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Oct 30 21:14:18.766913 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:18.766067    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-255f05-677cd4895b-ncgcl" podUID="b673d422-2d71-4f9d-bc6a-5bf0d3d09d90" containerName="payload-processing" containerID="cri-o://7b3efda7df5ed1a2bc25088c8f89f63289c6de3ba8501285602de7bcaaf385d9" gracePeriod=25
Oct 30 21:14:18.876260 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:18.876209    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-255f05-85b6fd8bf9-wdzgm" podUID="a7d22651-156e-4a5f-82f3-3aa7f3afc2b7" containerName="payload-pre-processing" containerID="cri-o://b81408ee16cb6c4ccbcde1bfa3cf0935562c54ac8669af9b0d26b0f19e4b6c6c" gracePeriod=25
Oct 30 21:14:19.224827 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:19.224232    5713 patch_prober.go:29] interesting pod/e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Oct 30 21:14:19.224827 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:19.224291    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk" podUID="237d222b-50be-4b97-b53d-49187434c09b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Oct 30 21:14:19.289316 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:19.289284    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-255f05-677cd4895b-ncgcl"
Oct 30 21:14:19.474653 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:19.474629    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-255f05-85b6fd8bf9-wdzgm"
Oct 30 21:14:19.476738 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:19.476663    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b673d422-2d71-4f9d-bc6a-5bf0d3d09d90-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b673d422-2d71-4f9d-bc6a-5bf0d3d09d90-plugins-config-volume\") pod \"b673d422-2d71-4f9d-bc6a-5bf0d3d09d90\" (UID: \"b673d422-2d71-4f9d-bc6a-5bf0d3d09d90\") "
Oct 30 21:14:19.476900 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:19.476826    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b673d422-2d71-4f9d-bc6a-5bf0d3d09d90-kube-api-access-9kl7n\" (UniqueName: \"kubernetes.io/projected/b673d422-2d71-4f9d-bc6a-5bf0d3d09d90-kube-api-access-9kl7n\") pod \"b673d422-2d71-4f9d-bc6a-5bf0d3d09d90\" (UID: \"b673d422-2d71-4f9d-bc6a-5bf0d3d09d90\") "
Oct 30 21:14:19.476970 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:19.476951    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7d22651-156e-4a5f-82f3-3aa7f3afc2b7-kube-api-access-l2jd6\" (UniqueName: \"kubernetes.io/projected/a7d22651-156e-4a5f-82f3-3aa7f3afc2b7-kube-api-access-l2jd6\") pod \"a7d22651-156e-4a5f-82f3-3aa7f3afc2b7\" (UID: \"a7d22651-156e-4a5f-82f3-3aa7f3afc2b7\") "
Oct 30 21:14:19.477121 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:19.477014    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7d22651-156e-4a5f-82f3-3aa7f3afc2b7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7d22651-156e-4a5f-82f3-3aa7f3afc2b7-plugins-config-volume\") pod \"a7d22651-156e-4a5f-82f3-3aa7f3afc2b7\" (UID: \"a7d22651-156e-4a5f-82f3-3aa7f3afc2b7\") "
Oct 30 21:14:19.477250 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:19.477144    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b673d422-2d71-4f9d-bc6a-5bf0d3d09d90-plugins-config-volume" pod "b673d422-2d71-4f9d-bc6a-5bf0d3d09d90" (UID: "b673d422-2d71-4f9d-bc6a-5bf0d3d09d90"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:14:19.477630 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:19.477604    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7d22651-156e-4a5f-82f3-3aa7f3afc2b7-plugins-config-volume" pod "a7d22651-156e-4a5f-82f3-3aa7f3afc2b7" (UID: "a7d22651-156e-4a5f-82f3-3aa7f3afc2b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:14:19.477744 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:19.477687    5713 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b673d422-2d71-4f9d-bc6a-5bf0d3d09d90-plugins-config-volume\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:14:19.479319 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:19.479296    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7d22651-156e-4a5f-82f3-3aa7f3afc2b7-kube-api-access-l2jd6" pod "a7d22651-156e-4a5f-82f3-3aa7f3afc2b7" (UID: "a7d22651-156e-4a5f-82f3-3aa7f3afc2b7"). InnerVolumeSpecName "kube-api-access-l2jd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:14:19.479630 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:19.479598    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b673d422-2d71-4f9d-bc6a-5bf0d3d09d90-kube-api-access-9kl7n" pod "b673d422-2d71-4f9d-bc6a-5bf0d3d09d90" (UID: "b673d422-2d71-4f9d-bc6a-5bf0d3d09d90"). InnerVolumeSpecName "kube-api-access-9kl7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:14:19.589055 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:19.589006    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kl7n\" (UniqueName: \"kubernetes.io/projected/b673d422-2d71-4f9d-bc6a-5bf0d3d09d90-kube-api-access-9kl7n\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:14:19.589055 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:19.589053    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2jd6\" (UniqueName: \"kubernetes.io/projected/a7d22651-156e-4a5f-82f3-3aa7f3afc2b7-kube-api-access-l2jd6\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:14:19.589326 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:19.589074    5713 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7d22651-156e-4a5f-82f3-3aa7f3afc2b7-plugins-config-volume\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:14:19.721113 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:19.721078    5713 generic.go:356] "Generic (PLEG): container finished" podID="a7d22651-156e-4a5f-82f3-3aa7f3afc2b7" containerID="b81408ee16cb6c4ccbcde1bfa3cf0935562c54ac8669af9b0d26b0f19e4b6c6c" exitCode=0
Oct 30 21:14:19.721312 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:19.721151    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-255f05-85b6fd8bf9-wdzgm" event={"ID":"a7d22651-156e-4a5f-82f3-3aa7f3afc2b7","Type":"ContainerDied","Data":"b81408ee16cb6c4ccbcde1bfa3cf0935562c54ac8669af9b0d26b0f19e4b6c6c"}
Oct 30 21:14:19.721312 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:19.721168    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-255f05-85b6fd8bf9-wdzgm"
Oct 30 21:14:19.721312 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:19.721236    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-255f05-85b6fd8bf9-wdzgm" event={"ID":"a7d22651-156e-4a5f-82f3-3aa7f3afc2b7","Type":"ContainerDied","Data":"30f5abb67174c46c622b7c46364448eb2353929be3b68af83d12138c440b63f0"}
Oct 30 21:14:19.721312 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:19.721253    5713 scope.go:122] "RemoveContainer" containerID="b81408ee16cb6c4ccbcde1bfa3cf0935562c54ac8669af9b0d26b0f19e4b6c6c"
Oct 30 21:14:19.723212 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:19.723184    5713 generic.go:356] "Generic (PLEG): container finished" podID="b673d422-2d71-4f9d-bc6a-5bf0d3d09d90" containerID="7b3efda7df5ed1a2bc25088c8f89f63289c6de3ba8501285602de7bcaaf385d9" exitCode=0
Oct 30 21:14:19.723347 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:19.723235    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-255f05-677cd4895b-ncgcl"
Oct 30 21:14:19.723347 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:19.723281    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-255f05-677cd4895b-ncgcl" event={"ID":"b673d422-2d71-4f9d-bc6a-5bf0d3d09d90","Type":"ContainerDied","Data":"7b3efda7df5ed1a2bc25088c8f89f63289c6de3ba8501285602de7bcaaf385d9"}
Oct 30 21:14:19.723347 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:19.723330    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-255f05-677cd4895b-ncgcl" event={"ID":"b673d422-2d71-4f9d-bc6a-5bf0d3d09d90","Type":"ContainerDied","Data":"fb2cf335c99b3034bab512bbb1a6aa7f14b5489f59b243c91f4cfb736421a1ba"}
Oct 30 21:14:19.759846 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:19.759821    5713 scope.go:122] "RemoveContainer" containerID="b81408ee16cb6c4ccbcde1bfa3cf0935562c54ac8669af9b0d26b0f19e4b6c6c"
Oct 30 21:14:19.760596 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:14:19.760565    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b81408ee16cb6c4ccbcde1bfa3cf0935562c54ac8669af9b0d26b0f19e4b6c6c\": container with ID starting with b81408ee16cb6c4ccbcde1bfa3cf0935562c54ac8669af9b0d26b0f19e4b6c6c not found: ID does not exist" containerID="b81408ee16cb6c4ccbcde1bfa3cf0935562c54ac8669af9b0d26b0f19e4b6c6c"
Oct 30 21:14:19.760714 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:19.760605    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b81408ee16cb6c4ccbcde1bfa3cf0935562c54ac8669af9b0d26b0f19e4b6c6c"} err="failed to get container status \"b81408ee16cb6c4ccbcde1bfa3cf0935562c54ac8669af9b0d26b0f19e4b6c6c\": rpc error: code = NotFound desc = could not find container \"b81408ee16cb6c4ccbcde1bfa3cf0935562c54ac8669af9b0d26b0f19e4b6c6c\": container with ID starting with b81408ee16cb6c4ccbcde1bfa3cf0935562c54ac8669af9b0d26b0f19e4b6c6c not found: ID does not exist"
Oct 30 21:14:19.760714 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:19.760630    5713 scope.go:122] "RemoveContainer" containerID="7b3efda7df5ed1a2bc25088c8f89f63289c6de3ba8501285602de7bcaaf385d9"
Oct 30 21:14:19.773746 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:19.773449    5713 scope.go:122] "RemoveContainer" containerID="7b3efda7df5ed1a2bc25088c8f89f63289c6de3ba8501285602de7bcaaf385d9"
Oct 30 21:14:19.773848 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:14:19.773744    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b3efda7df5ed1a2bc25088c8f89f63289c6de3ba8501285602de7bcaaf385d9\": container with ID starting with 7b3efda7df5ed1a2bc25088c8f89f63289c6de3ba8501285602de7bcaaf385d9 not found: ID does not exist" containerID="7b3efda7df5ed1a2bc25088c8f89f63289c6de3ba8501285602de7bcaaf385d9"
Oct 30 21:14:19.773848 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:19.773773    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b3efda7df5ed1a2bc25088c8f89f63289c6de3ba8501285602de7bcaaf385d9"} err="failed to get container status \"7b3efda7df5ed1a2bc25088c8f89f63289c6de3ba8501285602de7bcaaf385d9\": rpc error: code = NotFound desc = could not find container \"7b3efda7df5ed1a2bc25088c8f89f63289c6de3ba8501285602de7bcaaf385d9\": container with ID starting with 7b3efda7df5ed1a2bc25088c8f89f63289c6de3ba8501285602de7bcaaf385d9 not found: ID does not exist"
Oct 30 21:14:19.889242 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:19.889203    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-255f05-677cd4895b-ncgcl"]
Oct 30 21:14:19.893130 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:19.893095    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-255f05-677cd4895b-ncgcl"]
Oct 30 21:14:20.010265 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:20.010177    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-255f05-85b6fd8bf9-wdzgm"]
Oct 30 21:14:20.014115 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:20.014092    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-255f05-85b6fd8bf9-wdzgm"]
Oct 30 21:14:20.090447 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:20.090413    5713 patch_prober.go:29] interesting pod/e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Oct 30 21:14:20.090638 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:20.090485    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk" podUID="237d222b-50be-4b97-b53d-49187434c09b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Oct 30 21:14:20.466533 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:20.466499    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-255f05-jvw7k"]
Oct 30 21:14:20.470432 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:20.470402    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-255f05-jvw7k"]
Oct 30 21:14:21.091823 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:21.090575    5713 patch_prober.go:29] interesting pod/e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Oct 30 21:14:21.091823 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:21.090668    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk" podUID="237d222b-50be-4b97-b53d-49187434c09b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Oct 30 21:14:21.093801 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:21.093755    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66ce7582-3ef0-47b6-9a48-99a988749de4" path="/var/lib/kubelet/pods/66ce7582-3ef0-47b6-9a48-99a988749de4/volumes"
Oct 30 21:14:21.094490 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:21.094466    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7d22651-156e-4a5f-82f3-3aa7f3afc2b7" path="/var/lib/kubelet/pods/a7d22651-156e-4a5f-82f3-3aa7f3afc2b7/volumes"
Oct 30 21:14:21.095399 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:21.095367    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b673d422-2d71-4f9d-bc6a-5bf0d3d09d90" path="/var/lib/kubelet/pods/b673d422-2d71-4f9d-bc6a-5bf0d3d09d90/volumes"
Oct 30 21:14:22.229201 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:22.229170    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk"
Oct 30 21:14:22.230387 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:22.230100    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk"
Oct 30 21:14:22.231240 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:22.231218    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk"
Oct 30 21:14:24.335040 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:24.334995    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-677b45-9b4nw"]
Oct 30 21:14:24.335662 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:24.335648    5713 cpu_manager.go:414] "containerMap: removing container" podUID="b673d422-2d71-4f9d-bc6a-5bf0d3d09d90" containerName="payload-processing"
Oct 30 21:14:24.335707 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:24.335663    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b673d422-2d71-4f9d-bc6a-5bf0d3d09d90" containerName="payload-processing"
Oct 30 21:14:24.335707 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:24.335689    5713 cpu_manager.go:414] "containerMap: removing container" podUID="0da8c670-dd6c-4a19-856e-0241a7494dbb" containerName="istio-proxy"
Oct 30 21:14:24.335707 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:24.335694    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0da8c670-dd6c-4a19-856e-0241a7494dbb" containerName="istio-proxy"
Oct 30 21:14:24.335841 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:24.335725    5713 cpu_manager.go:414] "containerMap: removing container" podUID="a7d22651-156e-4a5f-82f3-3aa7f3afc2b7" containerName="payload-pre-processing"
Oct 30 21:14:24.335841 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:24.335730    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7d22651-156e-4a5f-82f3-3aa7f3afc2b7" containerName="payload-pre-processing"
Oct 30 21:14:24.335841 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:24.335839    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="b673d422-2d71-4f9d-bc6a-5bf0d3d09d90" containerName="payload-processing"
Oct 30 21:14:24.335951 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:24.335852    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="0da8c670-dd6c-4a19-856e-0241a7494dbb" containerName="istio-proxy"
Oct 30 21:14:24.335951 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:24.335864    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7d22651-156e-4a5f-82f3-3aa7f3afc2b7" containerName="payload-pre-processing"
Oct 30 21:14:24.341106 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:24.341083    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-677b45-9b4nw"
Oct 30 21:14:24.396488 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:24.396461    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-677b45-9b4nw"]
Oct 30 21:14:24.441227 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:24.441193    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:14:24.441227 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:24.441211    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:14:24.447601 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:24.447563    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:14:24.447740 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:24.447617    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:14:24.460317 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:14:24.460281    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:14:24.469334 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:24.469309    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b868dd4-50b6-44d9-b296-cf01ca7d1d92-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-677b45-9b4nw\" (UID: \"9b868dd4-50b6-44d9-b296-cf01ca7d1d92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-677b45-9b4nw"
Oct 30 21:14:24.469453 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:24.469340    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b868dd4-50b6-44d9-b296-cf01ca7d1d92-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-677b45-9b4nw\" (UID: \"9b868dd4-50b6-44d9-b296-cf01ca7d1d92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-677b45-9b4nw"
Oct 30 21:14:24.469532 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:24.469517    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76l75\" (UniqueName: \"kubernetes.io/projected/9b868dd4-50b6-44d9-b296-cf01ca7d1d92-kube-api-access-76l75\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-677b45-9b4nw\" (UID: \"9b868dd4-50b6-44d9-b296-cf01ca7d1d92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-677b45-9b4nw"
Oct 30 21:14:24.571155 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:24.571125    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b868dd4-50b6-44d9-b296-cf01ca7d1d92-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-677b45-9b4nw\" (UID: \"9b868dd4-50b6-44d9-b296-cf01ca7d1d92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-677b45-9b4nw"
Oct 30 21:14:24.571155 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:24.571158    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b868dd4-50b6-44d9-b296-cf01ca7d1d92-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-677b45-9b4nw\" (UID: \"9b868dd4-50b6-44d9-b296-cf01ca7d1d92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-677b45-9b4nw"
Oct 30 21:14:24.571351 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:24.571293    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76l75\" (UniqueName: \"kubernetes.io/projected/9b868dd4-50b6-44d9-b296-cf01ca7d1d92-kube-api-access-76l75\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-677b45-9b4nw\" (UID: \"9b868dd4-50b6-44d9-b296-cf01ca7d1d92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-677b45-9b4nw"
Oct 30 21:14:24.571920 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:24.571889    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b868dd4-50b6-44d9-b296-cf01ca7d1d92-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-677b45-9b4nw\" (UID: \"9b868dd4-50b6-44d9-b296-cf01ca7d1d92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-677b45-9b4nw"
Oct 30 21:14:24.573671 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:24.573652    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b868dd4-50b6-44d9-b296-cf01ca7d1d92-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-677b45-9b4nw\" (UID: \"9b868dd4-50b6-44d9-b296-cf01ca7d1d92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-677b45-9b4nw"
Oct 30 21:14:24.629173 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:24.629111    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76l75\" (UniqueName: \"kubernetes.io/projected/9b868dd4-50b6-44d9-b296-cf01ca7d1d92-kube-api-access-76l75\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-677b45-9b4nw\" (UID: \"9b868dd4-50b6-44d9-b296-cf01ca7d1d92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-677b45-9b4nw"
Oct 30 21:14:24.656734 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:24.656706    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-677b45-9b4nw"
Oct 30 21:14:25.063143 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:14:25.063114    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b868dd4_50b6_44d9_b296_cf01ca7d1d92.slice/crio-c5d67408cc5529f3a6250fb3a6610feb9cf0e8a3f96a0d8b7501c2e0729f46fa WatchSource:0}: Error finding container c5d67408cc5529f3a6250fb3a6610feb9cf0e8a3f96a0d8b7501c2e0729f46fa: Status 404 returned error can't find the container with id c5d67408cc5529f3a6250fb3a6610feb9cf0e8a3f96a0d8b7501c2e0729f46fa
Oct 30 21:14:25.068034 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:25.067909    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-677b45-9b4nw"]
Oct 30 21:14:25.803329 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:25.803288    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-677b45-9b4nw" event={"ID":"9b868dd4-50b6-44d9-b296-cf01ca7d1d92","Type":"ContainerStarted","Data":"127e3937b250e69bcfd4c7c03f0ac649d2731f4c7382a2a17153e7993c5c325f"}
Oct 30 21:14:25.803757 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:25.803351    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-677b45-9b4nw" event={"ID":"9b868dd4-50b6-44d9-b296-cf01ca7d1d92","Type":"ContainerStarted","Data":"c5d67408cc5529f3a6250fb3a6610feb9cf0e8a3f96a0d8b7501c2e0729f46fa"}
Oct 30 21:14:26.813986 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:26.813950    5713 generic.go:356] "Generic (PLEG): container finished" podID="9b868dd4-50b6-44d9-b296-cf01ca7d1d92" containerID="127e3937b250e69bcfd4c7c03f0ac649d2731f4c7382a2a17153e7993c5c325f" exitCode=0
Oct 30 21:14:26.814414 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:26.814028    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-677b45-9b4nw" event={"ID":"9b868dd4-50b6-44d9-b296-cf01ca7d1d92","Type":"ContainerDied","Data":"127e3937b250e69bcfd4c7c03f0ac649d2731f4c7382a2a17153e7993c5c325f"}
Oct 30 21:14:28.025944 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:28.025911    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-677b45-9b4nw"
Oct 30 21:14:28.128430 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:28.128338    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b868dd4-50b6-44d9-b296-cf01ca7d1d92-kube-api-access-76l75\" (UniqueName: \"kubernetes.io/projected/9b868dd4-50b6-44d9-b296-cf01ca7d1d92-kube-api-access-76l75\") pod \"9b868dd4-50b6-44d9-b296-cf01ca7d1d92\" (UID: \"9b868dd4-50b6-44d9-b296-cf01ca7d1d92\") "
Oct 30 21:14:28.128430 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:28.128405    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b868dd4-50b6-44d9-b296-cf01ca7d1d92-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b868dd4-50b6-44d9-b296-cf01ca7d1d92-maas-api-xks-ca\") pod \"9b868dd4-50b6-44d9-b296-cf01ca7d1d92\" (UID: \"9b868dd4-50b6-44d9-b296-cf01ca7d1d92\") "
Oct 30 21:14:28.128670 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:28.128615    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b868dd4-50b6-44d9-b296-cf01ca7d1d92-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b868dd4-50b6-44d9-b296-cf01ca7d1d92-maas-api-openshift-ca\") pod \"9b868dd4-50b6-44d9-b296-cf01ca7d1d92\" (UID: \"9b868dd4-50b6-44d9-b296-cf01ca7d1d92\") "
Oct 30 21:14:28.129069 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:28.129038    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b868dd4-50b6-44d9-b296-cf01ca7d1d92-maas-api-openshift-ca" pod "9b868dd4-50b6-44d9-b296-cf01ca7d1d92" (UID: "9b868dd4-50b6-44d9-b296-cf01ca7d1d92"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:14:28.129555 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:28.129528    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b868dd4-50b6-44d9-b296-cf01ca7d1d92-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:14:28.131049 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:28.131024    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b868dd4-50b6-44d9-b296-cf01ca7d1d92-maas-api-xks-ca" pod "9b868dd4-50b6-44d9-b296-cf01ca7d1d92" (UID: "9b868dd4-50b6-44d9-b296-cf01ca7d1d92"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:14:28.131148 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:28.131109    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b868dd4-50b6-44d9-b296-cf01ca7d1d92-kube-api-access-76l75" pod "9b868dd4-50b6-44d9-b296-cf01ca7d1d92" (UID: "9b868dd4-50b6-44d9-b296-cf01ca7d1d92"). InnerVolumeSpecName "kube-api-access-76l75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:14:28.231014 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:28.230976    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76l75\" (UniqueName: \"kubernetes.io/projected/9b868dd4-50b6-44d9-b296-cf01ca7d1d92-kube-api-access-76l75\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:14:28.231014 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:28.231015    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b868dd4-50b6-44d9-b296-cf01ca7d1d92-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:14:28.833200 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:28.833168    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-677b45-9b4nw" event={"ID":"9b868dd4-50b6-44d9-b296-cf01ca7d1d92","Type":"ContainerDied","Data":"c5d67408cc5529f3a6250fb3a6610feb9cf0e8a3f96a0d8b7501c2e0729f46fa"}
Oct 30 21:14:28.833200 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:28.833192    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-677b45-9b4nw"
Oct 30 21:14:28.833425 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:28.833220    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5d67408cc5529f3a6250fb3a6610feb9cf0e8a3f96a0d8b7501c2e0729f46fa"
Oct 30 21:14:29.258234 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:29.258191    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d061d0-sq9mk"]
Oct 30 21:14:29.259121 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:29.259097    5713 cpu_manager.go:414] "containerMap: removing container" podUID="9b868dd4-50b6-44d9-b296-cf01ca7d1d92" containerName="revoke-keys"
Oct 30 21:14:29.259121 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:29.259120    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b868dd4-50b6-44d9-b296-cf01ca7d1d92" containerName="revoke-keys"
Oct 30 21:14:29.259347 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:29.259332    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b868dd4-50b6-44d9-b296-cf01ca7d1d92" containerName="revoke-keys"
Oct 30 21:14:29.263126 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:29.263103    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d061d0-sq9mk"
Oct 30 21:14:29.349642 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:29.349578    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecf576f4-8582-4ddc-8768-8c04888e5e4b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-d061d0-sq9mk\" (UID: \"ecf576f4-8582-4ddc-8768-8c04888e5e4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d061d0-sq9mk"
Oct 30 21:14:29.349831 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:29.349755    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecf576f4-8582-4ddc-8768-8c04888e5e4b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-d061d0-sq9mk\" (UID: \"ecf576f4-8582-4ddc-8768-8c04888e5e4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d061d0-sq9mk"
Oct 30 21:14:29.349898 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:29.349872    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgvkl\" (UniqueName: \"kubernetes.io/projected/ecf576f4-8582-4ddc-8768-8c04888e5e4b-kube-api-access-dgvkl\") pod \"maas-api-revoke-keys-e2e-worker-w2-d061d0-sq9mk\" (UID: \"ecf576f4-8582-4ddc-8768-8c04888e5e4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d061d0-sq9mk"
Oct 30 21:14:29.367460 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:29.367427    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:14:29.367638 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:29.367459    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:14:29.367638 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:29.367464    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:14:29.367638 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:29.367592    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:14:29.371896 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:29.371867    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d061d0-sq9mk"]
Oct 30 21:14:29.451173 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:29.451136    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecf576f4-8582-4ddc-8768-8c04888e5e4b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-d061d0-sq9mk\" (UID: \"ecf576f4-8582-4ddc-8768-8c04888e5e4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d061d0-sq9mk"
Oct 30 21:14:29.451366 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:29.451244    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecf576f4-8582-4ddc-8768-8c04888e5e4b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-d061d0-sq9mk\" (UID: \"ecf576f4-8582-4ddc-8768-8c04888e5e4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d061d0-sq9mk"
Oct 30 21:14:29.451366 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:29.451287    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgvkl\" (UniqueName: \"kubernetes.io/projected/ecf576f4-8582-4ddc-8768-8c04888e5e4b-kube-api-access-dgvkl\") pod \"maas-api-revoke-keys-e2e-worker-w2-d061d0-sq9mk\" (UID: \"ecf576f4-8582-4ddc-8768-8c04888e5e4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d061d0-sq9mk"
Oct 30 21:14:29.451945 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:29.451924    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecf576f4-8582-4ddc-8768-8c04888e5e4b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-d061d0-sq9mk\" (UID: \"ecf576f4-8582-4ddc-8768-8c04888e5e4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d061d0-sq9mk"
Oct 30 21:14:29.453766 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:29.453744    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecf576f4-8582-4ddc-8768-8c04888e5e4b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-d061d0-sq9mk\" (UID: \"ecf576f4-8582-4ddc-8768-8c04888e5e4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d061d0-sq9mk"
Oct 30 21:14:29.509263 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:29.509187    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgvkl\" (UniqueName: \"kubernetes.io/projected/ecf576f4-8582-4ddc-8768-8c04888e5e4b-kube-api-access-dgvkl\") pod \"maas-api-revoke-keys-e2e-worker-w2-d061d0-sq9mk\" (UID: \"ecf576f4-8582-4ddc-8768-8c04888e5e4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d061d0-sq9mk"
Oct 30 21:14:29.578067 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:29.578036    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d061d0-sq9mk"
Oct 30 21:14:29.980657 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:29.980622    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d061d0-sq9mk"]
Oct 30 21:14:29.981595 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:14:29.981558    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecf576f4_8582_4ddc_8768_8c04888e5e4b.slice/crio-6e139a4e9d27d4c544f829112d37fdedb2f06e0a6d5319c790adc01ed050b3fc WatchSource:0}: Error finding container 6e139a4e9d27d4c544f829112d37fdedb2f06e0a6d5319c790adc01ed050b3fc: Status 404 returned error can't find the container with id 6e139a4e9d27d4c544f829112d37fdedb2f06e0a6d5319c790adc01ed050b3fc
Oct 30 21:14:30.851604 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:30.851570    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d061d0-sq9mk" event={"ID":"ecf576f4-8582-4ddc-8768-8c04888e5e4b","Type":"ContainerStarted","Data":"b0fb5920965f2e3c4da209c036db168ff00c88e2d4eee8084db343db1e8a2003"}
Oct 30 21:14:30.852115 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:30.851627    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d061d0-sq9mk" event={"ID":"ecf576f4-8582-4ddc-8768-8c04888e5e4b","Type":"ContainerStarted","Data":"6e139a4e9d27d4c544f829112d37fdedb2f06e0a6d5319c790adc01ed050b3fc"}
Oct 30 21:14:30.967375 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:30.967325    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d061d0-sq9mk" podStartSLOduration=1.9673116080000002 podStartE2EDuration="1.967311608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:14:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:14:30.967059808 +0000 UTC m=+2154.031836206" watchObservedRunningTime="2026-10-30 21:14:30.967311608 +0000 UTC m=+2154.032088005"
Oct 30 21:14:31.862566 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:31.862527    5713 generic.go:356] "Generic (PLEG): container finished" podID="ecf576f4-8582-4ddc-8768-8c04888e5e4b" containerID="b0fb5920965f2e3c4da209c036db168ff00c88e2d4eee8084db343db1e8a2003" exitCode=0
Oct 30 21:14:31.862985 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:31.862613    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d061d0-sq9mk" event={"ID":"ecf576f4-8582-4ddc-8768-8c04888e5e4b","Type":"ContainerDied","Data":"b0fb5920965f2e3c4da209c036db168ff00c88e2d4eee8084db343db1e8a2003"}
Oct 30 21:14:33.051327 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:33.051300    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d061d0-sq9mk"
Oct 30 21:14:33.210514 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:33.210487    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecf576f4-8582-4ddc-8768-8c04888e5e4b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecf576f4-8582-4ddc-8768-8c04888e5e4b-maas-api-openshift-ca\") pod \"ecf576f4-8582-4ddc-8768-8c04888e5e4b\" (UID: \"ecf576f4-8582-4ddc-8768-8c04888e5e4b\") "
Oct 30 21:14:33.210719 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:33.210578    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecf576f4-8582-4ddc-8768-8c04888e5e4b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecf576f4-8582-4ddc-8768-8c04888e5e4b-maas-api-xks-ca\") pod \"ecf576f4-8582-4ddc-8768-8c04888e5e4b\" (UID: \"ecf576f4-8582-4ddc-8768-8c04888e5e4b\") "
Oct 30 21:14:33.210719 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:33.210707    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecf576f4-8582-4ddc-8768-8c04888e5e4b-kube-api-access-dgvkl\" (UniqueName: \"kubernetes.io/projected/ecf576f4-8582-4ddc-8768-8c04888e5e4b-kube-api-access-dgvkl\") pod \"ecf576f4-8582-4ddc-8768-8c04888e5e4b\" (UID: \"ecf576f4-8582-4ddc-8768-8c04888e5e4b\") "
Oct 30 21:14:33.210999 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:33.210966    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecf576f4-8582-4ddc-8768-8c04888e5e4b-maas-api-openshift-ca" pod "ecf576f4-8582-4ddc-8768-8c04888e5e4b" (UID: "ecf576f4-8582-4ddc-8768-8c04888e5e4b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:14:33.212140 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:33.212116    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecf576f4-8582-4ddc-8768-8c04888e5e4b-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:14:33.213136 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:33.213106    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecf576f4-8582-4ddc-8768-8c04888e5e4b-maas-api-xks-ca" pod "ecf576f4-8582-4ddc-8768-8c04888e5e4b" (UID: "ecf576f4-8582-4ddc-8768-8c04888e5e4b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:14:33.214430 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:33.214397    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecf576f4-8582-4ddc-8768-8c04888e5e4b-kube-api-access-dgvkl" pod "ecf576f4-8582-4ddc-8768-8c04888e5e4b" (UID: "ecf576f4-8582-4ddc-8768-8c04888e5e4b"). InnerVolumeSpecName "kube-api-access-dgvkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:14:33.314094 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:33.314063    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecf576f4-8582-4ddc-8768-8c04888e5e4b-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:14:33.314094 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:33.314090    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgvkl\" (UniqueName: \"kubernetes.io/projected/ecf576f4-8582-4ddc-8768-8c04888e5e4b-kube-api-access-dgvkl\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:14:33.897308 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:33.897277    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d061d0-sq9mk"
Oct 30 21:14:33.897308 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:33.897301    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d061d0-sq9mk" event={"ID":"ecf576f4-8582-4ddc-8768-8c04888e5e4b","Type":"ContainerDied","Data":"6e139a4e9d27d4c544f829112d37fdedb2f06e0a6d5319c790adc01ed050b3fc"}
Oct 30 21:14:33.897561 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:33.897372    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e139a4e9d27d4c544f829112d37fdedb2f06e0a6d5319c790adc01ed050b3fc"
Oct 30 21:14:33.965868 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:33.965818    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7k46d"]
Oct 30 21:14:33.966045 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:33.966024    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7k46d" podUID="4242f534-4d88-40d1-a4e0-e45493a00d7a" containerName="istio-proxy" containerID="cri-o://940268c02595e8de006da84aabe30fe4c50150ccc152264561e4d80fbb6b2bca" gracePeriod=30
Oct 30 21:14:34.022089 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:34.022052    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6nvgb"]
Oct 30 21:14:34.022673 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:34.022632    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6nvgb" podUID="74a166cd-7a71-449c-9ec1-68950b90e678" containerName="istio-proxy" containerID="cri-o://ee2b358442f429882ecde2493a4d04ffe513f309b2d2bdf43fd7429a3965a5e4" gracePeriod=30
Oct 30 21:14:35.913209 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:35.913176    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-677b45-955d68994-fdprd"]
Oct 30 21:14:38.754010 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:38.753974    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-6nvgb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:14:38.754385 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:38.754044    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6nvgb" podUID="74a166cd-7a71-449c-9ec1-68950b90e678" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:14:38.798299 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:38.798269    5713 scope.go:122] "RemoveContainer" containerID="94cff754d8a6cf5cfe7ee7b26444ae6c4d46e15361940e6a4c03da889d1844c7"
Oct 30 21:14:38.891539 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:38.891496    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-7k46d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:14:38.891697 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:38.891594    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7k46d" podUID="4242f534-4d88-40d1-a4e0-e45493a00d7a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:14:39.266137 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.266106    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6nvgb"
Oct 30 21:14:39.304430 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.304405    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7k46d"
Oct 30 21:14:39.313459 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.313384    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74a166cd-7a71-449c-9ec1-68950b90e678-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74a166cd-7a71-449c-9ec1-68950b90e678-workload-certs\") pod \"74a166cd-7a71-449c-9ec1-68950b90e678\" (UID: \"74a166cd-7a71-449c-9ec1-68950b90e678\") "
Oct 30 21:14:39.313459 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.313440    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74a166cd-7a71-449c-9ec1-68950b90e678-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74a166cd-7a71-449c-9ec1-68950b90e678-istio-data\") pod \"74a166cd-7a71-449c-9ec1-68950b90e678\" (UID: \"74a166cd-7a71-449c-9ec1-68950b90e678\") "
Oct 30 21:14:39.313650 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.313522    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74a166cd-7a71-449c-9ec1-68950b90e678-kube-api-access-tsrb7\" (UniqueName: \"kubernetes.io/projected/74a166cd-7a71-449c-9ec1-68950b90e678-kube-api-access-tsrb7\") pod \"74a166cd-7a71-449c-9ec1-68950b90e678\" (UID: \"74a166cd-7a71-449c-9ec1-68950b90e678\") "
Oct 30 21:14:39.313650 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.313551    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/74a166cd-7a71-449c-9ec1-68950b90e678-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74a166cd-7a71-449c-9ec1-68950b90e678-istio-podinfo\") pod \"74a166cd-7a71-449c-9ec1-68950b90e678\" (UID: \"74a166cd-7a71-449c-9ec1-68950b90e678\") "
Oct 30 21:14:39.313759 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.313660    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74a166cd-7a71-449c-9ec1-68950b90e678-workload-certs" pod "74a166cd-7a71-449c-9ec1-68950b90e678" (UID: "74a166cd-7a71-449c-9ec1-68950b90e678"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:14:39.313759 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.313682    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74a166cd-7a71-449c-9ec1-68950b90e678-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a166cd-7a71-449c-9ec1-68950b90e678-credential-socket\") pod \"74a166cd-7a71-449c-9ec1-68950b90e678\" (UID: \"74a166cd-7a71-449c-9ec1-68950b90e678\") "
Oct 30 21:14:39.313759 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.313695    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74a166cd-7a71-449c-9ec1-68950b90e678-istio-data" pod "74a166cd-7a71-449c-9ec1-68950b90e678" (UID: "74a166cd-7a71-449c-9ec1-68950b90e678"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:14:39.313948 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.313887    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74a166cd-7a71-449c-9ec1-68950b90e678-istio-token\" (UniqueName: \"kubernetes.io/projected/74a166cd-7a71-449c-9ec1-68950b90e678-istio-token\") pod \"74a166cd-7a71-449c-9ec1-68950b90e678\" (UID: \"74a166cd-7a71-449c-9ec1-68950b90e678\") "
Oct 30 21:14:39.313948 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.313925    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74a166cd-7a71-449c-9ec1-68950b90e678-credential-socket" pod "74a166cd-7a71-449c-9ec1-68950b90e678" (UID: "74a166cd-7a71-449c-9ec1-68950b90e678"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:14:39.314068 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.313951    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74a166cd-7a71-449c-9ec1-68950b90e678-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74a166cd-7a71-449c-9ec1-68950b90e678-istio-envoy\") pod \"74a166cd-7a71-449c-9ec1-68950b90e678\" (UID: \"74a166cd-7a71-449c-9ec1-68950b90e678\") "
Oct 30 21:14:39.314068 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.313989    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74a166cd-7a71-449c-9ec1-68950b90e678-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74a166cd-7a71-449c-9ec1-68950b90e678-istiod-ca-cert\") pod \"74a166cd-7a71-449c-9ec1-68950b90e678\" (UID: \"74a166cd-7a71-449c-9ec1-68950b90e678\") "
Oct 30 21:14:39.314068 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.314053    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74a166cd-7a71-449c-9ec1-68950b90e678-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a166cd-7a71-449c-9ec1-68950b90e678-workload-socket\") pod \"74a166cd-7a71-449c-9ec1-68950b90e678\" (UID: \"74a166cd-7a71-449c-9ec1-68950b90e678\") "
Oct 30 21:14:39.314594 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.314491    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74a166cd-7a71-449c-9ec1-68950b90e678-workload-socket" pod "74a166cd-7a71-449c-9ec1-68950b90e678" (UID: "74a166cd-7a71-449c-9ec1-68950b90e678"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:14:39.314594 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.314570    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74a166cd-7a71-449c-9ec1-68950b90e678-istiod-ca-cert" pod "74a166cd-7a71-449c-9ec1-68950b90e678" (UID: "74a166cd-7a71-449c-9ec1-68950b90e678"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:14:39.315040 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.315016    5713 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a166cd-7a71-449c-9ec1-68950b90e678-credential-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:14:39.315116 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.315047    5713 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74a166cd-7a71-449c-9ec1-68950b90e678-istiod-ca-cert\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:14:39.315116 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.315062    5713 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a166cd-7a71-449c-9ec1-68950b90e678-workload-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:14:39.315116 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.315077    5713 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74a166cd-7a71-449c-9ec1-68950b90e678-workload-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:14:39.315116 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.315092    5713 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74a166cd-7a71-449c-9ec1-68950b90e678-istio-data\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:14:39.315950 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.315923    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74a166cd-7a71-449c-9ec1-68950b90e678-istio-podinfo" pod "74a166cd-7a71-449c-9ec1-68950b90e678" (UID: "74a166cd-7a71-449c-9ec1-68950b90e678"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:14:39.316048 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.316030    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74a166cd-7a71-449c-9ec1-68950b90e678-istio-token" pod "74a166cd-7a71-449c-9ec1-68950b90e678" (UID: "74a166cd-7a71-449c-9ec1-68950b90e678"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:14:39.316106 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.316045    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74a166cd-7a71-449c-9ec1-68950b90e678-kube-api-access-tsrb7" pod "74a166cd-7a71-449c-9ec1-68950b90e678" (UID: "74a166cd-7a71-449c-9ec1-68950b90e678"). InnerVolumeSpecName "kube-api-access-tsrb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:14:39.316142 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.316117    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74a166cd-7a71-449c-9ec1-68950b90e678-istio-envoy" pod "74a166cd-7a71-449c-9ec1-68950b90e678" (UID: "74a166cd-7a71-449c-9ec1-68950b90e678"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:14:39.416049 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.416017    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4242f534-4d88-40d1-a4e0-e45493a00d7a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4242f534-4d88-40d1-a4e0-e45493a00d7a-credential-socket\") pod \"4242f534-4d88-40d1-a4e0-e45493a00d7a\" (UID: \"4242f534-4d88-40d1-a4e0-e45493a00d7a\") "
Oct 30 21:14:39.416230 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.416125    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4242f534-4d88-40d1-a4e0-e45493a00d7a-kube-api-access-q6sjv\" (UniqueName: \"kubernetes.io/projected/4242f534-4d88-40d1-a4e0-e45493a00d7a-kube-api-access-q6sjv\") pod \"4242f534-4d88-40d1-a4e0-e45493a00d7a\" (UID: \"4242f534-4d88-40d1-a4e0-e45493a00d7a\") "
Oct 30 21:14:39.416230 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.416171    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4242f534-4d88-40d1-a4e0-e45493a00d7a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4242f534-4d88-40d1-a4e0-e45493a00d7a-istio-podinfo\") pod \"4242f534-4d88-40d1-a4e0-e45493a00d7a\" (UID: \"4242f534-4d88-40d1-a4e0-e45493a00d7a\") "
Oct 30 21:14:39.416230 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.416204    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4242f534-4d88-40d1-a4e0-e45493a00d7a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4242f534-4d88-40d1-a4e0-e45493a00d7a-istio-data\") pod \"4242f534-4d88-40d1-a4e0-e45493a00d7a\" (UID: \"4242f534-4d88-40d1-a4e0-e45493a00d7a\") "
Oct 30 21:14:39.416432 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.416279    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4242f534-4d88-40d1-a4e0-e45493a00d7a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4242f534-4d88-40d1-a4e0-e45493a00d7a-istio-envoy\") pod \"4242f534-4d88-40d1-a4e0-e45493a00d7a\" (UID: \"4242f534-4d88-40d1-a4e0-e45493a00d7a\") "
Oct 30 21:14:39.416432 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.416320    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4242f534-4d88-40d1-a4e0-e45493a00d7a-credential-socket" pod "4242f534-4d88-40d1-a4e0-e45493a00d7a" (UID: "4242f534-4d88-40d1-a4e0-e45493a00d7a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:14:39.416432 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.416338    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4242f534-4d88-40d1-a4e0-e45493a00d7a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4242f534-4d88-40d1-a4e0-e45493a00d7a-workload-certs\") pod \"4242f534-4d88-40d1-a4e0-e45493a00d7a\" (UID: \"4242f534-4d88-40d1-a4e0-e45493a00d7a\") "
Oct 30 21:14:39.416432 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.416372    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4242f534-4d88-40d1-a4e0-e45493a00d7a-istio-token\" (UniqueName: \"kubernetes.io/projected/4242f534-4d88-40d1-a4e0-e45493a00d7a-istio-token\") pod \"4242f534-4d88-40d1-a4e0-e45493a00d7a\" (UID: \"4242f534-4d88-40d1-a4e0-e45493a00d7a\") "
Oct 30 21:14:39.416657 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.416502    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4242f534-4d88-40d1-a4e0-e45493a00d7a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4242f534-4d88-40d1-a4e0-e45493a00d7a-istiod-ca-cert\") pod \"4242f534-4d88-40d1-a4e0-e45493a00d7a\" (UID: \"4242f534-4d88-40d1-a4e0-e45493a00d7a\") "
Oct 30 21:14:39.416657 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.416584    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4242f534-4d88-40d1-a4e0-e45493a00d7a-istio-data" pod "4242f534-4d88-40d1-a4e0-e45493a00d7a" (UID: "4242f534-4d88-40d1-a4e0-e45493a00d7a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:14:39.416657 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.416579    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4242f534-4d88-40d1-a4e0-e45493a00d7a-workload-certs" pod "4242f534-4d88-40d1-a4e0-e45493a00d7a" (UID: "4242f534-4d88-40d1-a4e0-e45493a00d7a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:14:39.416657 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.416640    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4242f534-4d88-40d1-a4e0-e45493a00d7a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4242f534-4d88-40d1-a4e0-e45493a00d7a-workload-socket\") pod \"4242f534-4d88-40d1-a4e0-e45493a00d7a\" (UID: \"4242f534-4d88-40d1-a4e0-e45493a00d7a\") "
Oct 30 21:14:39.417039 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.416972    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4242f534-4d88-40d1-a4e0-e45493a00d7a-istiod-ca-cert" pod "4242f534-4d88-40d1-a4e0-e45493a00d7a" (UID: "4242f534-4d88-40d1-a4e0-e45493a00d7a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:14:39.417180 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.417046    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4242f534-4d88-40d1-a4e0-e45493a00d7a-workload-socket" pod "4242f534-4d88-40d1-a4e0-e45493a00d7a" (UID: "4242f534-4d88-40d1-a4e0-e45493a00d7a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:14:39.417424 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.417404    5713 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74a166cd-7a71-449c-9ec1-68950b90e678-istio-envoy\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:14:39.417476 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.417434    5713 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4242f534-4d88-40d1-a4e0-e45493a00d7a-istiod-ca-cert\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:14:39.417476 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.417451    5713 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4242f534-4d88-40d1-a4e0-e45493a00d7a-workload-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:14:39.417476 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.417467    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsrb7\" (UniqueName: \"kubernetes.io/projected/74a166cd-7a71-449c-9ec1-68950b90e678-kube-api-access-tsrb7\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:14:39.417572 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.417484    5713 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74a166cd-7a71-449c-9ec1-68950b90e678-istio-podinfo\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:14:39.417572 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.417499    5713 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4242f534-4d88-40d1-a4e0-e45493a00d7a-credential-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:14:39.417572 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.417513    5713 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4242f534-4d88-40d1-a4e0-e45493a00d7a-istio-data\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:14:39.417572 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.417528    5713 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4242f534-4d88-40d1-a4e0-e45493a00d7a-workload-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:14:39.417572 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.417543    5713 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74a166cd-7a71-449c-9ec1-68950b90e678-istio-token\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:14:39.418626 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.418596    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4242f534-4d88-40d1-a4e0-e45493a00d7a-istio-podinfo" pod "4242f534-4d88-40d1-a4e0-e45493a00d7a" (UID: "4242f534-4d88-40d1-a4e0-e45493a00d7a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:14:39.418703 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.418601    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4242f534-4d88-40d1-a4e0-e45493a00d7a-istio-envoy" pod "4242f534-4d88-40d1-a4e0-e45493a00d7a" (UID: "4242f534-4d88-40d1-a4e0-e45493a00d7a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:14:39.419129 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.419111    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4242f534-4d88-40d1-a4e0-e45493a00d7a-kube-api-access-q6sjv" pod "4242f534-4d88-40d1-a4e0-e45493a00d7a" (UID: "4242f534-4d88-40d1-a4e0-e45493a00d7a"). InnerVolumeSpecName "kube-api-access-q6sjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:14:39.419203 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.419138    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4242f534-4d88-40d1-a4e0-e45493a00d7a-istio-token" pod "4242f534-4d88-40d1-a4e0-e45493a00d7a" (UID: "4242f534-4d88-40d1-a4e0-e45493a00d7a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:14:39.519447 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.519416    5713 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4242f534-4d88-40d1-a4e0-e45493a00d7a-istio-podinfo\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:14:39.519447 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.519445    5713 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4242f534-4d88-40d1-a4e0-e45493a00d7a-istio-envoy\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:14:39.519447 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.519457    5713 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4242f534-4d88-40d1-a4e0-e45493a00d7a-istio-token\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:14:39.519698 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.519467    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6sjv\" (UniqueName: \"kubernetes.io/projected/4242f534-4d88-40d1-a4e0-e45493a00d7a-kube-api-access-q6sjv\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:14:39.948994 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.948957    5713 generic.go:356] "Generic (PLEG): container finished" podID="74a166cd-7a71-449c-9ec1-68950b90e678" containerID="ee2b358442f429882ecde2493a4d04ffe513f309b2d2bdf43fd7429a3965a5e4" exitCode=0
Oct 30 21:14:39.949464 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.949025    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6nvgb"
Oct 30 21:14:39.949464 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.949045    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6nvgb" event={"ID":"74a166cd-7a71-449c-9ec1-68950b90e678","Type":"ContainerDied","Data":"ee2b358442f429882ecde2493a4d04ffe513f309b2d2bdf43fd7429a3965a5e4"}
Oct 30 21:14:39.949464 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.949104    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6nvgb" event={"ID":"74a166cd-7a71-449c-9ec1-68950b90e678","Type":"ContainerDied","Data":"6c80ca056eab2c6b1f1ef5f7e965500abc2a94e69e129d32a9f610f0cdaeb58b"}
Oct 30 21:14:39.949464 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.949129    5713 scope.go:122] "RemoveContainer" containerID="ee2b358442f429882ecde2493a4d04ffe513f309b2d2bdf43fd7429a3965a5e4"
Oct 30 21:14:39.950923 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.950890    5713 generic.go:356] "Generic (PLEG): container finished" podID="4242f534-4d88-40d1-a4e0-e45493a00d7a" containerID="940268c02595e8de006da84aabe30fe4c50150ccc152264561e4d80fbb6b2bca" exitCode=0
Oct 30 21:14:39.951040 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.950941    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7k46d" event={"ID":"4242f534-4d88-40d1-a4e0-e45493a00d7a","Type":"ContainerDied","Data":"940268c02595e8de006da84aabe30fe4c50150ccc152264561e4d80fbb6b2bca"}
Oct 30 21:14:39.951040 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.950953    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7k46d"
Oct 30 21:14:39.951040 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.950980    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7k46d" event={"ID":"4242f534-4d88-40d1-a4e0-e45493a00d7a","Type":"ContainerDied","Data":"db4cf48a8463d20dab36b103979e3763f51e44ac944767c0c2ac7aa88b418874"}
Oct 30 21:14:39.963732 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.963708    5713 scope.go:122] "RemoveContainer" containerID="ee2b358442f429882ecde2493a4d04ffe513f309b2d2bdf43fd7429a3965a5e4"
Oct 30 21:14:39.964143 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:14:39.964121    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee2b358442f429882ecde2493a4d04ffe513f309b2d2bdf43fd7429a3965a5e4\": container with ID starting with ee2b358442f429882ecde2493a4d04ffe513f309b2d2bdf43fd7429a3965a5e4 not found: ID does not exist" containerID="ee2b358442f429882ecde2493a4d04ffe513f309b2d2bdf43fd7429a3965a5e4"
Oct 30 21:14:39.964245 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.964151    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee2b358442f429882ecde2493a4d04ffe513f309b2d2bdf43fd7429a3965a5e4"} err="failed to get container status \"ee2b358442f429882ecde2493a4d04ffe513f309b2d2bdf43fd7429a3965a5e4\": rpc error: code = NotFound desc = could not find container \"ee2b358442f429882ecde2493a4d04ffe513f309b2d2bdf43fd7429a3965a5e4\": container with ID starting with ee2b358442f429882ecde2493a4d04ffe513f309b2d2bdf43fd7429a3965a5e4 not found: ID does not exist"
Oct 30 21:14:39.964245 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.964176    5713 scope.go:122] "RemoveContainer" containerID="940268c02595e8de006da84aabe30fe4c50150ccc152264561e4d80fbb6b2bca"
Oct 30 21:14:39.979230 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.979205    5713 scope.go:122] "RemoveContainer" containerID="940268c02595e8de006da84aabe30fe4c50150ccc152264561e4d80fbb6b2bca"
Oct 30 21:14:39.979524 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:14:39.979502    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"940268c02595e8de006da84aabe30fe4c50150ccc152264561e4d80fbb6b2bca\": container with ID starting with 940268c02595e8de006da84aabe30fe4c50150ccc152264561e4d80fbb6b2bca not found: ID does not exist" containerID="940268c02595e8de006da84aabe30fe4c50150ccc152264561e4d80fbb6b2bca"
Oct 30 21:14:39.979591 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:39.979533    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"940268c02595e8de006da84aabe30fe4c50150ccc152264561e4d80fbb6b2bca"} err="failed to get container status \"940268c02595e8de006da84aabe30fe4c50150ccc152264561e4d80fbb6b2bca\": rpc error: code = NotFound desc = could not find container \"940268c02595e8de006da84aabe30fe4c50150ccc152264561e4d80fbb6b2bca\": container with ID starting with 940268c02595e8de006da84aabe30fe4c50150ccc152264561e4d80fbb6b2bca not found: ID does not exist"
Oct 30 21:14:40.116832 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:40.116781    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7k46d"]
Oct 30 21:14:40.158124 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:40.158097    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-7k46d"]
Oct 30 21:14:40.268520 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:40.268442    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6nvgb"]
Oct 30 21:14:40.278237 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:40.278197    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6nvgb"]
Oct 30 21:14:40.794960 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:40.794924    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-d061d0-644b5ff884-cj67l"]
Oct 30 21:14:40.910232 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:40.910186    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-d061d0-9888979ff-kclfx"]
Oct 30 21:14:40.913609 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:40.913577    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-677b45-955d68994-fdprd" podUID="0b15b706-c879-4ada-a2b0-852c21ff3c8b" containerName="payload-processing" containerID="cri-o://d93b196730fd53491955c5258bcb9ed5d5a893db6fc0ae4c993a08081b728dea" gracePeriod=25
Oct 30 21:14:41.065933 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:41.065856    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4242f534-4d88-40d1-a4e0-e45493a00d7a" path="/var/lib/kubelet/pods/4242f534-4d88-40d1-a4e0-e45493a00d7a/volumes"
Oct 30 21:14:41.066447 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:41.066430    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74a166cd-7a71-449c-9ec1-68950b90e678" path="/var/lib/kubelet/pods/74a166cd-7a71-449c-9ec1-68950b90e678/volumes"
Oct 30 21:14:41.235869 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:41.235590    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-677b45-955d68994-fdprd"
Oct 30 21:14:41.347851 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:41.347700    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b15b706-c879-4ada-a2b0-852c21ff3c8b-kube-api-access-kxzjw\" (UniqueName: \"kubernetes.io/projected/0b15b706-c879-4ada-a2b0-852c21ff3c8b-kube-api-access-kxzjw\") pod \"0b15b706-c879-4ada-a2b0-852c21ff3c8b\" (UID: \"0b15b706-c879-4ada-a2b0-852c21ff3c8b\") "
Oct 30 21:14:41.348113 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:41.348090    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b15b706-c879-4ada-a2b0-852c21ff3c8b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b15b706-c879-4ada-a2b0-852c21ff3c8b-plugins-config-volume\") pod \"0b15b706-c879-4ada-a2b0-852c21ff3c8b\" (UID: \"0b15b706-c879-4ada-a2b0-852c21ff3c8b\") "
Oct 30 21:14:41.348490 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:41.348466    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b15b706-c879-4ada-a2b0-852c21ff3c8b-plugins-config-volume" pod "0b15b706-c879-4ada-a2b0-852c21ff3c8b" (UID: "0b15b706-c879-4ada-a2b0-852c21ff3c8b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:14:41.349160 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:41.349136    5713 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b15b706-c879-4ada-a2b0-852c21ff3c8b-plugins-config-volume\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:14:41.350603 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:41.350581    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b15b706-c879-4ada-a2b0-852c21ff3c8b-kube-api-access-kxzjw" pod "0b15b706-c879-4ada-a2b0-852c21ff3c8b" (UID: "0b15b706-c879-4ada-a2b0-852c21ff3c8b"). InnerVolumeSpecName "kube-api-access-kxzjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:14:41.443877 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:41.443835    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86f3e806-5dvf6"]
Oct 30 21:14:41.444567 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:41.444550    5713 cpu_manager.go:414] "containerMap: removing container" podUID="0b15b706-c879-4ada-a2b0-852c21ff3c8b" containerName="payload-processing"
Oct 30 21:14:41.444638 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:41.444569    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b15b706-c879-4ada-a2b0-852c21ff3c8b" containerName="payload-processing"
Oct 30 21:14:41.444638 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:41.444589    5713 cpu_manager.go:414] "containerMap: removing container" podUID="4242f534-4d88-40d1-a4e0-e45493a00d7a" containerName="istio-proxy"
Oct 30 21:14:41.444638 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:41.444594    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4242f534-4d88-40d1-a4e0-e45493a00d7a" containerName="istio-proxy"
Oct 30 21:14:41.444638 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:41.444605    5713 cpu_manager.go:414] "containerMap: removing container" podUID="74a166cd-7a71-449c-9ec1-68950b90e678" containerName="istio-proxy"
Oct 30 21:14:41.444638 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:41.444610    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74a166cd-7a71-449c-9ec1-68950b90e678" containerName="istio-proxy"
Oct 30 21:14:41.444638 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:41.444628    5713 cpu_manager.go:414] "containerMap: removing container" podUID="ecf576f4-8582-4ddc-8768-8c04888e5e4b" containerName="revoke-keys"
Oct 30 21:14:41.444638 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:41.444633    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecf576f4-8582-4ddc-8768-8c04888e5e4b" containerName="revoke-keys"
Oct 30 21:14:41.444869 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:41.444737    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b15b706-c879-4ada-a2b0-852c21ff3c8b" containerName="payload-processing"
Oct 30 21:14:41.444869 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:41.444746    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="4242f534-4d88-40d1-a4e0-e45493a00d7a" containerName="istio-proxy"
Oct 30 21:14:41.444869 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:41.444753    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="74a166cd-7a71-449c-9ec1-68950b90e678" containerName="istio-proxy"
Oct 30 21:14:41.444869 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:41.444763    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecf576f4-8582-4ddc-8768-8c04888e5e4b" containerName="revoke-keys"
Oct 30 21:14:41.448572 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:41.448550    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86f3e806-5dvf6"
Oct 30 21:14:41.450805 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:41.450759    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxzjw\" (UniqueName: \"kubernetes.io/projected/0b15b706-c879-4ada-a2b0-852c21ff3c8b-kube-api-access-kxzjw\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:14:41.505197 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:41.502841    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:14:41.505197 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:41.502890    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:14:41.505197 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:41.502995    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:14:41.505197 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:41.502864    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:14:41.505197 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:41.505095    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86f3e806-5dvf6"]
Oct 30 21:14:41.516762 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:41.516727    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8d71771-stmqp"]
Oct 30 21:14:41.521214 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:41.521190    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d71771-stmqp"
Oct 30 21:14:41.551859 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:41.551820    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b26e0d7b-ba75-4ed8-a45d-e51ee72fc0b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8d71771-stmqp\" (UID: \"b26e0d7b-ba75-4ed8-a45d-e51ee72fc0b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d71771-stmqp"
Oct 30 21:14:41.552024 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:41.551899    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92f99a0f-9d0e-43c3-ac42-eae9dc96c81a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86f3e806-5dvf6\" (UID: \"92f99a0f-9d0e-43c3-ac42-eae9dc96c81a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86f3e806-5dvf6"
Oct 30 21:14:41.552024 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:41.552014    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znkjd\" (UniqueName: \"kubernetes.io/projected/92f99a0f-9d0e-43c3-ac42-eae9dc96c81a-kube-api-access-znkjd\") pod \"maas-api-revoke-sub-86f3e806-5dvf6\" (UID: \"92f99a0f-9d0e-43c3-ac42-eae9dc96c81a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86f3e806-5dvf6"
Oct 30 21:14:41.552146 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:41.552091    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92f99a0f-9d0e-43c3-ac42-eae9dc96c81a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86f3e806-5dvf6\" (UID: \"92f99a0f-9d0e-43c3-ac42-eae9dc96c81a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86f3e806-5dvf6"
Oct 30 21:14:41.552213 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:41.552201    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8sk5\" (UniqueName: \"kubernetes.io/projected/b26e0d7b-ba75-4ed8-a45d-e51ee72fc0b9-kube-api-access-v8sk5\") pod \"maas-api-revoke-sub-d8d71771-stmqp\" (UID: \"b26e0d7b-ba75-4ed8-a45d-e51ee72fc0b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d71771-stmqp"
Oct 30 21:14:41.552263 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:41.552235    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b26e0d7b-ba75-4ed8-a45d-e51ee72fc0b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8d71771-stmqp\" (UID: \"b26e0d7b-ba75-4ed8-a45d-e51ee72fc0b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d71771-stmqp"
Oct 30 21:14:41.578030 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:41.577996    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8d71771-stmqp"]
Oct 30 21:14:41.653435 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:41.653325    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b26e0d7b-ba75-4ed8-a45d-e51ee72fc0b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8d71771-stmqp\" (UID: \"b26e0d7b-ba75-4ed8-a45d-e51ee72fc0b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d71771-stmqp"
Oct 30 21:14:41.653610 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:41.653534    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b26e0d7b-ba75-4ed8-a45d-e51ee72fc0b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8d71771-stmqp\" (UID: \"b26e0d7b-ba75-4ed8-a45d-e51ee72fc0b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d71771-stmqp"
Oct 30 21:14:41.653610 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:41.653593    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92f99a0f-9d0e-43c3-ac42-eae9dc96c81a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86f3e806-5dvf6\" (UID: \"92f99a0f-9d0e-43c3-ac42-eae9dc96c81a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86f3e806-5dvf6"
Oct 30 21:14:41.653741 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:41.653704    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znkjd\" (UniqueName: \"kubernetes.io/projected/92f99a0f-9d0e-43c3-ac42-eae9dc96c81a-kube-api-access-znkjd\") pod \"maas-api-revoke-sub-86f3e806-5dvf6\" (UID: \"92f99a0f-9d0e-43c3-ac42-eae9dc96c81a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86f3e806-5dvf6"
Oct 30 21:14:41.653969 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:41.653926    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92f99a0f-9d0e-43c3-ac42-eae9dc96c81a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86f3e806-5dvf6\" (UID: \"92f99a0f-9d0e-43c3-ac42-eae9dc96c81a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86f3e806-5dvf6"
Oct 30 21:14:41.654133 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:41.654046    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8sk5\" (UniqueName: \"kubernetes.io/projected/b26e0d7b-ba75-4ed8-a45d-e51ee72fc0b9-kube-api-access-v8sk5\") pod \"maas-api-revoke-sub-d8d71771-stmqp\" (UID: \"b26e0d7b-ba75-4ed8-a45d-e51ee72fc0b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d71771-stmqp"
Oct 30 21:14:41.654195 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:41.654137    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b26e0d7b-ba75-4ed8-a45d-e51ee72fc0b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8d71771-stmqp\" (UID: \"b26e0d7b-ba75-4ed8-a45d-e51ee72fc0b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d71771-stmqp"
Oct 30 21:14:41.654398 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:41.654330    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92f99a0f-9d0e-43c3-ac42-eae9dc96c81a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86f3e806-5dvf6\" (UID: \"92f99a0f-9d0e-43c3-ac42-eae9dc96c81a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86f3e806-5dvf6"
Oct 30 21:14:41.656304 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:41.656278    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b26e0d7b-ba75-4ed8-a45d-e51ee72fc0b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8d71771-stmqp\" (UID: \"b26e0d7b-ba75-4ed8-a45d-e51ee72fc0b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d71771-stmqp"
Oct 30 21:14:41.656419 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:41.656336    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92f99a0f-9d0e-43c3-ac42-eae9dc96c81a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86f3e806-5dvf6\" (UID: \"92f99a0f-9d0e-43c3-ac42-eae9dc96c81a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86f3e806-5dvf6"
Oct 30 21:14:41.712882 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:41.712848    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znkjd\" (UniqueName: \"kubernetes.io/projected/92f99a0f-9d0e-43c3-ac42-eae9dc96c81a-kube-api-access-znkjd\") pod \"maas-api-revoke-sub-86f3e806-5dvf6\" (UID: \"92f99a0f-9d0e-43c3-ac42-eae9dc96c81a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86f3e806-5dvf6"
Oct 30 21:14:41.713046 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:41.712889    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8sk5\" (UniqueName: \"kubernetes.io/projected/b26e0d7b-ba75-4ed8-a45d-e51ee72fc0b9-kube-api-access-v8sk5\") pod \"maas-api-revoke-sub-d8d71771-stmqp\" (UID: \"b26e0d7b-ba75-4ed8-a45d-e51ee72fc0b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d71771-stmqp"
Oct 30 21:14:41.763316 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:41.763271    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86f3e806-5dvf6"
Oct 30 21:14:41.837149 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:41.836835    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d71771-stmqp"
Oct 30 21:14:41.974706 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:41.974667    5713 generic.go:356] "Generic (PLEG): container finished" podID="0b15b706-c879-4ada-a2b0-852c21ff3c8b" containerID="d93b196730fd53491955c5258bcb9ed5d5a893db6fc0ae4c993a08081b728dea" exitCode=0
Oct 30 21:14:41.974918 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:41.974725    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-677b45-955d68994-fdprd"
Oct 30 21:14:41.974918 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:41.974740    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-677b45-955d68994-fdprd" event={"ID":"0b15b706-c879-4ada-a2b0-852c21ff3c8b","Type":"ContainerDied","Data":"d93b196730fd53491955c5258bcb9ed5d5a893db6fc0ae4c993a08081b728dea"}
Oct 30 21:14:41.974918 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:41.974843    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-677b45-955d68994-fdprd" event={"ID":"0b15b706-c879-4ada-a2b0-852c21ff3c8b","Type":"ContainerDied","Data":"71d7130fae6d7c221b25ba0169da2e9db6153f6cbecd5c961304d681016e1bf4"}
Oct 30 21:14:41.974918 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:41.974879    5713 scope.go:122] "RemoveContainer" containerID="d93b196730fd53491955c5258bcb9ed5d5a893db6fc0ae4c993a08081b728dea"
Oct 30 21:14:41.990164 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:41.990140    5713 scope.go:122] "RemoveContainer" containerID="d93b196730fd53491955c5258bcb9ed5d5a893db6fc0ae4c993a08081b728dea"
Oct 30 21:14:41.990541 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:14:41.990520    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d93b196730fd53491955c5258bcb9ed5d5a893db6fc0ae4c993a08081b728dea\": container with ID starting with d93b196730fd53491955c5258bcb9ed5d5a893db6fc0ae4c993a08081b728dea not found: ID does not exist" containerID="d93b196730fd53491955c5258bcb9ed5d5a893db6fc0ae4c993a08081b728dea"
Oct 30 21:14:41.990610 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:41.990550    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d93b196730fd53491955c5258bcb9ed5d5a893db6fc0ae4c993a08081b728dea"} err="failed to get container status \"d93b196730fd53491955c5258bcb9ed5d5a893db6fc0ae4c993a08081b728dea\": rpc error: code = NotFound desc = could not find container \"d93b196730fd53491955c5258bcb9ed5d5a893db6fc0ae4c993a08081b728dea\": container with ID starting with d93b196730fd53491955c5258bcb9ed5d5a893db6fc0ae4c993a08081b728dea not found: ID does not exist"
Oct 30 21:14:42.143250 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:42.143212    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-677b45-955d68994-fdprd"]
Oct 30 21:14:42.148938 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:42.148894    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-677b45-955d68994-fdprd"]
Oct 30 21:14:42.154451 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:42.154421    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86f3e806-5dvf6"]
Oct 30 21:14:42.193281 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:14:42.193247    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92f99a0f_9d0e_43c3_ac42_eae9dc96c81a.slice/crio-666d66e209310c506945ede7c429c937234909035daf1da738254518ff9a8a4e WatchSource:0}: Error finding container 666d66e209310c506945ede7c429c937234909035daf1da738254518ff9a8a4e: Status 404 returned error can't find the container with id 666d66e209310c506945ede7c429c937234909035daf1da738254518ff9a8a4e
Oct 30 21:14:42.256582 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:42.256540    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8d71771-stmqp"]
Oct 30 21:14:42.257423 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:14:42.257396    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb26e0d7b_ba75_4ed8_a45d_e51ee72fc0b9.slice/crio-c209f6a0ab94f7c823704ddb11a85045e01bb41a4ff51163ba4e16d170a93d33 WatchSource:0}: Error finding container c209f6a0ab94f7c823704ddb11a85045e01bb41a4ff51163ba4e16d170a93d33: Status 404 returned error can't find the container with id c209f6a0ab94f7c823704ddb11a85045e01bb41a4ff51163ba4e16d170a93d33
Oct 30 21:14:42.985686 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:42.985643    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d71771-stmqp" event={"ID":"b26e0d7b-ba75-4ed8-a45d-e51ee72fc0b9","Type":"ContainerStarted","Data":"835ee41964e483089fa87889a40cf316e41c65f2e500d5547745fb09c7e32334"}
Oct 30 21:14:42.985922 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:42.985743    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d71771-stmqp" event={"ID":"b26e0d7b-ba75-4ed8-a45d-e51ee72fc0b9","Type":"ContainerStarted","Data":"c209f6a0ab94f7c823704ddb11a85045e01bb41a4ff51163ba4e16d170a93d33"}
Oct 30 21:14:42.988688 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:42.988651    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86f3e806-5dvf6" event={"ID":"92f99a0f-9d0e-43c3-ac42-eae9dc96c81a","Type":"ContainerStarted","Data":"a783d0caf544c41f0c963fd8f2bc1c6a193c1c627ede560cbd11e44900d02165"}
Oct 30 21:14:42.988844 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:42.988701    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86f3e806-5dvf6" event={"ID":"92f99a0f-9d0e-43c3-ac42-eae9dc96c81a","Type":"ContainerStarted","Data":"666d66e209310c506945ede7c429c937234909035daf1da738254518ff9a8a4e"}
Oct 30 21:14:43.065820 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:43.065760    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b15b706-c879-4ada-a2b0-852c21ff3c8b" path="/var/lib/kubelet/pods/0b15b706-c879-4ada-a2b0-852c21ff3c8b/volumes"
Oct 30 21:14:43.221153 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:43.221102    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86f3e806-5dvf6" podStartSLOduration=2.221088159 podStartE2EDuration="2.221088159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:14:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:14:43.22044957 +0000 UTC m=+2166.285225967" watchObservedRunningTime="2026-10-30 21:14:43.221088159 +0000 UTC m=+2166.285864593"
Oct 30 21:14:43.222290 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:43.222253    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d71771-stmqp" podStartSLOduration=2.222242795 podStartE2EDuration="2.222242795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:14:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:14:43.104846608 +0000 UTC m=+2166.169623007" watchObservedRunningTime="2026-10-30 21:14:43.222242795 +0000 UTC m=+2166.287019192"
Oct 30 21:14:43.747090 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:43.747055    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd"]
Oct 30 21:14:43.747302 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:43.747278    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd" podUID="49e80bbd-dd2a-42f8-bdaf-e83abde1e625" containerName="istio-proxy" containerID="cri-o://927cfc6c9e6ca68d76ac93f90f1ee49f2d654dd2f1445aa215b30ca43c515c7a" gracePeriod=30
Oct 30 21:14:43.999923 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:43.999783    5713 generic.go:356] "Generic (PLEG): container finished" podID="b26e0d7b-ba75-4ed8-a45d-e51ee72fc0b9" containerID="835ee41964e483089fa87889a40cf316e41c65f2e500d5547745fb09c7e32334" exitCode=0
Oct 30 21:14:43.999923 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:43.999880    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d71771-stmqp" event={"ID":"b26e0d7b-ba75-4ed8-a45d-e51ee72fc0b9","Type":"ContainerDied","Data":"835ee41964e483089fa87889a40cf316e41c65f2e500d5547745fb09c7e32334"}
Oct 30 21:14:44.001999 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:44.001972    5713 generic.go:356] "Generic (PLEG): container finished" podID="92f99a0f-9d0e-43c3-ac42-eae9dc96c81a" containerID="a783d0caf544c41f0c963fd8f2bc1c6a193c1c627ede560cbd11e44900d02165" exitCode=0
Oct 30 21:14:44.002122 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:44.002059    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86f3e806-5dvf6" event={"ID":"92f99a0f-9d0e-43c3-ac42-eae9dc96c81a","Type":"ContainerDied","Data":"a783d0caf544c41f0c963fd8f2bc1c6a193c1c627ede560cbd11e44900d02165"}
Oct 30 21:14:45.190080 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:45.190051    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d71771-stmqp"
Oct 30 21:14:45.264588 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:45.264517    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86f3e806-5dvf6"
Oct 30 21:14:45.329467 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:45.329431    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92f99a0f-9d0e-43c3-ac42-eae9dc96c81a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92f99a0f-9d0e-43c3-ac42-eae9dc96c81a-maas-api-openshift-ca\") pod \"92f99a0f-9d0e-43c3-ac42-eae9dc96c81a\" (UID: \"92f99a0f-9d0e-43c3-ac42-eae9dc96c81a\") "
Oct 30 21:14:45.329634 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:45.329526    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92f99a0f-9d0e-43c3-ac42-eae9dc96c81a-kube-api-access-znkjd\" (UniqueName: \"kubernetes.io/projected/92f99a0f-9d0e-43c3-ac42-eae9dc96c81a-kube-api-access-znkjd\") pod \"92f99a0f-9d0e-43c3-ac42-eae9dc96c81a\" (UID: \"92f99a0f-9d0e-43c3-ac42-eae9dc96c81a\") "
Oct 30 21:14:45.329731 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:45.329712    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b26e0d7b-ba75-4ed8-a45d-e51ee72fc0b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b26e0d7b-ba75-4ed8-a45d-e51ee72fc0b9-maas-api-openshift-ca\") pod \"b26e0d7b-ba75-4ed8-a45d-e51ee72fc0b9\" (UID: \"b26e0d7b-ba75-4ed8-a45d-e51ee72fc0b9\") "
Oct 30 21:14:45.329882 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:45.329861    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92f99a0f-9d0e-43c3-ac42-eae9dc96c81a-maas-api-openshift-ca" pod "92f99a0f-9d0e-43c3-ac42-eae9dc96c81a" (UID: "92f99a0f-9d0e-43c3-ac42-eae9dc96c81a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:14:45.329882 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:45.329874    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92f99a0f-9d0e-43c3-ac42-eae9dc96c81a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92f99a0f-9d0e-43c3-ac42-eae9dc96c81a-maas-api-xks-ca\") pod \"92f99a0f-9d0e-43c3-ac42-eae9dc96c81a\" (UID: \"92f99a0f-9d0e-43c3-ac42-eae9dc96c81a\") "
Oct 30 21:14:45.330007 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:45.329921    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b26e0d7b-ba75-4ed8-a45d-e51ee72fc0b9-kube-api-access-v8sk5\" (UniqueName: \"kubernetes.io/projected/b26e0d7b-ba75-4ed8-a45d-e51ee72fc0b9-kube-api-access-v8sk5\") pod \"b26e0d7b-ba75-4ed8-a45d-e51ee72fc0b9\" (UID: \"b26e0d7b-ba75-4ed8-a45d-e51ee72fc0b9\") "
Oct 30 21:14:45.330007 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:45.329972    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b26e0d7b-ba75-4ed8-a45d-e51ee72fc0b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b26e0d7b-ba75-4ed8-a45d-e51ee72fc0b9-maas-api-xks-ca\") pod \"b26e0d7b-ba75-4ed8-a45d-e51ee72fc0b9\" (UID: \"b26e0d7b-ba75-4ed8-a45d-e51ee72fc0b9\") "
Oct 30 21:14:45.330206 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:45.330177    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b26e0d7b-ba75-4ed8-a45d-e51ee72fc0b9-maas-api-openshift-ca" pod "b26e0d7b-ba75-4ed8-a45d-e51ee72fc0b9" (UID: "b26e0d7b-ba75-4ed8-a45d-e51ee72fc0b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:14:45.330654 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:45.330630    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b26e0d7b-ba75-4ed8-a45d-e51ee72fc0b9-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:14:45.330766 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:45.330657    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92f99a0f-9d0e-43c3-ac42-eae9dc96c81a-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:14:45.332756 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:45.332450    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92f99a0f-9d0e-43c3-ac42-eae9dc96c81a-kube-api-access-znkjd" pod "92f99a0f-9d0e-43c3-ac42-eae9dc96c81a" (UID: "92f99a0f-9d0e-43c3-ac42-eae9dc96c81a"). InnerVolumeSpecName "kube-api-access-znkjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:14:45.332756 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:45.332684    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92f99a0f-9d0e-43c3-ac42-eae9dc96c81a-maas-api-xks-ca" pod "92f99a0f-9d0e-43c3-ac42-eae9dc96c81a" (UID: "92f99a0f-9d0e-43c3-ac42-eae9dc96c81a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:14:45.333020 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:45.332992    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b26e0d7b-ba75-4ed8-a45d-e51ee72fc0b9-maas-api-xks-ca" pod "b26e0d7b-ba75-4ed8-a45d-e51ee72fc0b9" (UID: "b26e0d7b-ba75-4ed8-a45d-e51ee72fc0b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:14:45.333111 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:45.333076    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b26e0d7b-ba75-4ed8-a45d-e51ee72fc0b9-kube-api-access-v8sk5" pod "b26e0d7b-ba75-4ed8-a45d-e51ee72fc0b9" (UID: "b26e0d7b-ba75-4ed8-a45d-e51ee72fc0b9"). InnerVolumeSpecName "kube-api-access-v8sk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:14:45.432020 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:45.431981    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92f99a0f-9d0e-43c3-ac42-eae9dc96c81a-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:14:45.432020 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:45.432019    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8sk5\" (UniqueName: \"kubernetes.io/projected/b26e0d7b-ba75-4ed8-a45d-e51ee72fc0b9-kube-api-access-v8sk5\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:14:45.432304 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:45.432037    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b26e0d7b-ba75-4ed8-a45d-e51ee72fc0b9-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:14:45.432304 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:45.432053    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znkjd\" (UniqueName: \"kubernetes.io/projected/92f99a0f-9d0e-43c3-ac42-eae9dc96c81a-kube-api-access-znkjd\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:14:45.795333 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:45.795272    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-d061d0-644b5ff884-cj67l" podUID="a08925fb-aee2-425a-84da-c407fb8cfb7f" containerName="payload-processing" containerID="cri-o://c815c95c65e21b95193016db19fc0cfce93d0c3cde714312f9408b5de525fb0f" gracePeriod=25
Oct 30 21:14:45.910853 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:45.910812    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d061d0-9888979ff-kclfx" podUID="b47eb722-c69a-4565-b0ae-1b46f99090df" containerName="payload-pre-processing" containerID="cri-o://11d2d335bebea202947eed0e482f1b44ea4f97286b2a3f48fbb9d4c0aea3ac3c" gracePeriod=25
Oct 30 21:14:46.021316 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:46.021287    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86f3e806-5dvf6"
Oct 30 21:14:46.021510 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:46.021314    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86f3e806-5dvf6" event={"ID":"92f99a0f-9d0e-43c3-ac42-eae9dc96c81a","Type":"ContainerDied","Data":"666d66e209310c506945ede7c429c937234909035daf1da738254518ff9a8a4e"}
Oct 30 21:14:46.021510 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:46.021406    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="666d66e209310c506945ede7c429c937234909035daf1da738254518ff9a8a4e"
Oct 30 21:14:46.023153 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:46.023125    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d71771-stmqp" event={"ID":"b26e0d7b-ba75-4ed8-a45d-e51ee72fc0b9","Type":"ContainerDied","Data":"c209f6a0ab94f7c823704ddb11a85045e01bb41a4ff51163ba4e16d170a93d33"}
Oct 30 21:14:46.023271 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:46.023168    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c209f6a0ab94f7c823704ddb11a85045e01bb41a4ff51163ba4e16d170a93d33"
Oct 30 21:14:46.023344 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:46.023327    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d71771-stmqp"
Oct 30 21:14:49.038734 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:49.038707    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd"
Oct 30 21:14:49.055476 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:49.055440    5713 generic.go:356] "Generic (PLEG): container finished" podID="49e80bbd-dd2a-42f8-bdaf-e83abde1e625" containerID="927cfc6c9e6ca68d76ac93f90f1ee49f2d654dd2f1445aa215b30ca43c515c7a" exitCode=0
Oct 30 21:14:49.055889 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:49.055857    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd"
Oct 30 21:14:49.065916 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:49.065893    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd" event={"ID":"49e80bbd-dd2a-42f8-bdaf-e83abde1e625","Type":"ContainerDied","Data":"927cfc6c9e6ca68d76ac93f90f1ee49f2d654dd2f1445aa215b30ca43c515c7a"}
Oct 30 21:14:49.066059 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:49.065938    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd" event={"ID":"49e80bbd-dd2a-42f8-bdaf-e83abde1e625","Type":"ContainerDied","Data":"83b34a4d32906d12bc4cd69129fc8fcf0ba8c69658d1035e9c7e57694b554b26"}
Oct 30 21:14:49.066059 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:49.065963    5713 scope.go:122] "RemoveContainer" containerID="927cfc6c9e6ca68d76ac93f90f1ee49f2d654dd2f1445aa215b30ca43c515c7a"
Oct 30 21:14:49.082025 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:49.082003    5713 scope.go:122] "RemoveContainer" containerID="927cfc6c9e6ca68d76ac93f90f1ee49f2d654dd2f1445aa215b30ca43c515c7a"
Oct 30 21:14:49.083747 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:14:49.083718    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"927cfc6c9e6ca68d76ac93f90f1ee49f2d654dd2f1445aa215b30ca43c515c7a\": container with ID starting with 927cfc6c9e6ca68d76ac93f90f1ee49f2d654dd2f1445aa215b30ca43c515c7a not found: ID does not exist" containerID="927cfc6c9e6ca68d76ac93f90f1ee49f2d654dd2f1445aa215b30ca43c515c7a"
Oct 30 21:14:49.083904 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:49.083755    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"927cfc6c9e6ca68d76ac93f90f1ee49f2d654dd2f1445aa215b30ca43c515c7a"} err="failed to get container status \"927cfc6c9e6ca68d76ac93f90f1ee49f2d654dd2f1445aa215b30ca43c515c7a\": rpc error: code = NotFound desc = could not find container \"927cfc6c9e6ca68d76ac93f90f1ee49f2d654dd2f1445aa215b30ca43c515c7a\": container with ID starting with 927cfc6c9e6ca68d76ac93f90f1ee49f2d654dd2f1445aa215b30ca43c515c7a not found: ID does not exist"
Oct 30 21:14:49.196050 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:49.196009    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49e80bbd-dd2a-42f8-bdaf-e83abde1e625-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49e80bbd-dd2a-42f8-bdaf-e83abde1e625-workload-socket\") pod \"49e80bbd-dd2a-42f8-bdaf-e83abde1e625\" (UID: \"49e80bbd-dd2a-42f8-bdaf-e83abde1e625\") "
Oct 30 21:14:49.196258 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:49.196070    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/49e80bbd-dd2a-42f8-bdaf-e83abde1e625-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49e80bbd-dd2a-42f8-bdaf-e83abde1e625-istio-podinfo\") pod \"49e80bbd-dd2a-42f8-bdaf-e83abde1e625\" (UID: \"49e80bbd-dd2a-42f8-bdaf-e83abde1e625\") "
Oct 30 21:14:49.196258 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:49.196121    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49e80bbd-dd2a-42f8-bdaf-e83abde1e625-kube-api-access-tbg7k\" (UniqueName: \"kubernetes.io/projected/49e80bbd-dd2a-42f8-bdaf-e83abde1e625-kube-api-access-tbg7k\") pod \"49e80bbd-dd2a-42f8-bdaf-e83abde1e625\" (UID: \"49e80bbd-dd2a-42f8-bdaf-e83abde1e625\") "
Oct 30 21:14:49.196258 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:49.196202    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49e80bbd-dd2a-42f8-bdaf-e83abde1e625-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49e80bbd-dd2a-42f8-bdaf-e83abde1e625-istio-envoy\") pod \"49e80bbd-dd2a-42f8-bdaf-e83abde1e625\" (UID: \"49e80bbd-dd2a-42f8-bdaf-e83abde1e625\") "
Oct 30 21:14:49.196258 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:49.196230    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49e80bbd-dd2a-42f8-bdaf-e83abde1e625-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49e80bbd-dd2a-42f8-bdaf-e83abde1e625-workload-certs\") pod \"49e80bbd-dd2a-42f8-bdaf-e83abde1e625\" (UID: \"49e80bbd-dd2a-42f8-bdaf-e83abde1e625\") "
Oct 30 21:14:49.196493 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:49.196261    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49e80bbd-dd2a-42f8-bdaf-e83abde1e625-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49e80bbd-dd2a-42f8-bdaf-e83abde1e625-credential-socket\") pod \"49e80bbd-dd2a-42f8-bdaf-e83abde1e625\" (UID: \"49e80bbd-dd2a-42f8-bdaf-e83abde1e625\") "
Oct 30 21:14:49.196493 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:49.196261    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49e80bbd-dd2a-42f8-bdaf-e83abde1e625-workload-socket" pod "49e80bbd-dd2a-42f8-bdaf-e83abde1e625" (UID: "49e80bbd-dd2a-42f8-bdaf-e83abde1e625"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:14:49.196493 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:49.196323    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49e80bbd-dd2a-42f8-bdaf-e83abde1e625-istio-token\" (UniqueName: \"kubernetes.io/projected/49e80bbd-dd2a-42f8-bdaf-e83abde1e625-istio-token\") pod \"49e80bbd-dd2a-42f8-bdaf-e83abde1e625\" (UID: \"49e80bbd-dd2a-42f8-bdaf-e83abde1e625\") "
Oct 30 21:14:49.196660 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:49.196504    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49e80bbd-dd2a-42f8-bdaf-e83abde1e625-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49e80bbd-dd2a-42f8-bdaf-e83abde1e625-istio-data\") pod \"49e80bbd-dd2a-42f8-bdaf-e83abde1e625\" (UID: \"49e80bbd-dd2a-42f8-bdaf-e83abde1e625\") "
Oct 30 21:14:49.196660 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:49.196544    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49e80bbd-dd2a-42f8-bdaf-e83abde1e625-credential-socket" pod "49e80bbd-dd2a-42f8-bdaf-e83abde1e625" (UID: "49e80bbd-dd2a-42f8-bdaf-e83abde1e625"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:14:49.196660 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:49.196638    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49e80bbd-dd2a-42f8-bdaf-e83abde1e625-workload-certs" pod "49e80bbd-dd2a-42f8-bdaf-e83abde1e625" (UID: "49e80bbd-dd2a-42f8-bdaf-e83abde1e625"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:14:49.196869 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:49.196830    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49e80bbd-dd2a-42f8-bdaf-e83abde1e625-istio-data" pod "49e80bbd-dd2a-42f8-bdaf-e83abde1e625" (UID: "49e80bbd-dd2a-42f8-bdaf-e83abde1e625"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:14:49.196976 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:49.196936    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49e80bbd-dd2a-42f8-bdaf-e83abde1e625-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49e80bbd-dd2a-42f8-bdaf-e83abde1e625-istiod-ca-cert\") pod \"49e80bbd-dd2a-42f8-bdaf-e83abde1e625\" (UID: \"49e80bbd-dd2a-42f8-bdaf-e83abde1e625\") "
Oct 30 21:14:49.197588 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:49.197533    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49e80bbd-dd2a-42f8-bdaf-e83abde1e625-istiod-ca-cert" pod "49e80bbd-dd2a-42f8-bdaf-e83abde1e625" (UID: "49e80bbd-dd2a-42f8-bdaf-e83abde1e625"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:14:49.198369 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:49.198337    5713 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49e80bbd-dd2a-42f8-bdaf-e83abde1e625-workload-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:14:49.198369 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:49.198369    5713 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49e80bbd-dd2a-42f8-bdaf-e83abde1e625-workload-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:14:49.198534 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:49.198385    5713 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49e80bbd-dd2a-42f8-bdaf-e83abde1e625-credential-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:14:49.198534 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:49.198396    5713 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49e80bbd-dd2a-42f8-bdaf-e83abde1e625-istio-data\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:14:49.198534 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:49.198408    5713 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49e80bbd-dd2a-42f8-bdaf-e83abde1e625-istiod-ca-cert\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:14:49.198931 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:49.198904    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49e80bbd-dd2a-42f8-bdaf-e83abde1e625-kube-api-access-tbg7k" pod "49e80bbd-dd2a-42f8-bdaf-e83abde1e625" (UID: "49e80bbd-dd2a-42f8-bdaf-e83abde1e625"). InnerVolumeSpecName "kube-api-access-tbg7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:14:49.199553 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:49.199528    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/49e80bbd-dd2a-42f8-bdaf-e83abde1e625-istio-podinfo" pod "49e80bbd-dd2a-42f8-bdaf-e83abde1e625" (UID: "49e80bbd-dd2a-42f8-bdaf-e83abde1e625"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:14:49.199643 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:49.199613    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49e80bbd-dd2a-42f8-bdaf-e83abde1e625-istio-token" pod "49e80bbd-dd2a-42f8-bdaf-e83abde1e625" (UID: "49e80bbd-dd2a-42f8-bdaf-e83abde1e625"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:14:49.199716 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:49.199701    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49e80bbd-dd2a-42f8-bdaf-e83abde1e625-istio-envoy" pod "49e80bbd-dd2a-42f8-bdaf-e83abde1e625" (UID: "49e80bbd-dd2a-42f8-bdaf-e83abde1e625"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:14:49.300019 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:49.299932    5713 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49e80bbd-dd2a-42f8-bdaf-e83abde1e625-istio-podinfo\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:14:49.300019 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:49.299962    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbg7k\" (UniqueName: \"kubernetes.io/projected/49e80bbd-dd2a-42f8-bdaf-e83abde1e625-kube-api-access-tbg7k\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:14:49.300019 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:49.299976    5713 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49e80bbd-dd2a-42f8-bdaf-e83abde1e625-istio-envoy\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:14:49.300019 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:49.299985    5713 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49e80bbd-dd2a-42f8-bdaf-e83abde1e625-istio-token\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:14:49.483112 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:49.483082    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd"]
Oct 30 21:14:49.502400 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:49.502361    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-2scfd"]
Oct 30 21:14:51.065252 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:51.065217    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49e80bbd-dd2a-42f8-bdaf-e83abde1e625" path="/var/lib/kubelet/pods/49e80bbd-dd2a-42f8-bdaf-e83abde1e625/volumes"
Oct 30 21:14:53.664895 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:53.664857    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-d061d0-644b5ff884-cj67l" podUID="a08925fb-aee2-425a-84da-c407fb8cfb7f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:14:53.664895 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:14:53.664878    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d061d0-9888979ff-kclfx" podUID="b47eb722-c69a-4565-b0ae-1b46f99090df" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:14:54.459918 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:14:54.459876    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:15:00.173144 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.173105    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889915-x55nz"]
Oct 30 21:15:00.173769 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.173754    5713 cpu_manager.go:414] "containerMap: removing container" podUID="92f99a0f-9d0e-43c3-ac42-eae9dc96c81a" containerName="revoke-keys"
Oct 30 21:15:00.173869 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.173770    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92f99a0f-9d0e-43c3-ac42-eae9dc96c81a" containerName="revoke-keys"
Oct 30 21:15:00.173869 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.173807    5713 cpu_manager.go:414] "containerMap: removing container" podUID="49e80bbd-dd2a-42f8-bdaf-e83abde1e625" containerName="istio-proxy"
Oct 30 21:15:00.173869 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.173813    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49e80bbd-dd2a-42f8-bdaf-e83abde1e625" containerName="istio-proxy"
Oct 30 21:15:00.173869 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.173841    5713 cpu_manager.go:414] "containerMap: removing container" podUID="b26e0d7b-ba75-4ed8-a45d-e51ee72fc0b9" containerName="revoke-keys"
Oct 30 21:15:00.173869 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.173847    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b26e0d7b-ba75-4ed8-a45d-e51ee72fc0b9" containerName="revoke-keys"
Oct 30 21:15:00.174039 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.173947    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="49e80bbd-dd2a-42f8-bdaf-e83abde1e625" containerName="istio-proxy"
Oct 30 21:15:00.174039 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.173962    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="b26e0d7b-ba75-4ed8-a45d-e51ee72fc0b9" containerName="revoke-keys"
Oct 30 21:15:00.174039 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.173972    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="92f99a0f-9d0e-43c3-ac42-eae9dc96c81a" containerName="revoke-keys"
Oct 30 21:15:00.177594 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.177572    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889915-x55nz"
Oct 30 21:15:00.189014 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.188987    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-390b71-29889915-lrfqc"]
Oct 30 21:15:00.193429 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.193408    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-390b71-29889915-lrfqc"
Oct 30 21:15:00.194464 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.194439    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88da25-29889915-gf4hb"]
Oct 30 21:15:00.199495 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.199469    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88da25-29889915-gf4hb"
Oct 30 21:15:00.223575 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.223540    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5c45e5-29889915-zm4fd"]
Oct 30 21:15:00.227952 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.227930    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f9f277-29889915-dbd9b"]
Oct 30 21:15:00.228052 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.228038    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5c45e5-29889915-zm4fd"
Oct 30 21:15:00.232319 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.232293    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f9f277-29889915-dbd9b"
Oct 30 21:15:00.238821 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.238039    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:15:00.238821 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.238080    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:15:00.238821 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.238171    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:15:00.266892 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.266853    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3543ab8c-8863-4207-ae05-d8094dbd8cb9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889915-x55nz\" (UID: \"3543ab8c-8863-4207-ae05-d8094dbd8cb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889915-x55nz"
Oct 30 21:15:00.266892 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.266892    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16adca55-5cc2-4452-aba7-3e891c005f98-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-88da25-29889915-gf4hb\" (UID: \"16adca55-5cc2-4452-aba7-3e891c005f98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88da25-29889915-gf4hb"
Oct 30 21:15:00.267171 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.266926    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmd6j\" (UniqueName: \"kubernetes.io/projected/3543ab8c-8863-4207-ae05-d8094dbd8cb9-kube-api-access-wmd6j\") pod \"maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889915-x55nz\" (UID: \"3543ab8c-8863-4207-ae05-d8094dbd8cb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889915-x55nz"
Oct 30 21:15:00.267171 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.267081    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa46459a-d46e-4af4-bdfc-d6872367429d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-390b71-29889915-lrfqc\" (UID: \"aa46459a-d46e-4af4-bdfc-d6872367429d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-390b71-29889915-lrfqc"
Oct 30 21:15:00.267283 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.267212    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkdgk\" (UniqueName: \"kubernetes.io/projected/16adca55-5cc2-4452-aba7-3e891c005f98-kube-api-access-bkdgk\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-88da25-29889915-gf4hb\" (UID: \"16adca55-5cc2-4452-aba7-3e891c005f98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88da25-29889915-gf4hb"
Oct 30 21:15:00.267283 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.267253    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hwcs\" (UniqueName: \"kubernetes.io/projected/aa46459a-d46e-4af4-bdfc-d6872367429d-kube-api-access-2hwcs\") pod \"maas-api-key-cleanup-e2e-inf-a-390b71-29889915-lrfqc\" (UID: \"aa46459a-d46e-4af4-bdfc-d6872367429d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-390b71-29889915-lrfqc"
Oct 30 21:15:00.277821 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.277766    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889915-27nkg"]
Oct 30 21:15:00.288415 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.288304    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-682999-29889915-vzbj5"]
Oct 30 21:15:00.288595 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.288491    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-27nkg"
Oct 30 21:15:00.293199 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.293163    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cb6404-29889915-4c7lb"]
Oct 30 21:15:00.293350 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.293297    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-682999-29889915-vzbj5"
Oct 30 21:15:00.299102 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.298912    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cb6404-29889915-4c7lb"
Oct 30 21:15:00.299556 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.299538    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889915-x55nz"]
Oct 30 21:15:00.303867 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.303843    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-390b71-29889915-lrfqc"]
Oct 30 21:15:00.308873 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.308849    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5c45e5-29889915-zm4fd"]
Oct 30 21:15:00.313660 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.313632    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f9eff-29889915-wpmls"]
Oct 30 21:15:00.321667 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.320490    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f9eff-29889915-wpmls"
Oct 30 21:15:00.345103 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.345070    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-677b45-29889915-4m4cj"]
Oct 30 21:15:00.354915 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.352737    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-677b45-29889915-4m4cj"
Oct 30 21:15:00.360039 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.359998    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-682999-29889915-vzbj5"]
Oct 30 21:15:00.368951 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.367107    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cb6404-29889915-4c7lb"]
Oct 30 21:15:00.369177 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.369103    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkdgk\" (UniqueName: \"kubernetes.io/projected/16adca55-5cc2-4452-aba7-3e891c005f98-kube-api-access-bkdgk\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-88da25-29889915-gf4hb\" (UID: \"16adca55-5cc2-4452-aba7-3e891c005f98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88da25-29889915-gf4hb"
Oct 30 21:15:00.369177 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.369157    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hwcs\" (UniqueName: \"kubernetes.io/projected/aa46459a-d46e-4af4-bdfc-d6872367429d-kube-api-access-2hwcs\") pod \"maas-api-key-cleanup-e2e-inf-a-390b71-29889915-lrfqc\" (UID: \"aa46459a-d46e-4af4-bdfc-d6872367429d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-390b71-29889915-lrfqc"
Oct 30 21:15:00.369323 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.369192    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dmlh\" (UniqueName: \"kubernetes.io/projected/9970e7c4-7661-47c5-a6df-4de2bea436e2-kube-api-access-8dmlh\") pod \"maas-api-key-cleanup-e2e-worker-w1-f9f277-29889915-dbd9b\" (UID: \"9970e7c4-7661-47c5-a6df-4de2bea436e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f9f277-29889915-dbd9b"
Oct 30 21:15:00.369414 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.369389    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsqft\" (UniqueName: \"kubernetes.io/projected/c3a522bc-746d-4cdf-940d-ccb562ade3c0-kube-api-access-nsqft\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-5c45e5-29889915-zm4fd\" (UID: \"c3a522bc-746d-4cdf-940d-ccb562ade3c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5c45e5-29889915-zm4fd"
Oct 30 21:15:00.369501 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.369481    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/126910b1-789a-4e00-9e84-2f0a9c899abb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-682999-29889915-vzbj5\" (UID: \"126910b1-789a-4e00-9e84-2f0a9c899abb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-682999-29889915-vzbj5"
Oct 30 21:15:00.369614 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.369597    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pm9b\" (UniqueName: \"kubernetes.io/projected/f59ff9d3-2927-4021-9103-fd2f60f7b655-kube-api-access-9pm9b\") pod \"maas-api-key-cleanup-e2e-inf-a-2f9eff-29889915-wpmls\" (UID: \"f59ff9d3-2927-4021-9103-fd2f60f7b655\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f9eff-29889915-wpmls"
Oct 30 21:15:00.369746 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.369726    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vsmv\" (UniqueName: \"kubernetes.io/projected/a8d5eebe-b80b-4b99-ace8-7e87724decaf-kube-api-access-2vsmv\") pod \"maas-api-key-cleanup-e2e-api-b-cb6404-29889915-4c7lb\" (UID: \"a8d5eebe-b80b-4b99-ace8-7e87724decaf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cb6404-29889915-4c7lb"
Oct 30 21:15:00.369873 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.369774    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2184c91-9d42-4a01-bf05-dd962ff62dcb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889915-27nkg\" (UID: \"c2184c91-9d42-4a01-bf05-dd962ff62dcb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-27nkg"
Oct 30 21:15:00.369873 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.369821    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8d5eebe-b80b-4b99-ace8-7e87724decaf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-cb6404-29889915-4c7lb\" (UID: \"a8d5eebe-b80b-4b99-ace8-7e87724decaf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cb6404-29889915-4c7lb"
Oct 30 21:15:00.369873 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.369848    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpzrc\" (UniqueName: \"kubernetes.io/projected/126910b1-789a-4e00-9e84-2f0a9c899abb-kube-api-access-wpzrc\") pod \"maas-api-key-cleanup-e2e-worker-w3-682999-29889915-vzbj5\" (UID: \"126910b1-789a-4e00-9e84-2f0a9c899abb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-682999-29889915-vzbj5"
Oct 30 21:15:00.370045 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.369912    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3543ab8c-8863-4207-ae05-d8094dbd8cb9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889915-x55nz\" (UID: \"3543ab8c-8863-4207-ae05-d8094dbd8cb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889915-x55nz"
Oct 30 21:15:00.370045 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.369943    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16adca55-5cc2-4452-aba7-3e891c005f98-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-88da25-29889915-gf4hb\" (UID: \"16adca55-5cc2-4452-aba7-3e891c005f98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88da25-29889915-gf4hb"
Oct 30 21:15:00.370045 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.369974    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmd6j\" (UniqueName: \"kubernetes.io/projected/3543ab8c-8863-4207-ae05-d8094dbd8cb9-kube-api-access-wmd6j\") pod \"maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889915-x55nz\" (UID: \"3543ab8c-8863-4207-ae05-d8094dbd8cb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889915-x55nz"
Oct 30 21:15:00.370216 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.370053    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa46459a-d46e-4af4-bdfc-d6872367429d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-390b71-29889915-lrfqc\" (UID: \"aa46459a-d46e-4af4-bdfc-d6872367429d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-390b71-29889915-lrfqc"
Oct 30 21:15:00.370216 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.370104    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3a522bc-746d-4cdf-940d-ccb562ade3c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-5c45e5-29889915-zm4fd\" (UID: \"c3a522bc-746d-4cdf-940d-ccb562ade3c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5c45e5-29889915-zm4fd"
Oct 30 21:15:00.370216 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.370143    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmmqr\" (UniqueName: \"kubernetes.io/projected/c2184c91-9d42-4a01-bf05-dd962ff62dcb-kube-api-access-tmmqr\") pod \"maas-api-key-cleanup-29889915-27nkg\" (UID: \"c2184c91-9d42-4a01-bf05-dd962ff62dcb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-27nkg"
Oct 30 21:15:00.370216 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.370167    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f59ff9d3-2927-4021-9103-fd2f60f7b655-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-2f9eff-29889915-wpmls\" (UID: \"f59ff9d3-2927-4021-9103-fd2f60f7b655\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f9eff-29889915-wpmls"
Oct 30 21:15:00.370435 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.370229    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9970e7c4-7661-47c5-a6df-4de2bea436e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-f9f277-29889915-dbd9b\" (UID: \"9970e7c4-7661-47c5-a6df-4de2bea436e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f9f277-29889915-dbd9b"
Oct 30 21:15:00.373854 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.371655    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16adca55-5cc2-4452-aba7-3e891c005f98-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-88da25-29889915-gf4hb\" (UID: \"16adca55-5cc2-4452-aba7-3e891c005f98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88da25-29889915-gf4hb"
Oct 30 21:15:00.373854 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.372589    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3543ab8c-8863-4207-ae05-d8094dbd8cb9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889915-x55nz\" (UID: \"3543ab8c-8863-4207-ae05-d8094dbd8cb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889915-x55nz"
Oct 30 21:15:00.373854 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.372620    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa46459a-d46e-4af4-bdfc-d6872367429d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-390b71-29889915-lrfqc\" (UID: \"aa46459a-d46e-4af4-bdfc-d6872367429d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-390b71-29889915-lrfqc"
Oct 30 21:15:00.373854 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.373117    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889915-27nkg"]
Oct 30 21:15:00.383344 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.383312    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88da25-29889915-gf4hb"]
Oct 30 21:15:00.389833 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.389802    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f9eff-29889915-wpmls"]
Oct 30 21:15:00.395063 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.395032    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f9f277-29889915-dbd9b"]
Oct 30 21:15:00.413837 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.413781    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-677b45-29889915-4m4cj"]
Oct 30 21:15:00.428945 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.428881    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkdgk\" (UniqueName: \"kubernetes.io/projected/16adca55-5cc2-4452-aba7-3e891c005f98-kube-api-access-bkdgk\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-88da25-29889915-gf4hb\" (UID: \"16adca55-5cc2-4452-aba7-3e891c005f98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88da25-29889915-gf4hb"
Oct 30 21:15:00.429586 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.429556    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hwcs\" (UniqueName: \"kubernetes.io/projected/aa46459a-d46e-4af4-bdfc-d6872367429d-kube-api-access-2hwcs\") pod \"maas-api-key-cleanup-e2e-inf-a-390b71-29889915-lrfqc\" (UID: \"aa46459a-d46e-4af4-bdfc-d6872367429d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-390b71-29889915-lrfqc"
Oct 30 21:15:00.430436 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.430414    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmd6j\" (UniqueName: \"kubernetes.io/projected/3543ab8c-8863-4207-ae05-d8094dbd8cb9-kube-api-access-wmd6j\") pod \"maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889915-x55nz\" (UID: \"3543ab8c-8863-4207-ae05-d8094dbd8cb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889915-x55nz"
Oct 30 21:15:00.472104 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.472060    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5117983-c7b5-4679-8c1d-90ea13ebca99-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-677b45-29889915-4m4cj\" (UID: \"b5117983-c7b5-4679-8c1d-90ea13ebca99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-677b45-29889915-4m4cj"
Oct 30 21:15:00.472104 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.472118    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vsmv\" (UniqueName: \"kubernetes.io/projected/a8d5eebe-b80b-4b99-ace8-7e87724decaf-kube-api-access-2vsmv\") pod \"maas-api-key-cleanup-e2e-api-b-cb6404-29889915-4c7lb\" (UID: \"a8d5eebe-b80b-4b99-ace8-7e87724decaf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cb6404-29889915-4c7lb"
Oct 30 21:15:00.472507 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.472194    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2184c91-9d42-4a01-bf05-dd962ff62dcb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889915-27nkg\" (UID: \"c2184c91-9d42-4a01-bf05-dd962ff62dcb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-27nkg"
Oct 30 21:15:00.472507 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.472235    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8d5eebe-b80b-4b99-ace8-7e87724decaf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-cb6404-29889915-4c7lb\" (UID: \"a8d5eebe-b80b-4b99-ace8-7e87724decaf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cb6404-29889915-4c7lb"
Oct 30 21:15:00.472507 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.472257    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpzrc\" (UniqueName: \"kubernetes.io/projected/126910b1-789a-4e00-9e84-2f0a9c899abb-kube-api-access-wpzrc\") pod \"maas-api-key-cleanup-e2e-worker-w3-682999-29889915-vzbj5\" (UID: \"126910b1-789a-4e00-9e84-2f0a9c899abb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-682999-29889915-vzbj5"
Oct 30 21:15:00.472507 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.472381    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3a522bc-746d-4cdf-940d-ccb562ade3c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-5c45e5-29889915-zm4fd\" (UID: \"c3a522bc-746d-4cdf-940d-ccb562ade3c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5c45e5-29889915-zm4fd"
Oct 30 21:15:00.472507 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.472418    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmmqr\" (UniqueName: \"kubernetes.io/projected/c2184c91-9d42-4a01-bf05-dd962ff62dcb-kube-api-access-tmmqr\") pod \"maas-api-key-cleanup-29889915-27nkg\" (UID: \"c2184c91-9d42-4a01-bf05-dd962ff62dcb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-27nkg"
Oct 30 21:15:00.472507 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.472445    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f59ff9d3-2927-4021-9103-fd2f60f7b655-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-2f9eff-29889915-wpmls\" (UID: \"f59ff9d3-2927-4021-9103-fd2f60f7b655\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f9eff-29889915-wpmls"
Oct 30 21:15:00.472507 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.472501    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9970e7c4-7661-47c5-a6df-4de2bea436e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-f9f277-29889915-dbd9b\" (UID: \"9970e7c4-7661-47c5-a6df-4de2bea436e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f9f277-29889915-dbd9b"
Oct 30 21:15:00.472913 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.472545    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dmlh\" (UniqueName: \"kubernetes.io/projected/9970e7c4-7661-47c5-a6df-4de2bea436e2-kube-api-access-8dmlh\") pod \"maas-api-key-cleanup-e2e-worker-w1-f9f277-29889915-dbd9b\" (UID: \"9970e7c4-7661-47c5-a6df-4de2bea436e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f9f277-29889915-dbd9b"
Oct 30 21:15:00.472913 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.472612    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsqft\" (UniqueName: \"kubernetes.io/projected/c3a522bc-746d-4cdf-940d-ccb562ade3c0-kube-api-access-nsqft\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-5c45e5-29889915-zm4fd\" (UID: \"c3a522bc-746d-4cdf-940d-ccb562ade3c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5c45e5-29889915-zm4fd"
Oct 30 21:15:00.472913 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.472653    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/126910b1-789a-4e00-9e84-2f0a9c899abb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-682999-29889915-vzbj5\" (UID: \"126910b1-789a-4e00-9e84-2f0a9c899abb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-682999-29889915-vzbj5"
Oct 30 21:15:00.472913 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.472767    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pm9b\" (UniqueName: \"kubernetes.io/projected/f59ff9d3-2927-4021-9103-fd2f60f7b655-kube-api-access-9pm9b\") pod \"maas-api-key-cleanup-e2e-inf-a-2f9eff-29889915-wpmls\" (UID: \"f59ff9d3-2927-4021-9103-fd2f60f7b655\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f9eff-29889915-wpmls"
Oct 30 21:15:00.472913 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.472822    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6whz8\" (UniqueName: \"kubernetes.io/projected/b5117983-c7b5-4679-8c1d-90ea13ebca99-kube-api-access-6whz8\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-677b45-29889915-4m4cj\" (UID: \"b5117983-c7b5-4679-8c1d-90ea13ebca99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-677b45-29889915-4m4cj"
Oct 30 21:15:00.473177 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.473123    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2184c91-9d42-4a01-bf05-dd962ff62dcb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889915-27nkg\" (UID: \"c2184c91-9d42-4a01-bf05-dd962ff62dcb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-27nkg"
Oct 30 21:15:00.473177 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.473156    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8d5eebe-b80b-4b99-ace8-7e87724decaf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-cb6404-29889915-4c7lb\" (UID: \"a8d5eebe-b80b-4b99-ace8-7e87724decaf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cb6404-29889915-4c7lb"
Oct 30 21:15:00.473282 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.473179    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3a522bc-746d-4cdf-940d-ccb562ade3c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-5c45e5-29889915-zm4fd\" (UID: \"c3a522bc-746d-4cdf-940d-ccb562ade3c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5c45e5-29889915-zm4fd"
Oct 30 21:15:00.473344 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.473307    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9970e7c4-7661-47c5-a6df-4de2bea436e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-f9f277-29889915-dbd9b\" (UID: \"9970e7c4-7661-47c5-a6df-4de2bea436e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f9f277-29889915-dbd9b"
Oct 30 21:15:00.473496 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.473473    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f59ff9d3-2927-4021-9103-fd2f60f7b655-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-2f9eff-29889915-wpmls\" (UID: \"f59ff9d3-2927-4021-9103-fd2f60f7b655\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f9eff-29889915-wpmls"
Oct 30 21:15:00.473736 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.473715    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/126910b1-789a-4e00-9e84-2f0a9c899abb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-682999-29889915-vzbj5\" (UID: \"126910b1-789a-4e00-9e84-2f0a9c899abb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-682999-29889915-vzbj5"
Oct 30 21:15:00.494847 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.494774    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889915-x55nz"
Oct 30 21:15:00.512565 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.512531    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-390b71-29889915-lrfqc"
Oct 30 21:15:00.518134 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.518107    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88da25-29889915-gf4hb"
Oct 30 21:15:00.532552 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.532508    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dmlh\" (UniqueName: \"kubernetes.io/projected/9970e7c4-7661-47c5-a6df-4de2bea436e2-kube-api-access-8dmlh\") pod \"maas-api-key-cleanup-e2e-worker-w1-f9f277-29889915-dbd9b\" (UID: \"9970e7c4-7661-47c5-a6df-4de2bea436e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f9f277-29889915-dbd9b"
Oct 30 21:15:00.532552 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.532537    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pm9b\" (UniqueName: \"kubernetes.io/projected/f59ff9d3-2927-4021-9103-fd2f60f7b655-kube-api-access-9pm9b\") pod \"maas-api-key-cleanup-e2e-inf-a-2f9eff-29889915-wpmls\" (UID: \"f59ff9d3-2927-4021-9103-fd2f60f7b655\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f9eff-29889915-wpmls"
Oct 30 21:15:00.533109 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.533084    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmmqr\" (UniqueName: \"kubernetes.io/projected/c2184c91-9d42-4a01-bf05-dd962ff62dcb-kube-api-access-tmmqr\") pod \"maas-api-key-cleanup-29889915-27nkg\" (UID: \"c2184c91-9d42-4a01-bf05-dd962ff62dcb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-27nkg"
Oct 30 21:15:00.534206 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.534180    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsqft\" (UniqueName: \"kubernetes.io/projected/c3a522bc-746d-4cdf-940d-ccb562ade3c0-kube-api-access-nsqft\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-5c45e5-29889915-zm4fd\" (UID: \"c3a522bc-746d-4cdf-940d-ccb562ade3c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5c45e5-29889915-zm4fd"
Oct 30 21:15:00.541957 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.541917    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpzrc\" (UniqueName: \"kubernetes.io/projected/126910b1-789a-4e00-9e84-2f0a9c899abb-kube-api-access-wpzrc\") pod \"maas-api-key-cleanup-e2e-worker-w3-682999-29889915-vzbj5\" (UID: \"126910b1-789a-4e00-9e84-2f0a9c899abb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-682999-29889915-vzbj5"
Oct 30 21:15:00.543465 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.543435    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vsmv\" (UniqueName: \"kubernetes.io/projected/a8d5eebe-b80b-4b99-ace8-7e87724decaf-kube-api-access-2vsmv\") pod \"maas-api-key-cleanup-e2e-api-b-cb6404-29889915-4c7lb\" (UID: \"a8d5eebe-b80b-4b99-ace8-7e87724decaf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cb6404-29889915-4c7lb"
Oct 30 21:15:00.545225 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.545201    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5c45e5-29889915-zm4fd"
Oct 30 21:15:00.552417 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.552392    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f9f277-29889915-dbd9b"
Oct 30 21:15:00.584826 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.583156    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6whz8\" (UniqueName: \"kubernetes.io/projected/b5117983-c7b5-4679-8c1d-90ea13ebca99-kube-api-access-6whz8\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-677b45-29889915-4m4cj\" (UID: \"b5117983-c7b5-4679-8c1d-90ea13ebca99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-677b45-29889915-4m4cj"
Oct 30 21:15:00.584826 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.583298    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5117983-c7b5-4679-8c1d-90ea13ebca99-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-677b45-29889915-4m4cj\" (UID: \"b5117983-c7b5-4679-8c1d-90ea13ebca99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-677b45-29889915-4m4cj"
Oct 30 21:15:00.584826 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.584434    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5117983-c7b5-4679-8c1d-90ea13ebca99-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-677b45-29889915-4m4cj\" (UID: \"b5117983-c7b5-4679-8c1d-90ea13ebca99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-677b45-29889915-4m4cj"
Oct 30 21:15:00.608367 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.608121    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-27nkg"
Oct 30 21:15:00.623183 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.622921    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-682999-29889915-vzbj5"
Oct 30 21:15:00.633123 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.630843    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cb6404-29889915-4c7lb"
Oct 30 21:15:00.649854 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.649423    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f9eff-29889915-wpmls"
Oct 30 21:15:00.651867 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.651606    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6whz8\" (UniqueName: \"kubernetes.io/projected/b5117983-c7b5-4679-8c1d-90ea13ebca99-kube-api-access-6whz8\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-677b45-29889915-4m4cj\" (UID: \"b5117983-c7b5-4679-8c1d-90ea13ebca99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-677b45-29889915-4m4cj"
Oct 30 21:15:00.688880 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.685529    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-677b45-29889915-4m4cj"
Oct 30 21:15:00.942776 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:15:00.942673    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3543ab8c_8863_4207_ae05_d8094dbd8cb9.slice/crio-dc2306e64d1ea3c84659438a4bb596720f12cc68d173b08febb55acae0d5be4a WatchSource:0}: Error finding container dc2306e64d1ea3c84659438a4bb596720f12cc68d173b08febb55acae0d5be4a: Status 404 returned error can't find the container with id dc2306e64d1ea3c84659438a4bb596720f12cc68d173b08febb55acae0d5be4a
Oct 30 21:15:00.944588 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:00.944558    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889915-x55nz"]
Oct 30 21:15:01.083050 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:01.083021    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f9f277-29889915-dbd9b"]
Oct 30 21:15:01.137495 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:15:01.137462    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9970e7c4_7661_47c5_a6df_4de2bea436e2.slice/crio-db031a964d81dfd6850aa26fb5d9ce2e8ddbca1a618447f20b665787d2ac8c8a WatchSource:0}: Error finding container db031a964d81dfd6850aa26fb5d9ce2e8ddbca1a618447f20b665787d2ac8c8a: Status 404 returned error can't find the container with id db031a964d81dfd6850aa26fb5d9ce2e8ddbca1a618447f20b665787d2ac8c8a
Oct 30 21:15:01.169082 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:01.169045    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f9f277-29889915-dbd9b" event={"ID":"9970e7c4-7661-47c5-a6df-4de2bea436e2","Type":"ContainerStarted","Data":"db031a964d81dfd6850aa26fb5d9ce2e8ddbca1a618447f20b665787d2ac8c8a"}
Oct 30 21:15:01.171087 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:01.171058    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889915-x55nz" event={"ID":"3543ab8c-8863-4207-ae05-d8094dbd8cb9","Type":"ContainerStarted","Data":"dc2306e64d1ea3c84659438a4bb596720f12cc68d173b08febb55acae0d5be4a"}
Oct 30 21:15:01.193599 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:01.193518    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f9eff-29889915-wpmls"]
Oct 30 21:15:01.200453 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:15:01.200416    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf59ff9d3_2927_4021_9103_fd2f60f7b655.slice/crio-e640375006d4cd8fafb9696a400e714f28a24d4083b888e30bdb10ec10c918e3 WatchSource:0}: Error finding container e640375006d4cd8fafb9696a400e714f28a24d4083b888e30bdb10ec10c918e3: Status 404 returned error can't find the container with id e640375006d4cd8fafb9696a400e714f28a24d4083b888e30bdb10ec10c918e3
Oct 30 21:15:01.218805 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:01.218761    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-390b71-29889915-lrfqc"]
Oct 30 21:15:01.231472 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:15:01.231438    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8d5eebe_b80b_4b99_ace8_7e87724decaf.slice/crio-4e93f8fd35297c0ef333120c31848be4a64e950f17652c6f656ed4079834e923 WatchSource:0}: Error finding container 4e93f8fd35297c0ef333120c31848be4a64e950f17652c6f656ed4079834e923: Status 404 returned error can't find the container with id 4e93f8fd35297c0ef333120c31848be4a64e950f17652c6f656ed4079834e923
Oct 30 21:15:01.233556 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:01.233536    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cb6404-29889915-4c7lb"]
Oct 30 21:15:01.242502 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:01.242475    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88da25-29889915-gf4hb"]
Oct 30 21:15:01.246612 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:01.246586    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-677b45-29889915-4m4cj"]
Oct 30 21:15:01.270458 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:01.270419    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5c45e5-29889915-zm4fd"]
Oct 30 21:15:01.336988 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:01.336948    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889915-27nkg"]
Oct 30 21:15:01.337647 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:15:01.337619    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5117983_c7b5_4679_8c1d_90ea13ebca99.slice/crio-1178eda94691d33d074ad95e5142499d24f892fba2929d997e1ccf0566181a14 WatchSource:0}: Error finding container 1178eda94691d33d074ad95e5142499d24f892fba2929d997e1ccf0566181a14: Status 404 returned error can't find the container with id 1178eda94691d33d074ad95e5142499d24f892fba2929d997e1ccf0566181a14
Oct 30 21:15:01.381649 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:01.381623    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-682999-29889915-vzbj5"]
Oct 30 21:15:01.439421 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:15:01.439388    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3a522bc_746d_4cdf_940d_ccb562ade3c0.slice/crio-aed3a280f06d4b5029de2fb16c29ceb74afba8d6cd7ea2dd68e775c8918b7099 WatchSource:0}: Error finding container aed3a280f06d4b5029de2fb16c29ceb74afba8d6cd7ea2dd68e775c8918b7099: Status 404 returned error can't find the container with id aed3a280f06d4b5029de2fb16c29ceb74afba8d6cd7ea2dd68e775c8918b7099
Oct 30 21:15:01.439593 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:15:01.439574    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2184c91_9d42_4a01_bf05_dd962ff62dcb.slice/crio-b6d41c6ded9e476f1e7180424f60c133aec38863f50fbb493a3af1bc399cce98 WatchSource:0}: Error finding container b6d41c6ded9e476f1e7180424f60c133aec38863f50fbb493a3af1bc399cce98: Status 404 returned error can't find the container with id b6d41c6ded9e476f1e7180424f60c133aec38863f50fbb493a3af1bc399cce98
Oct 30 21:15:01.440701 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:15:01.440552    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod126910b1_789a_4e00_9e84_2f0a9c899abb.slice/crio-51f21fb20b8c7f98b4b7edaa4a473ae4741a599792ce358c12c4cda15aaf86a8 WatchSource:0}: Error finding container 51f21fb20b8c7f98b4b7edaa4a473ae4741a599792ce358c12c4cda15aaf86a8: Status 404 returned error can't find the container with id 51f21fb20b8c7f98b4b7edaa4a473ae4741a599792ce358c12c4cda15aaf86a8
Oct 30 21:15:02.181709 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:02.181674    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-682999-29889915-vzbj5" event={"ID":"126910b1-789a-4e00-9e84-2f0a9c899abb","Type":"ContainerStarted","Data":"51f21fb20b8c7f98b4b7edaa4a473ae4741a599792ce358c12c4cda15aaf86a8"}
Oct 30 21:15:02.183087 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:02.183063    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5c45e5-29889915-zm4fd" event={"ID":"c3a522bc-746d-4cdf-940d-ccb562ade3c0","Type":"ContainerStarted","Data":"aed3a280f06d4b5029de2fb16c29ceb74afba8d6cd7ea2dd68e775c8918b7099"}
Oct 30 21:15:02.184776 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:02.184747    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f9f277-29889915-dbd9b" event={"ID":"9970e7c4-7661-47c5-a6df-4de2bea436e2","Type":"ContainerStarted","Data":"934ef3cefb2086adb3e8f5f567d374958d9788a7f92b0d532ea48b643ff8254b"}
Oct 30 21:15:02.186460 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:02.186436    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-677b45-29889915-4m4cj" event={"ID":"b5117983-c7b5-4679-8c1d-90ea13ebca99","Type":"ContainerStarted","Data":"bea0c092342a3363bdccd75007c651afbfdb069ab917f87ce6c5a6d03b200058"}
Oct 30 21:15:02.186592 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:02.186473    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-677b45-29889915-4m4cj" event={"ID":"b5117983-c7b5-4679-8c1d-90ea13ebca99","Type":"ContainerStarted","Data":"1178eda94691d33d074ad95e5142499d24f892fba2929d997e1ccf0566181a14"}
Oct 30 21:15:02.188208 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:02.188178    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88da25-29889915-gf4hb" event={"ID":"16adca55-5cc2-4452-aba7-3e891c005f98","Type":"ContainerStarted","Data":"a187529f417996076e59b36f94ed39e6d00e40905b42cc57126bf8ae25ebb2e3"}
Oct 30 21:15:02.188324 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:02.188226    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88da25-29889915-gf4hb" event={"ID":"16adca55-5cc2-4452-aba7-3e891c005f98","Type":"ContainerStarted","Data":"8cbf8227dfcff041317b9c0a18ee1ad27b93dd0570bf27e7a1a878c5191e87b4"}
Oct 30 21:15:02.189827 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:02.189766    5713 generic.go:356] "Generic (PLEG): container finished" podID="3543ab8c-8863-4207-ae05-d8094dbd8cb9" containerID="37bd304c29d143646874d63fc86d7e5ee822df8407cbeacee0d5351d7a605b91" exitCode=60
Oct 30 21:15:02.189827 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:02.189817    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889915-x55nz" event={"ID":"3543ab8c-8863-4207-ae05-d8094dbd8cb9","Type":"ContainerDied","Data":"37bd304c29d143646874d63fc86d7e5ee822df8407cbeacee0d5351d7a605b91"}
Oct 30 21:15:02.190082 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:02.190064    5713 scope.go:122] "RemoveContainer" containerID="37bd304c29d143646874d63fc86d7e5ee822df8407cbeacee0d5351d7a605b91"
Oct 30 21:15:02.191333 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:02.191307    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-27nkg" event={"ID":"c2184c91-9d42-4a01-bf05-dd962ff62dcb","Type":"ContainerStarted","Data":"b6d41c6ded9e476f1e7180424f60c133aec38863f50fbb493a3af1bc399cce98"}
Oct 30 21:15:02.192839 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:02.192812    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cb6404-29889915-4c7lb" event={"ID":"a8d5eebe-b80b-4b99-ace8-7e87724decaf","Type":"ContainerStarted","Data":"4e93f8fd35297c0ef333120c31848be4a64e950f17652c6f656ed4079834e923"}
Oct 30 21:15:02.194368 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:02.194345    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-390b71-29889915-lrfqc" event={"ID":"aa46459a-d46e-4af4-bdfc-d6872367429d","Type":"ContainerStarted","Data":"81233ce901a7f640950fc311323c949ba9052256ac34b6439a30f14d12502d7c"}
Oct 30 21:15:02.196154 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:02.196133    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f9eff-29889915-wpmls" event={"ID":"f59ff9d3-2927-4021-9103-fd2f60f7b655","Type":"ContainerStarted","Data":"ac02ce61ba56025188b22efc0155ca952667e04517b58f63000f26ce5f50cc30"}
Oct 30 21:15:02.196260 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:02.196170    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f9eff-29889915-wpmls" event={"ID":"f59ff9d3-2927-4021-9103-fd2f60f7b655","Type":"ContainerStarted","Data":"e640375006d4cd8fafb9696a400e714f28a24d4083b888e30bdb10ec10c918e3"}
Oct 30 21:15:02.305091 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:02.304993    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f9f277-29889915-dbd9b" podStartSLOduration=2.304978688 podStartE2EDuration="2.304978688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:02.300637209 +0000 UTC m=+2185.365413608" watchObservedRunningTime="2026-10-30 21:15:02.304978688 +0000 UTC m=+2185.369755119"
Oct 30 21:15:02.415393 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:02.415332    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88da25-29889915-gf4hb" podStartSLOduration=2.415316079 podStartE2EDuration="2.415316079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:02.414907169 +0000 UTC m=+2185.479683567" watchObservedRunningTime="2026-10-30 21:15:02.415316079 +0000 UTC m=+2185.480092479"
Oct 30 21:15:02.532195 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:02.532140    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f9eff-29889915-wpmls" podStartSLOduration=2.532125871 podStartE2EDuration="2.532125871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:02.52972935 +0000 UTC m=+2185.594505756" watchObservedRunningTime="2026-10-30 21:15:02.532125871 +0000 UTC m=+2185.596902269"
Oct 30 21:15:02.759694 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:02.759182    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-677b45-29889915-4m4cj" podStartSLOduration=2.759162926 podStartE2EDuration="2.759162926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:02.757622726 +0000 UTC m=+2185.822399125" watchObservedRunningTime="2026-10-30 21:15:02.759162926 +0000 UTC m=+2185.823939325"
Oct 30 21:15:03.207308 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:03.207266    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889915-x55nz" event={"ID":"3543ab8c-8863-4207-ae05-d8094dbd8cb9","Type":"ContainerStarted","Data":"28283d6f522f6c008824c593f00cc1895db9b58ae0a7780388f27608a28a6ed3"}
Oct 30 21:15:03.208973 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:03.208943    5713 generic.go:356] "Generic (PLEG): container finished" podID="c2184c91-9d42-4a01-bf05-dd962ff62dcb" containerID="3642e53ac06d37bea5e412865e799fab12618fe641f447c193c2238b21cf6ba7" exitCode=60
Oct 30 21:15:03.209119 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:03.209002    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-27nkg" event={"ID":"c2184c91-9d42-4a01-bf05-dd962ff62dcb","Type":"ContainerDied","Data":"3642e53ac06d37bea5e412865e799fab12618fe641f447c193c2238b21cf6ba7"}
Oct 30 21:15:03.209330 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:03.209313    5713 scope.go:122] "RemoveContainer" containerID="3642e53ac06d37bea5e412865e799fab12618fe641f447c193c2238b21cf6ba7"
Oct 30 21:15:03.210858 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:03.210833    5713 generic.go:356] "Generic (PLEG): container finished" podID="a8d5eebe-b80b-4b99-ace8-7e87724decaf" containerID="3b2a4ed2eb83a76878b55b29a0ca8f33a292d0783440a01d7d47d5c0df471692" exitCode=60
Oct 30 21:15:03.210993 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:03.210918    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cb6404-29889915-4c7lb" event={"ID":"a8d5eebe-b80b-4b99-ace8-7e87724decaf","Type":"ContainerDied","Data":"3b2a4ed2eb83a76878b55b29a0ca8f33a292d0783440a01d7d47d5c0df471692"}
Oct 30 21:15:03.211191 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:03.211174    5713 scope.go:122] "RemoveContainer" containerID="3b2a4ed2eb83a76878b55b29a0ca8f33a292d0783440a01d7d47d5c0df471692"
Oct 30 21:15:03.212700 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:03.212680    5713 generic.go:356] "Generic (PLEG): container finished" podID="aa46459a-d46e-4af4-bdfc-d6872367429d" containerID="ce7732ed6066f27023ec833fd7c4557d49d41d71efe70f0aff1612412193c3e9" exitCode=60
Oct 30 21:15:03.212844 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:03.212754    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-390b71-29889915-lrfqc" event={"ID":"aa46459a-d46e-4af4-bdfc-d6872367429d","Type":"ContainerDied","Data":"ce7732ed6066f27023ec833fd7c4557d49d41d71efe70f0aff1612412193c3e9"}
Oct 30 21:15:03.215913 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:03.215884    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f9eff-29889915-wpmls" event={"ID":"f59ff9d3-2927-4021-9103-fd2f60f7b655","Type":"ContainerDied","Data":"ac02ce61ba56025188b22efc0155ca952667e04517b58f63000f26ce5f50cc30"}
Oct 30 21:15:03.215913 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:03.212959    5713 scope.go:122] "RemoveContainer" containerID="ce7732ed6066f27023ec833fd7c4557d49d41d71efe70f0aff1612412193c3e9"
Oct 30 21:15:03.216035 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:03.214616    5713 generic.go:356] "Generic (PLEG): container finished" podID="f59ff9d3-2927-4021-9103-fd2f60f7b655" containerID="ac02ce61ba56025188b22efc0155ca952667e04517b58f63000f26ce5f50cc30" exitCode=60
Oct 30 21:15:03.216090 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:03.216051    5713 scope.go:122] "RemoveContainer" containerID="ac02ce61ba56025188b22efc0155ca952667e04517b58f63000f26ce5f50cc30"
Oct 30 21:15:03.217650 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:03.217629    5713 generic.go:356] "Generic (PLEG): container finished" podID="126910b1-789a-4e00-9e84-2f0a9c899abb" containerID="990f8cd7d3e2d2b0228642e803b8f36815e5198122c5befde265f581c9a66106" exitCode=60
Oct 30 21:15:03.217732 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:03.217700    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-682999-29889915-vzbj5" event={"ID":"126910b1-789a-4e00-9e84-2f0a9c899abb","Type":"ContainerDied","Data":"990f8cd7d3e2d2b0228642e803b8f36815e5198122c5befde265f581c9a66106"}
Oct 30 21:15:03.225841 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:03.218702    5713 scope.go:122] "RemoveContainer" containerID="990f8cd7d3e2d2b0228642e803b8f36815e5198122c5befde265f581c9a66106"
Oct 30 21:15:03.232806 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:03.232763    5713 generic.go:356] "Generic (PLEG): container finished" podID="c3a522bc-746d-4cdf-940d-ccb562ade3c0" containerID="8aef81d64f4840aa8353ecea23419343aa7a4970407a1d1cb9f5cd1e2c3ea7df" exitCode=60
Oct 30 21:15:03.233265 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:03.233236    5713 scope.go:122] "RemoveContainer" containerID="8aef81d64f4840aa8353ecea23419343aa7a4970407a1d1cb9f5cd1e2c3ea7df"
Oct 30 21:15:03.233321 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:03.233275    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5c45e5-29889915-zm4fd" event={"ID":"c3a522bc-746d-4cdf-940d-ccb562ade3c0","Type":"ContainerDied","Data":"8aef81d64f4840aa8353ecea23419343aa7a4970407a1d1cb9f5cd1e2c3ea7df"}
Oct 30 21:15:03.235823 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:03.235742    5713 generic.go:356] "Generic (PLEG): container finished" podID="9970e7c4-7661-47c5-a6df-4de2bea436e2" containerID="934ef3cefb2086adb3e8f5f567d374958d9788a7f92b0d532ea48b643ff8254b" exitCode=60
Oct 30 21:15:03.236061 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:03.236034    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f9f277-29889915-dbd9b" event={"ID":"9970e7c4-7661-47c5-a6df-4de2bea436e2","Type":"ContainerDied","Data":"934ef3cefb2086adb3e8f5f567d374958d9788a7f92b0d532ea48b643ff8254b"}
Oct 30 21:15:03.236432 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:03.236410    5713 scope.go:122] "RemoveContainer" containerID="934ef3cefb2086adb3e8f5f567d374958d9788a7f92b0d532ea48b643ff8254b"
Oct 30 21:15:03.237880 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:03.237857    5713 generic.go:356] "Generic (PLEG): container finished" podID="b5117983-c7b5-4679-8c1d-90ea13ebca99" containerID="bea0c092342a3363bdccd75007c651afbfdb069ab917f87ce6c5a6d03b200058" exitCode=60
Oct 30 21:15:03.237991 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:03.237924    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-677b45-29889915-4m4cj" event={"ID":"b5117983-c7b5-4679-8c1d-90ea13ebca99","Type":"ContainerDied","Data":"bea0c092342a3363bdccd75007c651afbfdb069ab917f87ce6c5a6d03b200058"}
Oct 30 21:15:03.238132 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:03.238115    5713 scope.go:122] "RemoveContainer" containerID="bea0c092342a3363bdccd75007c651afbfdb069ab917f87ce6c5a6d03b200058"
Oct 30 21:15:03.239706 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:03.239686    5713 generic.go:356] "Generic (PLEG): container finished" podID="16adca55-5cc2-4452-aba7-3e891c005f98" containerID="a187529f417996076e59b36f94ed39e6d00e40905b42cc57126bf8ae25ebb2e3" exitCode=60
Oct 30 21:15:03.239813 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:03.239769    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88da25-29889915-gf4hb" event={"ID":"16adca55-5cc2-4452-aba7-3e891c005f98","Type":"ContainerDied","Data":"a187529f417996076e59b36f94ed39e6d00e40905b42cc57126bf8ae25ebb2e3"}
Oct 30 21:15:03.240013 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:03.240000    5713 scope.go:122] "RemoveContainer" containerID="a187529f417996076e59b36f94ed39e6d00e40905b42cc57126bf8ae25ebb2e3"
Oct 30 21:15:03.324544 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:03.324491    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889915-x55nz" podStartSLOduration=3.324477089 podStartE2EDuration="3.324477089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:03.322476655 +0000 UTC m=+2186.387253065" watchObservedRunningTime="2026-10-30 21:15:03.324477089 +0000 UTC m=+2186.389253488"
Oct 30 21:15:03.664699 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:03.664655    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-d061d0-644b5ff884-cj67l" podUID="a08925fb-aee2-425a-84da-c407fb8cfb7f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:15:03.664858 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:03.664747    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d061d0-9888979ff-kclfx" podUID="b47eb722-c69a-4565-b0ae-1b46f99090df" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:15:04.250049 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:04.250008    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88da25-29889915-gf4hb" event={"ID":"16adca55-5cc2-4452-aba7-3e891c005f98","Type":"ContainerStarted","Data":"5e9f0ad8e8e97cc8c62b1fbf9772c971c576de7112bcd9155d331bcc45d1f797"}
Oct 30 21:15:04.252083 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:04.252039    5713 generic.go:356] "Generic (PLEG): container finished" podID="3543ab8c-8863-4207-ae05-d8094dbd8cb9" containerID="28283d6f522f6c008824c593f00cc1895db9b58ae0a7780388f27608a28a6ed3" exitCode=60
Oct 30 21:15:04.252238 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:04.252125    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889915-x55nz" event={"ID":"3543ab8c-8863-4207-ae05-d8094dbd8cb9","Type":"ContainerDied","Data":"28283d6f522f6c008824c593f00cc1895db9b58ae0a7780388f27608a28a6ed3"}
Oct 30 21:15:04.252238 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:04.252190    5713 scope.go:122] "RemoveContainer" containerID="37bd304c29d143646874d63fc86d7e5ee822df8407cbeacee0d5351d7a605b91"
Oct 30 21:15:04.252384 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:04.252364    5713 scope.go:122] "RemoveContainer" containerID="28283d6f522f6c008824c593f00cc1895db9b58ae0a7780388f27608a28a6ed3"
Oct 30 21:15:04.252658 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:15:04.252634    5713 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889915-x55nz_odh-ai-gateway-infra(3543ab8c-8863-4207-ae05-d8094dbd8cb9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889915-x55nz" podUID="3543ab8c-8863-4207-ae05-d8094dbd8cb9"
Oct 30 21:15:04.254356 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:04.254333    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-27nkg" event={"ID":"c2184c91-9d42-4a01-bf05-dd962ff62dcb","Type":"ContainerStarted","Data":"3bacf903345540117b3aebca7ce72028ef5721b2232c813deb2527a119bd6c6e"}
Oct 30 21:15:04.256369 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:04.256343    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cb6404-29889915-4c7lb" event={"ID":"a8d5eebe-b80b-4b99-ace8-7e87724decaf","Type":"ContainerStarted","Data":"325c0a83c19298df8512df9f26e9f9fed0e2bbf3b8d23d506f9c662004ad974b"}
Oct 30 21:15:04.258452 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:04.258430    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-390b71-29889915-lrfqc" event={"ID":"aa46459a-d46e-4af4-bdfc-d6872367429d","Type":"ContainerStarted","Data":"e63727cab85802becf3e0a803256dfb821b9d6137db5f155a3462f6c405e5405"}
Oct 30 21:15:04.260647 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:04.260621    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f9eff-29889915-wpmls" event={"ID":"f59ff9d3-2927-4021-9103-fd2f60f7b655","Type":"ContainerStarted","Data":"c73222e686d9cbe7e8a0e317f04fda67016d298f2efd4011acc282c5436b6914"}
Oct 30 21:15:04.263861 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:04.263836    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f9f277-29889915-dbd9b" event={"ID":"9970e7c4-7661-47c5-a6df-4de2bea436e2","Type":"ContainerStarted","Data":"7fe97e301ee843caf143d1182cefe60b87fed3180e56faf490513a7213b71597"}
Oct 30 21:15:04.644650 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:04.644596    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cb6404-29889915-4c7lb" podStartSLOduration=4.6445846809999995 podStartE2EDuration="4.644584681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:04.644253834 +0000 UTC m=+2187.709030232" watchObservedRunningTime="2026-10-30 21:15:04.644584681 +0000 UTC m=+2187.709361078"
Oct 30 21:15:04.648619 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:04.648566    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-390b71-29889915-lrfqc" podStartSLOduration=4.648551823 podStartE2EDuration="4.648551823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:04.520278644 +0000 UTC m=+2187.585055041" watchObservedRunningTime="2026-10-30 21:15:04.648551823 +0000 UTC m=+2187.713328220"
Oct 30 21:15:04.994887 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:04.994756    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-27nkg" podStartSLOduration=4.994742007 podStartE2EDuration="4.994742007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:04.992780344 +0000 UTC m=+2188.057556743" watchObservedRunningTime="2026-10-30 21:15:04.994742007 +0000 UTC m=+2188.059518405"
Oct 30 21:15:05.274922 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:05.274820    5713 generic.go:356] "Generic (PLEG): container finished" podID="9970e7c4-7661-47c5-a6df-4de2bea436e2" containerID="7fe97e301ee843caf143d1182cefe60b87fed3180e56faf490513a7213b71597" exitCode=60
Oct 30 21:15:05.274922 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:05.274882    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f9f277-29889915-dbd9b" event={"ID":"9970e7c4-7661-47c5-a6df-4de2bea436e2","Type":"ContainerDied","Data":"7fe97e301ee843caf143d1182cefe60b87fed3180e56faf490513a7213b71597"}
Oct 30 21:15:05.275429 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:05.274981    5713 scope.go:122] "RemoveContainer" containerID="934ef3cefb2086adb3e8f5f567d374958d9788a7f92b0d532ea48b643ff8254b"
Oct 30 21:15:05.275429 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:05.275203    5713 scope.go:122] "RemoveContainer" containerID="7fe97e301ee843caf143d1182cefe60b87fed3180e56faf490513a7213b71597"
Oct 30 21:15:05.275614 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:15:05.275585    5713 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-f9f277-29889915-dbd9b_odh-ai-gateway-infra(9970e7c4-7661-47c5-a6df-4de2bea436e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f9f277-29889915-dbd9b" podUID="9970e7c4-7661-47c5-a6df-4de2bea436e2"
Oct 30 21:15:05.277221 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:05.277197    5713 generic.go:356] "Generic (PLEG): container finished" podID="b5117983-c7b5-4679-8c1d-90ea13ebca99" containerID="12f1188b26fa4dca1c4bcf439a3088825c79c2492bf68230d6fc52dd0dccfca6" exitCode=60
Oct 30 21:15:05.277346 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:05.277280    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-677b45-29889915-4m4cj" event={"ID":"b5117983-c7b5-4679-8c1d-90ea13ebca99","Type":"ContainerDied","Data":"12f1188b26fa4dca1c4bcf439a3088825c79c2492bf68230d6fc52dd0dccfca6"}
Oct 30 21:15:05.277607 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:05.277590    5713 scope.go:122] "RemoveContainer" containerID="12f1188b26fa4dca1c4bcf439a3088825c79c2492bf68230d6fc52dd0dccfca6"
Oct 30 21:15:05.278013 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:15:05.277990    5713 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-677b45-29889915-4m4cj_odh-ai-gateway-infra(b5117983-c7b5-4679-8c1d-90ea13ebca99)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-677b45-29889915-4m4cj" podUID="b5117983-c7b5-4679-8c1d-90ea13ebca99"
Oct 30 21:15:05.279334 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:05.279313    5713 generic.go:356] "Generic (PLEG): container finished" podID="16adca55-5cc2-4452-aba7-3e891c005f98" containerID="5e9f0ad8e8e97cc8c62b1fbf9772c971c576de7112bcd9155d331bcc45d1f797" exitCode=60
Oct 30 21:15:05.279412 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:05.279396    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88da25-29889915-gf4hb" event={"ID":"16adca55-5cc2-4452-aba7-3e891c005f98","Type":"ContainerDied","Data":"5e9f0ad8e8e97cc8c62b1fbf9772c971c576de7112bcd9155d331bcc45d1f797"}
Oct 30 21:15:05.279596 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:05.279576    5713 scope.go:122] "RemoveContainer" containerID="5e9f0ad8e8e97cc8c62b1fbf9772c971c576de7112bcd9155d331bcc45d1f797"
Oct 30 21:15:05.279900 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:15:05.279875    5713 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-88da25-29889915-gf4hb_odh-ai-gateway-infra(16adca55-5cc2-4452-aba7-3e891c005f98)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88da25-29889915-gf4hb" podUID="16adca55-5cc2-4452-aba7-3e891c005f98"
Oct 30 21:15:05.281473 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:05.281454    5713 scope.go:122] "RemoveContainer" containerID="28283d6f522f6c008824c593f00cc1895db9b58ae0a7780388f27608a28a6ed3"
Oct 30 21:15:05.281750 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:15:05.281725    5713 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889915-x55nz_odh-ai-gateway-infra(3543ab8c-8863-4207-ae05-d8094dbd8cb9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889915-x55nz" podUID="3543ab8c-8863-4207-ae05-d8094dbd8cb9"
Oct 30 21:15:05.283082 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:05.283061    5713 generic.go:356] "Generic (PLEG): container finished" podID="c2184c91-9d42-4a01-bf05-dd962ff62dcb" containerID="3bacf903345540117b3aebca7ce72028ef5721b2232c813deb2527a119bd6c6e" exitCode=60
Oct 30 21:15:05.283171 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:05.283135    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-27nkg" event={"ID":"c2184c91-9d42-4a01-bf05-dd962ff62dcb","Type":"ContainerDied","Data":"3bacf903345540117b3aebca7ce72028ef5721b2232c813deb2527a119bd6c6e"}
Oct 30 21:15:05.283382 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:05.283368    5713 scope.go:122] "RemoveContainer" containerID="3bacf903345540117b3aebca7ce72028ef5721b2232c813deb2527a119bd6c6e"
Oct 30 21:15:05.283630 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:15:05.283607    5713 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889915-27nkg_odh-ai-gateway-infra(c2184c91-9d42-4a01-bf05-dd962ff62dcb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-27nkg" podUID="c2184c91-9d42-4a01-bf05-dd962ff62dcb"
Oct 30 21:15:05.285178 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:05.285154    5713 generic.go:356] "Generic (PLEG): container finished" podID="a8d5eebe-b80b-4b99-ace8-7e87724decaf" containerID="325c0a83c19298df8512df9f26e9f9fed0e2bbf3b8d23d506f9c662004ad974b" exitCode=60
Oct 30 21:15:05.285272 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:05.285225    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cb6404-29889915-4c7lb" event={"ID":"a8d5eebe-b80b-4b99-ace8-7e87724decaf","Type":"ContainerDied","Data":"325c0a83c19298df8512df9f26e9f9fed0e2bbf3b8d23d506f9c662004ad974b"}
Oct 30 21:15:05.285440 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:05.285423    5713 scope.go:122] "RemoveContainer" containerID="325c0a83c19298df8512df9f26e9f9fed0e2bbf3b8d23d506f9c662004ad974b"
Oct 30 21:15:05.285661 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:15:05.285639    5713 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-cb6404-29889915-4c7lb_odh-ai-gateway-infra(a8d5eebe-b80b-4b99-ace8-7e87724decaf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cb6404-29889915-4c7lb" podUID="a8d5eebe-b80b-4b99-ace8-7e87724decaf"
Oct 30 21:15:05.287248 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:05.287225    5713 generic.go:356] "Generic (PLEG): container finished" podID="aa46459a-d46e-4af4-bdfc-d6872367429d" containerID="e63727cab85802becf3e0a803256dfb821b9d6137db5f155a3462f6c405e5405" exitCode=60
Oct 30 21:15:05.287349 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:05.287292    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-390b71-29889915-lrfqc" event={"ID":"aa46459a-d46e-4af4-bdfc-d6872367429d","Type":"ContainerDied","Data":"e63727cab85802becf3e0a803256dfb821b9d6137db5f155a3462f6c405e5405"}
Oct 30 21:15:05.287518 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:05.287501    5713 scope.go:122] "RemoveContainer" containerID="e63727cab85802becf3e0a803256dfb821b9d6137db5f155a3462f6c405e5405"
Oct 30 21:15:05.287752 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:15:05.287731    5713 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-390b71-29889915-lrfqc_odh-ai-gateway-infra(aa46459a-d46e-4af4-bdfc-d6872367429d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-390b71-29889915-lrfqc" podUID="aa46459a-d46e-4af4-bdfc-d6872367429d"
Oct 30 21:15:05.289238 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:05.289220    5713 generic.go:356] "Generic (PLEG): container finished" podID="f59ff9d3-2927-4021-9103-fd2f60f7b655" containerID="c73222e686d9cbe7e8a0e317f04fda67016d298f2efd4011acc282c5436b6914" exitCode=60
Oct 30 21:15:05.289324 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:05.289301    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f9eff-29889915-wpmls" event={"ID":"f59ff9d3-2927-4021-9103-fd2f60f7b655","Type":"ContainerDied","Data":"c73222e686d9cbe7e8a0e317f04fda67016d298f2efd4011acc282c5436b6914"}
Oct 30 21:15:05.289504 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:05.289488    5713 scope.go:122] "RemoveContainer" containerID="c73222e686d9cbe7e8a0e317f04fda67016d298f2efd4011acc282c5436b6914"
Oct 30 21:15:05.289731 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:15:05.289710    5713 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-2f9eff-29889915-wpmls_odh-ai-gateway-infra(f59ff9d3-2927-4021-9103-fd2f60f7b655)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f9eff-29889915-wpmls" podUID="f59ff9d3-2927-4021-9103-fd2f60f7b655"
Oct 30 21:15:05.291250 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:05.291231    5713 generic.go:356] "Generic (PLEG): container finished" podID="126910b1-789a-4e00-9e84-2f0a9c899abb" containerID="275326fe0b138876b3a03f1e9b84e7548fbdd44053ffde8667c1cebefb86a2c6" exitCode=60
Oct 30 21:15:05.291334 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:05.291259    5713 scope.go:122] "RemoveContainer" containerID="bea0c092342a3363bdccd75007c651afbfdb069ab917f87ce6c5a6d03b200058"
Oct 30 21:15:05.291334 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:05.291312    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-682999-29889915-vzbj5" event={"ID":"126910b1-789a-4e00-9e84-2f0a9c899abb","Type":"ContainerDied","Data":"275326fe0b138876b3a03f1e9b84e7548fbdd44053ffde8667c1cebefb86a2c6"}
Oct 30 21:15:05.291557 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:05.291540    5713 scope.go:122] "RemoveContainer" containerID="275326fe0b138876b3a03f1e9b84e7548fbdd44053ffde8667c1cebefb86a2c6"
Oct 30 21:15:05.291823 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:15:05.291778    5713 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-682999-29889915-vzbj5_odh-ai-gateway-infra(126910b1-789a-4e00-9e84-2f0a9c899abb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-682999-29889915-vzbj5" podUID="126910b1-789a-4e00-9e84-2f0a9c899abb"
Oct 30 21:15:05.293226 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:05.293205    5713 generic.go:356] "Generic (PLEG): container finished" podID="c3a522bc-746d-4cdf-940d-ccb562ade3c0" containerID="e762675006598988b9be035a8b5eb41d1873843016136c9f1561f983a7aadcb3" exitCode=60
Oct 30 21:15:05.293315 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:05.293270    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5c45e5-29889915-zm4fd" event={"ID":"c3a522bc-746d-4cdf-940d-ccb562ade3c0","Type":"ContainerDied","Data":"e762675006598988b9be035a8b5eb41d1873843016136c9f1561f983a7aadcb3"}
Oct 30 21:15:05.293515 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:05.293498    5713 scope.go:122] "RemoveContainer" containerID="e762675006598988b9be035a8b5eb41d1873843016136c9f1561f983a7aadcb3"
Oct 30 21:15:05.293739 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:15:05.293719    5713 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-5c45e5-29889915-zm4fd_odh-ai-gateway-infra(c3a522bc-746d-4cdf-940d-ccb562ade3c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5c45e5-29889915-zm4fd" podUID="c3a522bc-746d-4cdf-940d-ccb562ade3c0"
Oct 30 21:15:05.306256 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:05.306232    5713 scope.go:122] "RemoveContainer" containerID="a187529f417996076e59b36f94ed39e6d00e40905b42cc57126bf8ae25ebb2e3"
Oct 30 21:15:05.321978 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:05.321956    5713 scope.go:122] "RemoveContainer" containerID="3642e53ac06d37bea5e412865e799fab12618fe641f447c193c2238b21cf6ba7"
Oct 30 21:15:05.336609 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:05.336587    5713 scope.go:122] "RemoveContainer" containerID="3b2a4ed2eb83a76878b55b29a0ca8f33a292d0783440a01d7d47d5c0df471692"
Oct 30 21:15:05.352028 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:05.352006    5713 scope.go:122] "RemoveContainer" containerID="ce7732ed6066f27023ec833fd7c4557d49d41d71efe70f0aff1612412193c3e9"
Oct 30 21:15:05.367242 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:05.367208    5713 scope.go:122] "RemoveContainer" containerID="ac02ce61ba56025188b22efc0155ca952667e04517b58f63000f26ce5f50cc30"
Oct 30 21:15:05.382471 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:05.382446    5713 scope.go:122] "RemoveContainer" containerID="990f8cd7d3e2d2b0228642e803b8f36815e5198122c5befde265f581c9a66106"
Oct 30 21:15:05.398356 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:05.398332    5713 scope.go:122] "RemoveContainer" containerID="8aef81d64f4840aa8353ecea23419343aa7a4970407a1d1cb9f5cd1e2c3ea7df"
Oct 30 21:15:06.112971 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:06.112940    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-d061d0-644b5ff884-cj67l"
Oct 30 21:15:06.211112 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:06.211085    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d061d0-9888979ff-kclfx"
Oct 30 21:15:06.283263 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:06.283231    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a08925fb-aee2-425a-84da-c407fb8cfb7f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a08925fb-aee2-425a-84da-c407fb8cfb7f-plugins-config-volume\") pod \"a08925fb-aee2-425a-84da-c407fb8cfb7f\" (UID: \"a08925fb-aee2-425a-84da-c407fb8cfb7f\") "
Oct 30 21:15:06.283263 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:06.283274    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a08925fb-aee2-425a-84da-c407fb8cfb7f-kube-api-access-t89rm\" (UniqueName: \"kubernetes.io/projected/a08925fb-aee2-425a-84da-c407fb8cfb7f-kube-api-access-t89rm\") pod \"a08925fb-aee2-425a-84da-c407fb8cfb7f\" (UID: \"a08925fb-aee2-425a-84da-c407fb8cfb7f\") "
Oct 30 21:15:06.283713 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:06.283622    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a08925fb-aee2-425a-84da-c407fb8cfb7f-plugins-config-volume" pod "a08925fb-aee2-425a-84da-c407fb8cfb7f" (UID: "a08925fb-aee2-425a-84da-c407fb8cfb7f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:06.284028 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:06.284011    5713 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a08925fb-aee2-425a-84da-c407fb8cfb7f-plugins-config-volume\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:06.285578 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:06.285557    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a08925fb-aee2-425a-84da-c407fb8cfb7f-kube-api-access-t89rm" pod "a08925fb-aee2-425a-84da-c407fb8cfb7f" (UID: "a08925fb-aee2-425a-84da-c407fb8cfb7f"). InnerVolumeSpecName "kube-api-access-t89rm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:06.304311 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:06.304286    5713 scope.go:122] "RemoveContainer" containerID="12f1188b26fa4dca1c4bcf439a3088825c79c2492bf68230d6fc52dd0dccfca6"
Oct 30 21:15:06.304707 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:15:06.304680    5713 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-677b45-29889915-4m4cj_odh-ai-gateway-infra(b5117983-c7b5-4679-8c1d-90ea13ebca99)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-677b45-29889915-4m4cj" podUID="b5117983-c7b5-4679-8c1d-90ea13ebca99"
Oct 30 21:15:06.305955 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:06.305935    5713 scope.go:122] "RemoveContainer" containerID="5e9f0ad8e8e97cc8c62b1fbf9772c971c576de7112bcd9155d331bcc45d1f797"
Oct 30 21:15:06.306290 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:15:06.306269    5713 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-88da25-29889915-gf4hb_odh-ai-gateway-infra(16adca55-5cc2-4452-aba7-3e891c005f98)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88da25-29889915-gf4hb" podUID="16adca55-5cc2-4452-aba7-3e891c005f98"
Oct 30 21:15:06.307013 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:06.306991    5713 generic.go:356] "Generic (PLEG): container finished" podID="a08925fb-aee2-425a-84da-c407fb8cfb7f" containerID="c815c95c65e21b95193016db19fc0cfce93d0c3cde714312f9408b5de525fb0f" exitCode=0
Oct 30 21:15:06.307053 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:06.307029    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-d061d0-644b5ff884-cj67l"
Oct 30 21:15:06.307106 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:06.307068    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-d061d0-644b5ff884-cj67l" event={"ID":"a08925fb-aee2-425a-84da-c407fb8cfb7f","Type":"ContainerDied","Data":"c815c95c65e21b95193016db19fc0cfce93d0c3cde714312f9408b5de525fb0f"}
Oct 30 21:15:06.307148 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:06.307118    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-d061d0-644b5ff884-cj67l" event={"ID":"a08925fb-aee2-425a-84da-c407fb8cfb7f","Type":"ContainerDied","Data":"339d1f349064852d0124c70abb184796cafbfda1e72b8d5e460932ddd2cc8211"}
Oct 30 21:15:06.307190 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:06.307145    5713 scope.go:122] "RemoveContainer" containerID="c815c95c65e21b95193016db19fc0cfce93d0c3cde714312f9408b5de525fb0f"
Oct 30 21:15:06.309248 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:06.309219    5713 scope.go:122] "RemoveContainer" containerID="3bacf903345540117b3aebca7ce72028ef5721b2232c813deb2527a119bd6c6e"
Oct 30 21:15:06.309506 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:15:06.309479    5713 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889915-27nkg_odh-ai-gateway-infra(c2184c91-9d42-4a01-bf05-dd962ff62dcb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-27nkg" podUID="c2184c91-9d42-4a01-bf05-dd962ff62dcb"
Oct 30 21:15:06.311063 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:06.311037    5713 scope.go:122] "RemoveContainer" containerID="325c0a83c19298df8512df9f26e9f9fed0e2bbf3b8d23d506f9c662004ad974b"
Oct 30 21:15:06.311315 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:15:06.311279    5713 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-cb6404-29889915-4c7lb_odh-ai-gateway-infra(a8d5eebe-b80b-4b99-ace8-7e87724decaf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cb6404-29889915-4c7lb" podUID="a8d5eebe-b80b-4b99-ace8-7e87724decaf"
Oct 30 21:15:06.312935 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:06.312917    5713 scope.go:122] "RemoveContainer" containerID="e63727cab85802becf3e0a803256dfb821b9d6137db5f155a3462f6c405e5405"
Oct 30 21:15:06.313166 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:15:06.313145    5713 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-390b71-29889915-lrfqc_odh-ai-gateway-infra(aa46459a-d46e-4af4-bdfc-d6872367429d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-390b71-29889915-lrfqc" podUID="aa46459a-d46e-4af4-bdfc-d6872367429d"
Oct 30 21:15:06.314626 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:06.314604    5713 scope.go:122] "RemoveContainer" containerID="c73222e686d9cbe7e8a0e317f04fda67016d298f2efd4011acc282c5436b6914"
Oct 30 21:15:06.314923 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:15:06.314898    5713 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-2f9eff-29889915-wpmls_odh-ai-gateway-infra(f59ff9d3-2927-4021-9103-fd2f60f7b655)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f9eff-29889915-wpmls" podUID="f59ff9d3-2927-4021-9103-fd2f60f7b655"
Oct 30 21:15:06.315859 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:06.315833    5713 generic.go:356] "Generic (PLEG): container finished" podID="b47eb722-c69a-4565-b0ae-1b46f99090df" containerID="11d2d335bebea202947eed0e482f1b44ea4f97286b2a3f48fbb9d4c0aea3ac3c" exitCode=0
Oct 30 21:15:06.315929 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:06.315880    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d061d0-9888979ff-kclfx"
Oct 30 21:15:06.315975 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:06.315929    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d061d0-9888979ff-kclfx" event={"ID":"b47eb722-c69a-4565-b0ae-1b46f99090df","Type":"ContainerDied","Data":"11d2d335bebea202947eed0e482f1b44ea4f97286b2a3f48fbb9d4c0aea3ac3c"}
Oct 30 21:15:06.316029 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:06.315971    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d061d0-9888979ff-kclfx" event={"ID":"b47eb722-c69a-4565-b0ae-1b46f99090df","Type":"ContainerDied","Data":"c94b3f7ffe04da83cd245a72ec4250da65071d8b2fb9514be6673d598b29a3b1"}
Oct 30 21:15:06.317997 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:06.317979    5713 scope.go:122] "RemoveContainer" containerID="275326fe0b138876b3a03f1e9b84e7548fbdd44053ffde8667c1cebefb86a2c6"
Oct 30 21:15:06.318245 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:15:06.318226    5713 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-682999-29889915-vzbj5_odh-ai-gateway-infra(126910b1-789a-4e00-9e84-2f0a9c899abb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-682999-29889915-vzbj5" podUID="126910b1-789a-4e00-9e84-2f0a9c899abb"
Oct 30 21:15:06.319568 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:06.319551    5713 scope.go:122] "RemoveContainer" containerID="e762675006598988b9be035a8b5eb41d1873843016136c9f1561f983a7aadcb3"
Oct 30 21:15:06.319843 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:15:06.319824    5713 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-5c45e5-29889915-zm4fd_odh-ai-gateway-infra(c3a522bc-746d-4cdf-940d-ccb562ade3c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5c45e5-29889915-zm4fd" podUID="c3a522bc-746d-4cdf-940d-ccb562ade3c0"
Oct 30 21:15:06.320704 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:06.320688    5713 scope.go:122] "RemoveContainer" containerID="c815c95c65e21b95193016db19fc0cfce93d0c3cde714312f9408b5de525fb0f"
Oct 30 21:15:06.321005 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:15:06.320986    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c815c95c65e21b95193016db19fc0cfce93d0c3cde714312f9408b5de525fb0f\": container with ID starting with c815c95c65e21b95193016db19fc0cfce93d0c3cde714312f9408b5de525fb0f not found: ID does not exist" containerID="c815c95c65e21b95193016db19fc0cfce93d0c3cde714312f9408b5de525fb0f"
Oct 30 21:15:06.321088 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:06.321010    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c815c95c65e21b95193016db19fc0cfce93d0c3cde714312f9408b5de525fb0f"} err="failed to get container status \"c815c95c65e21b95193016db19fc0cfce93d0c3cde714312f9408b5de525fb0f\": rpc error: code = NotFound desc = could not find container \"c815c95c65e21b95193016db19fc0cfce93d0c3cde714312f9408b5de525fb0f\": container with ID starting with c815c95c65e21b95193016db19fc0cfce93d0c3cde714312f9408b5de525fb0f not found: ID does not exist"
Oct 30 21:15:06.321088 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:06.321026    5713 scope.go:122] "RemoveContainer" containerID="11d2d335bebea202947eed0e482f1b44ea4f97286b2a3f48fbb9d4c0aea3ac3c"
Oct 30 21:15:06.321223 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:06.321207    5713 scope.go:122] "RemoveContainer" containerID="7fe97e301ee843caf143d1182cefe60b87fed3180e56faf490513a7213b71597"
Oct 30 21:15:06.321428 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:15:06.321407    5713 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-f9f277-29889915-dbd9b_odh-ai-gateway-infra(9970e7c4-7661-47c5-a6df-4de2bea436e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f9f277-29889915-dbd9b" podUID="9970e7c4-7661-47c5-a6df-4de2bea436e2"
Oct 30 21:15:06.332336 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:06.332309    5713 scope.go:122] "RemoveContainer" containerID="11d2d335bebea202947eed0e482f1b44ea4f97286b2a3f48fbb9d4c0aea3ac3c"
Oct 30 21:15:06.332634 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:15:06.332613    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11d2d335bebea202947eed0e482f1b44ea4f97286b2a3f48fbb9d4c0aea3ac3c\": container with ID starting with 11d2d335bebea202947eed0e482f1b44ea4f97286b2a3f48fbb9d4c0aea3ac3c not found: ID does not exist" containerID="11d2d335bebea202947eed0e482f1b44ea4f97286b2a3f48fbb9d4c0aea3ac3c"
Oct 30 21:15:06.332683 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:06.332642    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11d2d335bebea202947eed0e482f1b44ea4f97286b2a3f48fbb9d4c0aea3ac3c"} err="failed to get container status \"11d2d335bebea202947eed0e482f1b44ea4f97286b2a3f48fbb9d4c0aea3ac3c\": rpc error: code = NotFound desc = could not find container \"11d2d335bebea202947eed0e482f1b44ea4f97286b2a3f48fbb9d4c0aea3ac3c\": container with ID starting with 11d2d335bebea202947eed0e482f1b44ea4f97286b2a3f48fbb9d4c0aea3ac3c not found: ID does not exist"
Oct 30 21:15:06.385010 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:06.384977    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b47eb722-c69a-4565-b0ae-1b46f99090df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b47eb722-c69a-4565-b0ae-1b46f99090df-plugins-config-volume\") pod \"b47eb722-c69a-4565-b0ae-1b46f99090df\" (UID: \"b47eb722-c69a-4565-b0ae-1b46f99090df\") "
Oct 30 21:15:06.385221 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:06.385208    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b47eb722-c69a-4565-b0ae-1b46f99090df-kube-api-access-589r8\" (UniqueName: \"kubernetes.io/projected/b47eb722-c69a-4565-b0ae-1b46f99090df-kube-api-access-589r8\") pod \"b47eb722-c69a-4565-b0ae-1b46f99090df\" (UID: \"b47eb722-c69a-4565-b0ae-1b46f99090df\") "
Oct 30 21:15:06.385356 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:06.385335    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b47eb722-c69a-4565-b0ae-1b46f99090df-plugins-config-volume" pod "b47eb722-c69a-4565-b0ae-1b46f99090df" (UID: "b47eb722-c69a-4565-b0ae-1b46f99090df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:06.385676 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:06.385660    5713 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b47eb722-c69a-4565-b0ae-1b46f99090df-plugins-config-volume\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:06.385713 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:06.385680    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t89rm\" (UniqueName: \"kubernetes.io/projected/a08925fb-aee2-425a-84da-c407fb8cfb7f-kube-api-access-t89rm\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:06.387477 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:06.387456    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b47eb722-c69a-4565-b0ae-1b46f99090df-kube-api-access-589r8" pod "b47eb722-c69a-4565-b0ae-1b46f99090df" (UID: "b47eb722-c69a-4565-b0ae-1b46f99090df"). InnerVolumeSpecName "kube-api-access-589r8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:06.488380 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:06.488324    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-589r8\" (UniqueName: \"kubernetes.io/projected/b47eb722-c69a-4565-b0ae-1b46f99090df-kube-api-access-589r8\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:07.553317 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:07.553280    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-d061d0-644b5ff884-cj67l"]
Oct 30 21:15:07.557339 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:07.557310    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-d061d0-644b5ff884-cj67l"]
Oct 30 21:15:07.913224 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:07.913144    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-d061d0-9888979ff-kclfx"]
Oct 30 21:15:07.917500 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:07.917473    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-d061d0-9888979ff-kclfx"]
Oct 30 21:15:08.653665 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:08.653628    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2"]
Oct 30 21:15:08.654193 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:08.653913    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2" podUID="e1255405-8724-4977-854d-11f13d8a847a" containerName="istio-proxy" containerID="cri-o://eceac2f023bbcb511f2a9a8ce3078ed19b64ce3e11aa5701a7509788dcc1b859" gracePeriod=30
Oct 30 21:15:08.832613 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:08.832581    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2"]
Oct 30 21:15:08.832813 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:08.832774    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2" podUID="3faef396-57dc-4ff5-af9f-c1f940cce913" containerName="istio-proxy" containerID="cri-o://6c3f052bb8ec591c4c5dff555fbca6d71e69e33f79582ba339fde1b02474b8dc" gracePeriod=30
Oct 30 21:15:09.064683 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:09.064647    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a08925fb-aee2-425a-84da-c407fb8cfb7f" path="/var/lib/kubelet/pods/a08925fb-aee2-425a-84da-c407fb8cfb7f/volumes"
Oct 30 21:15:09.065124 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:09.065094    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b47eb722-c69a-4565-b0ae-1b46f99090df" path="/var/lib/kubelet/pods/b47eb722-c69a-4565-b0ae-1b46f99090df/volumes"
Oct 30 21:15:11.923653 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:11.923611    5713 patch_prober.go:29] interesting pod/e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:15:11.924094 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:11.923670    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2" podUID="e1255405-8724-4977-854d-11f13d8a847a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:15:13.971912 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:13.971867    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2"
Oct 30 21:15:14.099278 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.099251    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2"
Oct 30 21:15:14.125950 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.125920    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e1255405-8724-4977-854d-11f13d8a847a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1255405-8724-4977-854d-11f13d8a847a-istio-podinfo\") pod \"e1255405-8724-4977-854d-11f13d8a847a\" (UID: \"e1255405-8724-4977-854d-11f13d8a847a\") "
Oct 30 21:15:14.126104 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.126010    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1255405-8724-4977-854d-11f13d8a847a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1255405-8724-4977-854d-11f13d8a847a-credential-socket\") pod \"e1255405-8724-4977-854d-11f13d8a847a\" (UID: \"e1255405-8724-4977-854d-11f13d8a847a\") "
Oct 30 21:15:14.126104 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.126072    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1255405-8724-4977-854d-11f13d8a847a-istio-token\" (UniqueName: \"kubernetes.io/projected/e1255405-8724-4977-854d-11f13d8a847a-istio-token\") pod \"e1255405-8724-4977-854d-11f13d8a847a\" (UID: \"e1255405-8724-4977-854d-11f13d8a847a\") "
Oct 30 21:15:14.126218 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.126161    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1255405-8724-4977-854d-11f13d8a847a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1255405-8724-4977-854d-11f13d8a847a-istiod-ca-cert\") pod \"e1255405-8724-4977-854d-11f13d8a847a\" (UID: \"e1255405-8724-4977-854d-11f13d8a847a\") "
Oct 30 21:15:14.126271 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.126253    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1255405-8724-4977-854d-11f13d8a847a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1255405-8724-4977-854d-11f13d8a847a-istio-envoy\") pod \"e1255405-8724-4977-854d-11f13d8a847a\" (UID: \"e1255405-8724-4977-854d-11f13d8a847a\") "
Oct 30 21:15:14.126350 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.126320    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1255405-8724-4977-854d-11f13d8a847a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1255405-8724-4977-854d-11f13d8a847a-workload-certs\") pod \"e1255405-8724-4977-854d-11f13d8a847a\" (UID: \"e1255405-8724-4977-854d-11f13d8a847a\") "
Oct 30 21:15:14.126407 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.126359    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1255405-8724-4977-854d-11f13d8a847a-credential-socket" pod "e1255405-8724-4977-854d-11f13d8a847a" (UID: "e1255405-8724-4977-854d-11f13d8a847a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:15:14.126461 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.126408    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1255405-8724-4977-854d-11f13d8a847a-kube-api-access-qf99v\" (UniqueName: \"kubernetes.io/projected/e1255405-8724-4977-854d-11f13d8a847a-kube-api-access-qf99v\") pod \"e1255405-8724-4977-854d-11f13d8a847a\" (UID: \"e1255405-8724-4977-854d-11f13d8a847a\") "
Oct 30 21:15:14.126521 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.126466    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1255405-8724-4977-854d-11f13d8a847a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1255405-8724-4977-854d-11f13d8a847a-istio-data\") pod \"e1255405-8724-4977-854d-11f13d8a847a\" (UID: \"e1255405-8724-4977-854d-11f13d8a847a\") "
Oct 30 21:15:14.126584 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.126564    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1255405-8724-4977-854d-11f13d8a847a-workload-certs" pod "e1255405-8724-4977-854d-11f13d8a847a" (UID: "e1255405-8724-4977-854d-11f13d8a847a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:15:14.126634 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.126578    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1255405-8724-4977-854d-11f13d8a847a-istiod-ca-cert" pod "e1255405-8724-4977-854d-11f13d8a847a" (UID: "e1255405-8724-4977-854d-11f13d8a847a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:14.126634 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.126587    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1255405-8724-4977-854d-11f13d8a847a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1255405-8724-4977-854d-11f13d8a847a-workload-socket\") pod \"e1255405-8724-4977-854d-11f13d8a847a\" (UID: \"e1255405-8724-4977-854d-11f13d8a847a\") "
Oct 30 21:15:14.126853 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.126831    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1255405-8724-4977-854d-11f13d8a847a-workload-socket" pod "e1255405-8724-4977-854d-11f13d8a847a" (UID: "e1255405-8724-4977-854d-11f13d8a847a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:15:14.127220 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.127193    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1255405-8724-4977-854d-11f13d8a847a-istio-data" pod "e1255405-8724-4977-854d-11f13d8a847a" (UID: "e1255405-8724-4977-854d-11f13d8a847a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:15:14.128045 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.128018    5713 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1255405-8724-4977-854d-11f13d8a847a-istiod-ca-cert\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:14.128155 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.128048    5713 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1255405-8724-4977-854d-11f13d8a847a-workload-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:14.128155 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.128063    5713 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1255405-8724-4977-854d-11f13d8a847a-istio-data\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:14.128155 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.128086    5713 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1255405-8724-4977-854d-11f13d8a847a-workload-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:14.128155 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.128100    5713 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1255405-8724-4977-854d-11f13d8a847a-credential-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:14.129022 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.128983    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1255405-8724-4977-854d-11f13d8a847a-istio-envoy" pod "e1255405-8724-4977-854d-11f13d8a847a" (UID: "e1255405-8724-4977-854d-11f13d8a847a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:15:14.129022 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.128983    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1255405-8724-4977-854d-11f13d8a847a-kube-api-access-qf99v" pod "e1255405-8724-4977-854d-11f13d8a847a" (UID: "e1255405-8724-4977-854d-11f13d8a847a"). InnerVolumeSpecName "kube-api-access-qf99v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:14.129202 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.129183    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1255405-8724-4977-854d-11f13d8a847a-istio-token" pod "e1255405-8724-4977-854d-11f13d8a847a" (UID: "e1255405-8724-4977-854d-11f13d8a847a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:14.129339 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.129320    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e1255405-8724-4977-854d-11f13d8a847a-istio-podinfo" pod "e1255405-8724-4977-854d-11f13d8a847a" (UID: "e1255405-8724-4977-854d-11f13d8a847a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:15:14.229578 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.229493    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3faef396-57dc-4ff5-af9f-c1f940cce913-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3faef396-57dc-4ff5-af9f-c1f940cce913-credential-socket\") pod \"3faef396-57dc-4ff5-af9f-c1f940cce913\" (UID: \"3faef396-57dc-4ff5-af9f-c1f940cce913\") "
Oct 30 21:15:14.229755 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.229598    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3faef396-57dc-4ff5-af9f-c1f940cce913-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3faef396-57dc-4ff5-af9f-c1f940cce913-workload-certs\") pod \"3faef396-57dc-4ff5-af9f-c1f940cce913\" (UID: \"3faef396-57dc-4ff5-af9f-c1f940cce913\") "
Oct 30 21:15:14.229755 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.229685    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3faef396-57dc-4ff5-af9f-c1f940cce913-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3faef396-57dc-4ff5-af9f-c1f940cce913-istiod-ca-cert\") pod \"3faef396-57dc-4ff5-af9f-c1f940cce913\" (UID: \"3faef396-57dc-4ff5-af9f-c1f940cce913\") "
Oct 30 21:15:14.229902 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.229759    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3faef396-57dc-4ff5-af9f-c1f940cce913-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3faef396-57dc-4ff5-af9f-c1f940cce913-istio-podinfo\") pod \"3faef396-57dc-4ff5-af9f-c1f940cce913\" (UID: \"3faef396-57dc-4ff5-af9f-c1f940cce913\") "
Oct 30 21:15:14.229902 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.229812    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3faef396-57dc-4ff5-af9f-c1f940cce913-istio-token\" (UniqueName: \"kubernetes.io/projected/3faef396-57dc-4ff5-af9f-c1f940cce913-istio-token\") pod \"3faef396-57dc-4ff5-af9f-c1f940cce913\" (UID: \"3faef396-57dc-4ff5-af9f-c1f940cce913\") "
Oct 30 21:15:14.230005 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.229901    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3faef396-57dc-4ff5-af9f-c1f940cce913-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3faef396-57dc-4ff5-af9f-c1f940cce913-istio-data\") pod \"3faef396-57dc-4ff5-af9f-c1f940cce913\" (UID: \"3faef396-57dc-4ff5-af9f-c1f940cce913\") "
Oct 30 21:15:14.230058 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.229995    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3faef396-57dc-4ff5-af9f-c1f940cce913-workload-certs" pod "3faef396-57dc-4ff5-af9f-c1f940cce913" (UID: "3faef396-57dc-4ff5-af9f-c1f940cce913"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:15:14.230120 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.230069    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3faef396-57dc-4ff5-af9f-c1f940cce913-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3faef396-57dc-4ff5-af9f-c1f940cce913-workload-socket\") pod \"3faef396-57dc-4ff5-af9f-c1f940cce913\" (UID: \"3faef396-57dc-4ff5-af9f-c1f940cce913\") "
Oct 30 21:15:14.230120 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.230100    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3faef396-57dc-4ff5-af9f-c1f940cce913-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3faef396-57dc-4ff5-af9f-c1f940cce913-istio-envoy\") pod \"3faef396-57dc-4ff5-af9f-c1f940cce913\" (UID: \"3faef396-57dc-4ff5-af9f-c1f940cce913\") "
Oct 30 21:15:14.230224 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.230158    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3faef396-57dc-4ff5-af9f-c1f940cce913-kube-api-access-ht7zn\" (UniqueName: \"kubernetes.io/projected/3faef396-57dc-4ff5-af9f-c1f940cce913-kube-api-access-ht7zn\") pod \"3faef396-57dc-4ff5-af9f-c1f940cce913\" (UID: \"3faef396-57dc-4ff5-af9f-c1f940cce913\") "
Oct 30 21:15:14.230508 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.230431    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3faef396-57dc-4ff5-af9f-c1f940cce913-workload-socket" pod "3faef396-57dc-4ff5-af9f-c1f940cce913" (UID: "3faef396-57dc-4ff5-af9f-c1f940cce913"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:15:14.230690 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.230664    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3faef396-57dc-4ff5-af9f-c1f940cce913-istio-data" pod "3faef396-57dc-4ff5-af9f-c1f940cce913" (UID: "3faef396-57dc-4ff5-af9f-c1f940cce913"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:15:14.231080 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.231038    5713 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1255405-8724-4977-854d-11f13d8a847a-istio-token\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:14.231404 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.231389    5713 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1255405-8724-4977-854d-11f13d8a847a-istio-envoy\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:14.231561 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.231544    5713 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3faef396-57dc-4ff5-af9f-c1f940cce913-istio-data\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:14.231717 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.231700    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qf99v\" (UniqueName: \"kubernetes.io/projected/e1255405-8724-4977-854d-11f13d8a847a-kube-api-access-qf99v\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:14.231926 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.231911    5713 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3faef396-57dc-4ff5-af9f-c1f940cce913-workload-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:14.232072 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.232053    5713 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1255405-8724-4977-854d-11f13d8a847a-istio-podinfo\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:14.232211 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.232197    5713 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3faef396-57dc-4ff5-af9f-c1f940cce913-workload-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:14.232346 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.231209    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3faef396-57dc-4ff5-af9f-c1f940cce913-istiod-ca-cert" pod "3faef396-57dc-4ff5-af9f-c1f940cce913" (UID: "3faef396-57dc-4ff5-af9f-c1f940cce913"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:14.232464 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.231259    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3faef396-57dc-4ff5-af9f-c1f940cce913-credential-socket" pod "3faef396-57dc-4ff5-af9f-c1f940cce913" (UID: "3faef396-57dc-4ff5-af9f-c1f940cce913"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:15:14.233089 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.233050    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3faef396-57dc-4ff5-af9f-c1f940cce913-istio-envoy" pod "3faef396-57dc-4ff5-af9f-c1f940cce913" (UID: "3faef396-57dc-4ff5-af9f-c1f940cce913"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:15:14.233179 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.233155    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3faef396-57dc-4ff5-af9f-c1f940cce913-kube-api-access-ht7zn" pod "3faef396-57dc-4ff5-af9f-c1f940cce913" (UID: "3faef396-57dc-4ff5-af9f-c1f940cce913"). InnerVolumeSpecName "kube-api-access-ht7zn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:14.233382 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.233354    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3faef396-57dc-4ff5-af9f-c1f940cce913-istio-token" pod "3faef396-57dc-4ff5-af9f-c1f940cce913" (UID: "3faef396-57dc-4ff5-af9f-c1f940cce913"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:14.233663 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.233634    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3faef396-57dc-4ff5-af9f-c1f940cce913-istio-podinfo" pod "3faef396-57dc-4ff5-af9f-c1f940cce913" (UID: "3faef396-57dc-4ff5-af9f-c1f940cce913"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:15:14.334326 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.334294    5713 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3faef396-57dc-4ff5-af9f-c1f940cce913-istio-token\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:14.334326 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.334325    5713 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3faef396-57dc-4ff5-af9f-c1f940cce913-istio-envoy\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:14.334326 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.334336    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ht7zn\" (UniqueName: \"kubernetes.io/projected/3faef396-57dc-4ff5-af9f-c1f940cce913-kube-api-access-ht7zn\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:14.334571 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.334347    5713 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3faef396-57dc-4ff5-af9f-c1f940cce913-credential-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:14.334571 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.334356    5713 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3faef396-57dc-4ff5-af9f-c1f940cce913-istiod-ca-cert\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:14.334571 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.334365    5713 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3faef396-57dc-4ff5-af9f-c1f940cce913-istio-podinfo\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:14.429797 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.429747    5713 generic.go:356] "Generic (PLEG): container finished" podID="e1255405-8724-4977-854d-11f13d8a847a" containerID="eceac2f023bbcb511f2a9a8ce3078ed19b64ce3e11aa5701a7509788dcc1b859" exitCode=0
Oct 30 21:15:14.430001 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.429833    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2"
Oct 30 21:15:14.430001 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.429855    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2" event={"ID":"e1255405-8724-4977-854d-11f13d8a847a","Type":"ContainerDied","Data":"eceac2f023bbcb511f2a9a8ce3078ed19b64ce3e11aa5701a7509788dcc1b859"}
Oct 30 21:15:14.430001 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.429920    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2" event={"ID":"e1255405-8724-4977-854d-11f13d8a847a","Type":"ContainerDied","Data":"71765f9523506cf9462cf503da06c5129da2a0b5a13074bd125a1b0e32d1c723"}
Oct 30 21:15:14.430001 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.429949    5713 scope.go:122] "RemoveContainer" containerID="eceac2f023bbcb511f2a9a8ce3078ed19b64ce3e11aa5701a7509788dcc1b859"
Oct 30 21:15:14.431605 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.431578    5713 generic.go:356] "Generic (PLEG): container finished" podID="3faef396-57dc-4ff5-af9f-c1f940cce913" containerID="6c3f052bb8ec591c4c5dff555fbca6d71e69e33f79582ba339fde1b02474b8dc" exitCode=0
Oct 30 21:15:14.431721 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.431621    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2"
Oct 30 21:15:14.431721 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.431656    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2" event={"ID":"3faef396-57dc-4ff5-af9f-c1f940cce913","Type":"ContainerDied","Data":"6c3f052bb8ec591c4c5dff555fbca6d71e69e33f79582ba339fde1b02474b8dc"}
Oct 30 21:15:14.431721 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.431699    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2" event={"ID":"3faef396-57dc-4ff5-af9f-c1f940cce913","Type":"ContainerDied","Data":"44befe62a1c3aabdf6a96543f00f139f50911c9443099a8220cfd44c0384c4ed"}
Oct 30 21:15:14.450740 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.450721    5713 scope.go:122] "RemoveContainer" containerID="eceac2f023bbcb511f2a9a8ce3078ed19b64ce3e11aa5701a7509788dcc1b859"
Oct 30 21:15:14.451133 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:15:14.451112    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eceac2f023bbcb511f2a9a8ce3078ed19b64ce3e11aa5701a7509788dcc1b859\": container with ID starting with eceac2f023bbcb511f2a9a8ce3078ed19b64ce3e11aa5701a7509788dcc1b859 not found: ID does not exist" containerID="eceac2f023bbcb511f2a9a8ce3078ed19b64ce3e11aa5701a7509788dcc1b859"
Oct 30 21:15:14.451223 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.451142    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eceac2f023bbcb511f2a9a8ce3078ed19b64ce3e11aa5701a7509788dcc1b859"} err="failed to get container status \"eceac2f023bbcb511f2a9a8ce3078ed19b64ce3e11aa5701a7509788dcc1b859\": rpc error: code = NotFound desc = could not find container \"eceac2f023bbcb511f2a9a8ce3078ed19b64ce3e11aa5701a7509788dcc1b859\": container with ID starting with eceac2f023bbcb511f2a9a8ce3078ed19b64ce3e11aa5701a7509788dcc1b859 not found: ID does not exist"
Oct 30 21:15:14.451223 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.451161    5713 scope.go:122] "RemoveContainer" containerID="6c3f052bb8ec591c4c5dff555fbca6d71e69e33f79582ba339fde1b02474b8dc"
Oct 30 21:15:14.461780 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.461747    5713 scope.go:122] "RemoveContainer" containerID="6c3f052bb8ec591c4c5dff555fbca6d71e69e33f79582ba339fde1b02474b8dc"
Oct 30 21:15:14.462108 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:15:14.462085    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c3f052bb8ec591c4c5dff555fbca6d71e69e33f79582ba339fde1b02474b8dc\": container with ID starting with 6c3f052bb8ec591c4c5dff555fbca6d71e69e33f79582ba339fde1b02474b8dc not found: ID does not exist" containerID="6c3f052bb8ec591c4c5dff555fbca6d71e69e33f79582ba339fde1b02474b8dc"
Oct 30 21:15:14.462191 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.462120    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c3f052bb8ec591c4c5dff555fbca6d71e69e33f79582ba339fde1b02474b8dc"} err="failed to get container status \"6c3f052bb8ec591c4c5dff555fbca6d71e69e33f79582ba339fde1b02474b8dc\": rpc error: code = NotFound desc = could not find container \"6c3f052bb8ec591c4c5dff555fbca6d71e69e33f79582ba339fde1b02474b8dc\": container with ID starting with 6c3f052bb8ec591c4c5dff555fbca6d71e69e33f79582ba339fde1b02474b8dc not found: ID does not exist"
Oct 30 21:15:14.598571 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.598487    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2"]
Oct 30 21:15:14.605224 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.605190    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-682999-openshift-default-54dcb797b5-kxbh2"]
Oct 30 21:15:14.717023 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.716983    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2"]
Oct 30 21:15:14.720921 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:14.720897    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-lshp2"]
Oct 30 21:15:15.065003 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:15.064970    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3faef396-57dc-4ff5-af9f-c1f940cce913" path="/var/lib/kubelet/pods/3faef396-57dc-4ff5-af9f-c1f940cce913/volumes"
Oct 30 21:15:15.065528 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:15.065510    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1255405-8724-4977-854d-11f13d8a847a" path="/var/lib/kubelet/pods/e1255405-8724-4977-854d-11f13d8a847a/volumes"
Oct 30 21:15:16.056399 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:16.056345    5713 scope.go:122] "RemoveContainer" containerID="c73222e686d9cbe7e8a0e317f04fda67016d298f2efd4011acc282c5436b6914"
Oct 30 21:15:16.056590 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:16.056513    5713 scope.go:122] "RemoveContainer" containerID="3bacf903345540117b3aebca7ce72028ef5721b2232c813deb2527a119bd6c6e"
Oct 30 21:15:16.056710 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:16.056689    5713 scope.go:122] "RemoveContainer" containerID="325c0a83c19298df8512df9f26e9f9fed0e2bbf3b8d23d506f9c662004ad974b"
Oct 30 21:15:16.056875 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:16.056855    5713 scope.go:122] "RemoveContainer" containerID="28283d6f522f6c008824c593f00cc1895db9b58ae0a7780388f27608a28a6ed3"
Oct 30 21:15:16.057012 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:16.056989    5713 scope.go:122] "RemoveContainer" containerID="e63727cab85802becf3e0a803256dfb821b9d6137db5f155a3462f6c405e5405"
Oct 30 21:15:17.119310 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:17.119219    5713 scope.go:122] "RemoveContainer" containerID="e762675006598988b9be035a8b5eb41d1873843016136c9f1561f983a7aadcb3"
Oct 30 21:15:17.467225 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:17.467197    5713 generic.go:356] "Generic (PLEG): container finished" podID="3543ab8c-8863-4207-ae05-d8094dbd8cb9" containerID="0c1960c85908f3d1986cc874eb9acdb6938e69b1860d2d991a64fe40c4f4c0e5" exitCode=60
Oct 30 21:15:17.467339 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:17.467271    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889915-x55nz" event={"ID":"3543ab8c-8863-4207-ae05-d8094dbd8cb9","Type":"ContainerDied","Data":"0c1960c85908f3d1986cc874eb9acdb6938e69b1860d2d991a64fe40c4f4c0e5"}
Oct 30 21:15:17.467390 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:17.467346    5713 scope.go:122] "RemoveContainer" containerID="28283d6f522f6c008824c593f00cc1895db9b58ae0a7780388f27608a28a6ed3"
Oct 30 21:15:17.467799 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:17.467761    5713 scope.go:122] "RemoveContainer" containerID="0c1960c85908f3d1986cc874eb9acdb6938e69b1860d2d991a64fe40c4f4c0e5"
Oct 30 21:15:17.468196 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:15:17.468173    5713 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889915-x55nz_odh-ai-gateway-infra(3543ab8c-8863-4207-ae05-d8094dbd8cb9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889915-x55nz" podUID="3543ab8c-8863-4207-ae05-d8094dbd8cb9"
Oct 30 21:15:17.469478 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:17.469457    5713 generic.go:356] "Generic (PLEG): container finished" podID="c2184c91-9d42-4a01-bf05-dd962ff62dcb" containerID="91d96c9ed2fe3cb7c1a61e7d1eb75aed69a366d4d48b96e67782fbceb96ae245" exitCode=60
Oct 30 21:15:17.469577 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:17.469534    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-27nkg" event={"ID":"c2184c91-9d42-4a01-bf05-dd962ff62dcb","Type":"ContainerDied","Data":"91d96c9ed2fe3cb7c1a61e7d1eb75aed69a366d4d48b96e67782fbceb96ae245"}
Oct 30 21:15:17.470014 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:17.469992    5713 scope.go:122] "RemoveContainer" containerID="91d96c9ed2fe3cb7c1a61e7d1eb75aed69a366d4d48b96e67782fbceb96ae245"
Oct 30 21:15:17.470379 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:15:17.470354    5713 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889915-27nkg_odh-ai-gateway-infra(c2184c91-9d42-4a01-bf05-dd962ff62dcb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-27nkg" podUID="c2184c91-9d42-4a01-bf05-dd962ff62dcb"
Oct 30 21:15:17.471527 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:17.471509    5713 generic.go:356] "Generic (PLEG): container finished" podID="a8d5eebe-b80b-4b99-ace8-7e87724decaf" containerID="911dffa62ed5139836fa0404ac197b629df89792b298f44e94be70017f5783f4" exitCode=60
Oct 30 21:15:17.471611 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:17.471558    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cb6404-29889915-4c7lb" event={"ID":"a8d5eebe-b80b-4b99-ace8-7e87724decaf","Type":"ContainerDied","Data":"911dffa62ed5139836fa0404ac197b629df89792b298f44e94be70017f5783f4"}
Oct 30 21:15:17.472006 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:17.471978    5713 scope.go:122] "RemoveContainer" containerID="911dffa62ed5139836fa0404ac197b629df89792b298f44e94be70017f5783f4"
Oct 30 21:15:17.472336 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:15:17.472316    5713 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-cb6404-29889915-4c7lb_odh-ai-gateway-infra(a8d5eebe-b80b-4b99-ace8-7e87724decaf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cb6404-29889915-4c7lb" podUID="a8d5eebe-b80b-4b99-ace8-7e87724decaf"
Oct 30 21:15:17.475255 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:17.475236    5713 generic.go:356] "Generic (PLEG): container finished" podID="aa46459a-d46e-4af4-bdfc-d6872367429d" containerID="bbffa1125ea3fa908cfd36a98aaf846dc6ea736190a30bf2185eaaec85c73841" exitCode=60
Oct 30 21:15:17.475333 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:17.475306    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-390b71-29889915-lrfqc" event={"ID":"aa46459a-d46e-4af4-bdfc-d6872367429d","Type":"ContainerDied","Data":"bbffa1125ea3fa908cfd36a98aaf846dc6ea736190a30bf2185eaaec85c73841"}
Oct 30 21:15:17.475653 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:17.475633    5713 scope.go:122] "RemoveContainer" containerID="bbffa1125ea3fa908cfd36a98aaf846dc6ea736190a30bf2185eaaec85c73841"
Oct 30 21:15:17.475939 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:15:17.475917    5713 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-390b71-29889915-lrfqc_odh-ai-gateway-infra(aa46459a-d46e-4af4-bdfc-d6872367429d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-390b71-29889915-lrfqc" podUID="aa46459a-d46e-4af4-bdfc-d6872367429d"
Oct 30 21:15:17.477286 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:17.477262    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f9eff-29889915-wpmls" event={"ID":"f59ff9d3-2927-4021-9103-fd2f60f7b655","Type":"ContainerStarted","Data":"b261d33b47b44922f13d648a2a1c35fb051bdc37fcdc010573c7f2998a609ad9"}
Oct 30 21:15:17.720849 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:17.720826    5713 scope.go:122] "RemoveContainer" containerID="3bacf903345540117b3aebca7ce72028ef5721b2232c813deb2527a119bd6c6e"
Oct 30 21:15:17.739503 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:17.739447    5713 scope.go:122] "RemoveContainer" containerID="325c0a83c19298df8512df9f26e9f9fed0e2bbf3b8d23d506f9c662004ad974b"
Oct 30 21:15:17.754674 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:17.754647    5713 scope.go:122] "RemoveContainer" containerID="e63727cab85802becf3e0a803256dfb821b9d6137db5f155a3462f6c405e5405"
Oct 30 21:15:18.094902 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:18.067532    5713 scope.go:122] "RemoveContainer" containerID="12f1188b26fa4dca1c4bcf439a3088825c79c2492bf68230d6fc52dd0dccfca6"
Oct 30 21:15:18.094902 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:18.069220    5713 scope.go:122] "RemoveContainer" containerID="5e9f0ad8e8e97cc8c62b1fbf9772c971c576de7112bcd9155d331bcc45d1f797"
Oct 30 21:15:18.494312 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:18.494281    5713 generic.go:356] "Generic (PLEG): container finished" podID="f59ff9d3-2927-4021-9103-fd2f60f7b655" containerID="b261d33b47b44922f13d648a2a1c35fb051bdc37fcdc010573c7f2998a609ad9" exitCode=60
Oct 30 21:15:18.494818 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:18.494348    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f9eff-29889915-wpmls" event={"ID":"f59ff9d3-2927-4021-9103-fd2f60f7b655","Type":"ContainerDied","Data":"b261d33b47b44922f13d648a2a1c35fb051bdc37fcdc010573c7f2998a609ad9"}
Oct 30 21:15:18.494818 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:18.494407    5713 scope.go:122] "RemoveContainer" containerID="c73222e686d9cbe7e8a0e317f04fda67016d298f2efd4011acc282c5436b6914"
Oct 30 21:15:18.495006 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:18.494890    5713 scope.go:122] "RemoveContainer" containerID="b261d33b47b44922f13d648a2a1c35fb051bdc37fcdc010573c7f2998a609ad9"
Oct 30 21:15:18.495310 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:15:18.495281    5713 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-2f9eff-29889915-wpmls_odh-ai-gateway-infra(f59ff9d3-2927-4021-9103-fd2f60f7b655)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f9eff-29889915-wpmls" podUID="f59ff9d3-2927-4021-9103-fd2f60f7b655"
Oct 30 21:15:18.496752 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:18.496729    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5c45e5-29889915-zm4fd" event={"ID":"c3a522bc-746d-4cdf-940d-ccb562ade3c0","Type":"ContainerStarted","Data":"17433a2005cefd7c24eea71d6cde4729703f077f7f42bfc06c421df981786e6b"}
Oct 30 21:15:18.608837 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:18.608798    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889915-x55nz"]
Oct 30 21:15:18.774582 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:18.774261    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-390b71-29889915-lrfqc"]
Oct 30 21:15:18.852121 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:18.852006    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889915-x55nz"
Oct 30 21:15:18.893850 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:18.893758    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5c45e5-29889915-zm4fd" podStartSLOduration=18.893736542 podStartE2EDuration="18.893736542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:18.875066669 +0000 UTC m=+2201.939843070" watchObservedRunningTime="2026-10-30 21:15:18.893736542 +0000 UTC m=+2201.958512940"
Oct 30 21:15:18.902664 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:18.902628    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cb6404-29889915-4c7lb"]
Oct 30 21:15:18.923422 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:18.923008    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3543ab8c-8863-4207-ae05-d8094dbd8cb9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3543ab8c-8863-4207-ae05-d8094dbd8cb9-maas-api-service-ca\") pod \"3543ab8c-8863-4207-ae05-d8094dbd8cb9\" (UID: \"3543ab8c-8863-4207-ae05-d8094dbd8cb9\") "
Oct 30 21:15:18.923422 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:18.923133    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3543ab8c-8863-4207-ae05-d8094dbd8cb9-kube-api-access-wmd6j\" (UniqueName: \"kubernetes.io/projected/3543ab8c-8863-4207-ae05-d8094dbd8cb9-kube-api-access-wmd6j\") pod \"3543ab8c-8863-4207-ae05-d8094dbd8cb9\" (UID: \"3543ab8c-8863-4207-ae05-d8094dbd8cb9\") "
Oct 30 21:15:18.927151 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:18.927090    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3543ab8c-8863-4207-ae05-d8094dbd8cb9-maas-api-service-ca" pod "3543ab8c-8863-4207-ae05-d8094dbd8cb9" (UID: "3543ab8c-8863-4207-ae05-d8094dbd8cb9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:18.968162 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:18.967478    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3543ab8c-8863-4207-ae05-d8094dbd8cb9-kube-api-access-wmd6j" pod "3543ab8c-8863-4207-ae05-d8094dbd8cb9" (UID: "3543ab8c-8863-4207-ae05-d8094dbd8cb9"). InnerVolumeSpecName "kube-api-access-wmd6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:19.009634 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:19.009598    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-390b71-29889915-lrfqc"
Oct 30 21:15:19.026607 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:19.026517    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3543ab8c-8863-4207-ae05-d8094dbd8cb9-maas-api-service-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:19.026607 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:19.026544    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmd6j\" (UniqueName: \"kubernetes.io/projected/3543ab8c-8863-4207-ae05-d8094dbd8cb9-kube-api-access-wmd6j\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:19.091674 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:19.080344    5713 scope.go:122] "RemoveContainer" containerID="275326fe0b138876b3a03f1e9b84e7548fbdd44053ffde8667c1cebefb86a2c6"
Oct 30 21:15:19.091674 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:19.082465    5713 scope.go:122] "RemoveContainer" containerID="7fe97e301ee843caf143d1182cefe60b87fed3180e56faf490513a7213b71597"
Oct 30 21:15:19.116123 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:19.116095    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f9eff-29889915-wpmls"]
Oct 30 21:15:19.128998 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:19.127560    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa46459a-d46e-4af4-bdfc-d6872367429d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa46459a-d46e-4af4-bdfc-d6872367429d-maas-api-service-ca\") pod \"aa46459a-d46e-4af4-bdfc-d6872367429d\" (UID: \"aa46459a-d46e-4af4-bdfc-d6872367429d\") "
Oct 30 21:15:19.128998 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:19.127766    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa46459a-d46e-4af4-bdfc-d6872367429d-kube-api-access-2hwcs\" (UniqueName: \"kubernetes.io/projected/aa46459a-d46e-4af4-bdfc-d6872367429d-kube-api-access-2hwcs\") pod \"aa46459a-d46e-4af4-bdfc-d6872367429d\" (UID: \"aa46459a-d46e-4af4-bdfc-d6872367429d\") "
Oct 30 21:15:19.129866 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:19.129344    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa46459a-d46e-4af4-bdfc-d6872367429d-maas-api-service-ca" pod "aa46459a-d46e-4af4-bdfc-d6872367429d" (UID: "aa46459a-d46e-4af4-bdfc-d6872367429d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:19.131760 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:19.131738    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa46459a-d46e-4af4-bdfc-d6872367429d-kube-api-access-2hwcs" pod "aa46459a-d46e-4af4-bdfc-d6872367429d" (UID: "aa46459a-d46e-4af4-bdfc-d6872367429d"). InnerVolumeSpecName "kube-api-access-2hwcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:19.204299 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:19.204273    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cb6404-29889915-4c7lb"
Oct 30 21:15:19.206574 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:19.206557    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889915-27nkg"]
Oct 30 21:15:19.230695 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:19.229437    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8d5eebe-b80b-4b99-ace8-7e87724decaf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8d5eebe-b80b-4b99-ace8-7e87724decaf-maas-api-service-ca\") pod \"a8d5eebe-b80b-4b99-ace8-7e87724decaf\" (UID: \"a8d5eebe-b80b-4b99-ace8-7e87724decaf\") "
Oct 30 21:15:19.230695 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:19.229650    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8d5eebe-b80b-4b99-ace8-7e87724decaf-kube-api-access-2vsmv\" (UniqueName: \"kubernetes.io/projected/a8d5eebe-b80b-4b99-ace8-7e87724decaf-kube-api-access-2vsmv\") pod \"a8d5eebe-b80b-4b99-ace8-7e87724decaf\" (UID: \"a8d5eebe-b80b-4b99-ace8-7e87724decaf\") "
Oct 30 21:15:19.230695 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:19.230123    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hwcs\" (UniqueName: \"kubernetes.io/projected/aa46459a-d46e-4af4-bdfc-d6872367429d-kube-api-access-2hwcs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:19.230695 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:19.230137    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa46459a-d46e-4af4-bdfc-d6872367429d-maas-api-service-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:19.231091 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:19.230821    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8d5eebe-b80b-4b99-ace8-7e87724decaf-maas-api-service-ca" pod "a8d5eebe-b80b-4b99-ace8-7e87724decaf" (UID: "a8d5eebe-b80b-4b99-ace8-7e87724decaf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:19.232870 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:19.232830    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8d5eebe-b80b-4b99-ace8-7e87724decaf-kube-api-access-2vsmv" pod "a8d5eebe-b80b-4b99-ace8-7e87724decaf" (UID: "a8d5eebe-b80b-4b99-ace8-7e87724decaf"). InnerVolumeSpecName "kube-api-access-2vsmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:19.331716 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:19.331687    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8d5eebe-b80b-4b99-ace8-7e87724decaf-maas-api-service-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:19.331989 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:19.331959    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vsmv\" (UniqueName: \"kubernetes.io/projected/a8d5eebe-b80b-4b99-ace8-7e87724decaf-kube-api-access-2vsmv\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:19.511421 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:19.511389    5713 generic.go:356] "Generic (PLEG): container finished" podID="b5117983-c7b5-4679-8c1d-90ea13ebca99" containerID="28b55757ef5d4f15ed47a6c8d6f717044502f253e26d2a67cc36fe8a9dfe8b5a" exitCode=60
Oct 30 21:15:19.511853 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:19.511476    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-677b45-29889915-4m4cj" event={"ID":"b5117983-c7b5-4679-8c1d-90ea13ebca99","Type":"ContainerDied","Data":"28b55757ef5d4f15ed47a6c8d6f717044502f253e26d2a67cc36fe8a9dfe8b5a"}
Oct 30 21:15:19.511853 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:19.511563    5713 scope.go:122] "RemoveContainer" containerID="12f1188b26fa4dca1c4bcf439a3088825c79c2492bf68230d6fc52dd0dccfca6"
Oct 30 21:15:19.512010 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:19.511993    5713 scope.go:122] "RemoveContainer" containerID="28b55757ef5d4f15ed47a6c8d6f717044502f253e26d2a67cc36fe8a9dfe8b5a"
Oct 30 21:15:19.512395 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:15:19.512364    5713 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-677b45-29889915-4m4cj_odh-ai-gateway-infra(b5117983-c7b5-4679-8c1d-90ea13ebca99)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-677b45-29889915-4m4cj" podUID="b5117983-c7b5-4679-8c1d-90ea13ebca99"
Oct 30 21:15:19.513780 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:19.513761    5713 generic.go:356] "Generic (PLEG): container finished" podID="16adca55-5cc2-4452-aba7-3e891c005f98" containerID="b6add717ab64b5ac643e2852e41b13d3bb7cee241f8ba64699fa0b05d5b33e26" exitCode=60
Oct 30 21:15:19.513888 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:19.513835    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88da25-29889915-gf4hb" event={"ID":"16adca55-5cc2-4452-aba7-3e891c005f98","Type":"ContainerDied","Data":"b6add717ab64b5ac643e2852e41b13d3bb7cee241f8ba64699fa0b05d5b33e26"}
Oct 30 21:15:19.514265 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:19.514203    5713 scope.go:122] "RemoveContainer" containerID="b6add717ab64b5ac643e2852e41b13d3bb7cee241f8ba64699fa0b05d5b33e26"
Oct 30 21:15:19.514506 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:15:19.514484    5713 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-88da25-29889915-gf4hb_odh-ai-gateway-infra(16adca55-5cc2-4452-aba7-3e891c005f98)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88da25-29889915-gf4hb" podUID="16adca55-5cc2-4452-aba7-3e891c005f98"
Oct 30 21:15:19.515500 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:19.515480    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889915-x55nz"
Oct 30 21:15:19.515500 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:19.515489    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889915-x55nz" event={"ID":"3543ab8c-8863-4207-ae05-d8094dbd8cb9","Type":"ContainerDied","Data":"dc2306e64d1ea3c84659438a4bb596720f12cc68d173b08febb55acae0d5be4a"}
Oct 30 21:15:19.516986 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:19.516960    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-27nkg" event={"ID":"c2184c91-9d42-4a01-bf05-dd962ff62dcb","Type":"ContainerDied","Data":"b6d41c6ded9e476f1e7180424f60c133aec38863f50fbb493a3af1bc399cce98"}
Oct 30 21:15:19.517070 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:19.517005    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6d41c6ded9e476f1e7180424f60c133aec38863f50fbb493a3af1bc399cce98"
Oct 30 21:15:19.518532 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:19.518513    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cb6404-29889915-4c7lb"
Oct 30 21:15:19.518638 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:19.518533    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cb6404-29889915-4c7lb" event={"ID":"a8d5eebe-b80b-4b99-ace8-7e87724decaf","Type":"ContainerDied","Data":"4e93f8fd35297c0ef333120c31848be4a64e950f17652c6f656ed4079834e923"}
Oct 30 21:15:19.520370 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:19.520018    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-390b71-29889915-lrfqc"
Oct 30 21:15:19.520370 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:19.520017    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-390b71-29889915-lrfqc" event={"ID":"aa46459a-d46e-4af4-bdfc-d6872367429d","Type":"ContainerDied","Data":"81233ce901a7f640950fc311323c949ba9052256ac34b6439a30f14d12502d7c"}
Oct 30 21:15:19.526994 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:19.526969    5713 generic.go:356] "Generic (PLEG): container finished" podID="c3a522bc-746d-4cdf-940d-ccb562ade3c0" containerID="17433a2005cefd7c24eea71d6cde4729703f077f7f42bfc06c421df981786e6b" exitCode=60
Oct 30 21:15:19.527098 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:19.527038    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5c45e5-29889915-zm4fd" event={"ID":"c3a522bc-746d-4cdf-940d-ccb562ade3c0","Type":"ContainerDied","Data":"17433a2005cefd7c24eea71d6cde4729703f077f7f42bfc06c421df981786e6b"}
Oct 30 21:15:19.527455 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:19.527436    5713 scope.go:122] "RemoveContainer" containerID="17433a2005cefd7c24eea71d6cde4729703f077f7f42bfc06c421df981786e6b"
Oct 30 21:15:19.527825 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:15:19.527765    5713 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-5c45e5-29889915-zm4fd_odh-ai-gateway-infra(c3a522bc-746d-4cdf-940d-ccb562ade3c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5c45e5-29889915-zm4fd" podUID="c3a522bc-746d-4cdf-940d-ccb562ade3c0"
Oct 30 21:15:19.533469 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:19.533446    5713 scope.go:122] "RemoveContainer" containerID="5e9f0ad8e8e97cc8c62b1fbf9772c971c576de7112bcd9155d331bcc45d1f797"
Oct 30 21:15:19.560822 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:19.558993    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-27nkg"
Oct 30 21:15:19.586476 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:19.586250    5713 scope.go:122] "RemoveContainer" containerID="0c1960c85908f3d1986cc874eb9acdb6938e69b1860d2d991a64fe40c4f4c0e5"
Oct 30 21:15:19.600014 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:19.599978    5713 scope.go:122] "RemoveContainer" containerID="911dffa62ed5139836fa0404ac197b629df89792b298f44e94be70017f5783f4"
Oct 30 21:15:19.612124 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:19.612100    5713 scope.go:122] "RemoveContainer" containerID="bbffa1125ea3fa908cfd36a98aaf846dc6ea736190a30bf2185eaaec85c73841"
Oct 30 21:15:19.623573 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:19.623535    5713 scope.go:122] "RemoveContainer" containerID="e762675006598988b9be035a8b5eb41d1873843016136c9f1561f983a7aadcb3"
Oct 30 21:15:19.666232 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:19.666198    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2184c91-9d42-4a01-bf05-dd962ff62dcb-kube-api-access-tmmqr\" (UniqueName: \"kubernetes.io/projected/c2184c91-9d42-4a01-bf05-dd962ff62dcb-kube-api-access-tmmqr\") pod \"c2184c91-9d42-4a01-bf05-dd962ff62dcb\" (UID: \"c2184c91-9d42-4a01-bf05-dd962ff62dcb\") "
Oct 30 21:15:19.666442 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:19.666414    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2184c91-9d42-4a01-bf05-dd962ff62dcb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2184c91-9d42-4a01-bf05-dd962ff62dcb-maas-api-service-ca\") pod \"c2184c91-9d42-4a01-bf05-dd962ff62dcb\" (UID: \"c2184c91-9d42-4a01-bf05-dd962ff62dcb\") "
Oct 30 21:15:19.669169 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:19.669110    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2184c91-9d42-4a01-bf05-dd962ff62dcb-maas-api-service-ca" pod "c2184c91-9d42-4a01-bf05-dd962ff62dcb" (UID: "c2184c91-9d42-4a01-bf05-dd962ff62dcb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:19.671819 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:19.671771    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2184c91-9d42-4a01-bf05-dd962ff62dcb-kube-api-access-tmmqr" pod "c2184c91-9d42-4a01-bf05-dd962ff62dcb" (UID: "c2184c91-9d42-4a01-bf05-dd962ff62dcb"). InnerVolumeSpecName "kube-api-access-tmmqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:19.769491 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:19.769452    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmmqr\" (UniqueName: \"kubernetes.io/projected/c2184c91-9d42-4a01-bf05-dd962ff62dcb-kube-api-access-tmmqr\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:19.769491 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:19.769491    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2184c91-9d42-4a01-bf05-dd962ff62dcb-maas-api-service-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:19.774730 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:19.774708    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f9eff-29889915-wpmls"
Oct 30 21:15:19.870437 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:19.870344    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f59ff9d3-2927-4021-9103-fd2f60f7b655-kube-api-access-9pm9b\" (UniqueName: \"kubernetes.io/projected/f59ff9d3-2927-4021-9103-fd2f60f7b655-kube-api-access-9pm9b\") pod \"f59ff9d3-2927-4021-9103-fd2f60f7b655\" (UID: \"f59ff9d3-2927-4021-9103-fd2f60f7b655\") "
Oct 30 21:15:19.870611 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:19.870511    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f59ff9d3-2927-4021-9103-fd2f60f7b655-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f59ff9d3-2927-4021-9103-fd2f60f7b655-maas-api-service-ca\") pod \"f59ff9d3-2927-4021-9103-fd2f60f7b655\" (UID: \"f59ff9d3-2927-4021-9103-fd2f60f7b655\") "
Oct 30 21:15:19.871170 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:19.871135    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f59ff9d3-2927-4021-9103-fd2f60f7b655-maas-api-service-ca" pod "f59ff9d3-2927-4021-9103-fd2f60f7b655" (UID: "f59ff9d3-2927-4021-9103-fd2f60f7b655"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:19.873729 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:19.873704    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f59ff9d3-2927-4021-9103-fd2f60f7b655-kube-api-access-9pm9b" pod "f59ff9d3-2927-4021-9103-fd2f60f7b655" (UID: "f59ff9d3-2927-4021-9103-fd2f60f7b655"). InnerVolumeSpecName "kube-api-access-9pm9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:19.881782 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:19.881758    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5c45e5-29889915-zm4fd"]
Oct 30 21:15:19.960823 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:19.955838    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-390b71-29889915-lrfqc"]
Oct 30 21:15:19.966055 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:19.965564    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-390b71-29889915-lrfqc"]
Oct 30 21:15:19.972463 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:19.972431    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pm9b\" (UniqueName: \"kubernetes.io/projected/f59ff9d3-2927-4021-9103-fd2f60f7b655-kube-api-access-9pm9b\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:19.972463 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:19.972462    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f59ff9d3-2927-4021-9103-fd2f60f7b655-maas-api-service-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:20.067913 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:20.067689    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cb6404-29889915-4c7lb"]
Oct 30 21:15:20.075317 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:20.075287    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-cb6404-29889915-4c7lb"]
Oct 30 21:15:20.295729 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:20.295659    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889915-x55nz"]
Oct 30 21:15:20.299858 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:20.299812    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8fa2a7-29889915-x55nz"]
Oct 30 21:15:20.565206 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:20.565143    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f9eff-29889915-wpmls" event={"ID":"f59ff9d3-2927-4021-9103-fd2f60f7b655","Type":"ContainerDied","Data":"e640375006d4cd8fafb9696a400e714f28a24d4083b888e30bdb10ec10c918e3"}
Oct 30 21:15:20.565852 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:20.565216    5713 scope.go:122] "RemoveContainer" containerID="b261d33b47b44922f13d648a2a1c35fb051bdc37fcdc010573c7f2998a609ad9"
Oct 30 21:15:20.565852 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:20.565235    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f9eff-29889915-wpmls"
Oct 30 21:15:20.571219 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:20.571166    5713 generic.go:356] "Generic (PLEG): container finished" podID="126910b1-789a-4e00-9e84-2f0a9c899abb" containerID="3fc56164212a377e3865a8d2ca10a82743ca5d0851e87d639f6df2c883bf1e33" exitCode=60
Oct 30 21:15:20.571538 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:20.571436    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-682999-29889915-vzbj5" event={"ID":"126910b1-789a-4e00-9e84-2f0a9c899abb","Type":"ContainerDied","Data":"3fc56164212a377e3865a8d2ca10a82743ca5d0851e87d639f6df2c883bf1e33"}
Oct 30 21:15:20.572642 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:20.572406    5713 scope.go:122] "RemoveContainer" containerID="3fc56164212a377e3865a8d2ca10a82743ca5d0851e87d639f6df2c883bf1e33"
Oct 30 21:15:20.572948 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:15:20.572892    5713 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-682999-29889915-vzbj5_odh-ai-gateway-infra(126910b1-789a-4e00-9e84-2f0a9c899abb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-682999-29889915-vzbj5" podUID="126910b1-789a-4e00-9e84-2f0a9c899abb"
Oct 30 21:15:20.576342 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:20.576318    5713 generic.go:356] "Generic (PLEG): container finished" podID="9970e7c4-7661-47c5-a6df-4de2bea436e2" containerID="08f4604e94056363333e54f16bd2a34389e65f3ad6c912d64ad5915b6ec5e356" exitCode=60
Oct 30 21:15:20.576482 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:20.576443    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f9f277-29889915-dbd9b" event={"ID":"9970e7c4-7661-47c5-a6df-4de2bea436e2","Type":"ContainerDied","Data":"08f4604e94056363333e54f16bd2a34389e65f3ad6c912d64ad5915b6ec5e356"}
Oct 30 21:15:20.576863 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:20.576844    5713 scope.go:122] "RemoveContainer" containerID="08f4604e94056363333e54f16bd2a34389e65f3ad6c912d64ad5915b6ec5e356"
Oct 30 21:15:20.577191 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:15:20.577170    5713 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-f9f277-29889915-dbd9b_odh-ai-gateway-infra(9970e7c4-7661-47c5-a6df-4de2bea436e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f9f277-29889915-dbd9b" podUID="9970e7c4-7661-47c5-a6df-4de2bea436e2"
Oct 30 21:15:20.584555 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:20.584533    5713 scope.go:122] "RemoveContainer" containerID="275326fe0b138876b3a03f1e9b84e7548fbdd44053ffde8667c1cebefb86a2c6"
Oct 30 21:15:20.584710 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:20.584694    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-27nkg"
Oct 30 21:15:20.603332 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:20.603290    5713 scope.go:122] "RemoveContainer" containerID="7fe97e301ee843caf143d1182cefe60b87fed3180e56faf490513a7213b71597"
Oct 30 21:15:20.798743 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:20.798710    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5c45e5-29889915-zm4fd"
Oct 30 21:15:20.864752 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:20.864720    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88da25-29889915-gf4hb"]
Oct 30 21:15:20.869080 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:20.869029    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f9eff-29889915-wpmls"]
Oct 30 21:15:20.873165 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:20.873139    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f9eff-29889915-wpmls"]
Oct 30 21:15:20.890770 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:20.890737    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3a522bc-746d-4cdf-940d-ccb562ade3c0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3a522bc-746d-4cdf-940d-ccb562ade3c0-maas-api-service-ca\") pod \"c3a522bc-746d-4cdf-940d-ccb562ade3c0\" (UID: \"c3a522bc-746d-4cdf-940d-ccb562ade3c0\") "
Oct 30 21:15:20.890993 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:20.890816    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3a522bc-746d-4cdf-940d-ccb562ade3c0-kube-api-access-nsqft\" (UniqueName: \"kubernetes.io/projected/c3a522bc-746d-4cdf-940d-ccb562ade3c0-kube-api-access-nsqft\") pod \"c3a522bc-746d-4cdf-940d-ccb562ade3c0\" (UID: \"c3a522bc-746d-4cdf-940d-ccb562ade3c0\") "
Oct 30 21:15:20.891185 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:20.891157    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3a522bc-746d-4cdf-940d-ccb562ade3c0-maas-api-service-ca" pod "c3a522bc-746d-4cdf-940d-ccb562ade3c0" (UID: "c3a522bc-746d-4cdf-940d-ccb562ade3c0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:20.891364 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:20.891345    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3a522bc-746d-4cdf-940d-ccb562ade3c0-maas-api-service-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:20.893260 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:20.893234    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3a522bc-746d-4cdf-940d-ccb562ade3c0-kube-api-access-nsqft" pod "c3a522bc-746d-4cdf-940d-ccb562ade3c0" (UID: "c3a522bc-746d-4cdf-940d-ccb562ade3c0"). InnerVolumeSpecName "kube-api-access-nsqft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:20.992524 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:20.992493    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsqft\" (UniqueName: \"kubernetes.io/projected/c3a522bc-746d-4cdf-940d-ccb562ade3c0-kube-api-access-nsqft\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:21.069920 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:21.069885    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88da25-29889915-gf4hb"
Oct 30 21:15:21.085177 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:21.085147    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3543ab8c-8863-4207-ae05-d8094dbd8cb9" path="/var/lib/kubelet/pods/3543ab8c-8863-4207-ae05-d8094dbd8cb9/volumes"
Oct 30 21:15:21.085939 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:21.085922    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8d5eebe-b80b-4b99-ace8-7e87724decaf" path="/var/lib/kubelet/pods/a8d5eebe-b80b-4b99-ace8-7e87724decaf/volumes"
Oct 30 21:15:21.086493 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:21.086479    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa46459a-d46e-4af4-bdfc-d6872367429d" path="/var/lib/kubelet/pods/aa46459a-d46e-4af4-bdfc-d6872367429d/volumes"
Oct 30 21:15:21.087308 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:21.087293    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f59ff9d3-2927-4021-9103-fd2f60f7b655" path="/var/lib/kubelet/pods/f59ff9d3-2927-4021-9103-fd2f60f7b655/volumes"
Oct 30 21:15:21.131391 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:21.131363    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889915-27nkg"]
Oct 30 21:15:21.135500 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:21.135404    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889915-27nkg"]
Oct 30 21:15:21.195297 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:21.195256    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16adca55-5cc2-4452-aba7-3e891c005f98-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16adca55-5cc2-4452-aba7-3e891c005f98-maas-api-service-ca\") pod \"16adca55-5cc2-4452-aba7-3e891c005f98\" (UID: \"16adca55-5cc2-4452-aba7-3e891c005f98\") "
Oct 30 21:15:21.195547 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:21.195516    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16adca55-5cc2-4452-aba7-3e891c005f98-kube-api-access-bkdgk\" (UniqueName: \"kubernetes.io/projected/16adca55-5cc2-4452-aba7-3e891c005f98-kube-api-access-bkdgk\") pod \"16adca55-5cc2-4452-aba7-3e891c005f98\" (UID: \"16adca55-5cc2-4452-aba7-3e891c005f98\") "
Oct 30 21:15:21.197399 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:21.197376    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16adca55-5cc2-4452-aba7-3e891c005f98-maas-api-service-ca" pod "16adca55-5cc2-4452-aba7-3e891c005f98" (UID: "16adca55-5cc2-4452-aba7-3e891c005f98"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:21.200277 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:21.200249    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16adca55-5cc2-4452-aba7-3e891c005f98-kube-api-access-bkdgk" pod "16adca55-5cc2-4452-aba7-3e891c005f98" (UID: "16adca55-5cc2-4452-aba7-3e891c005f98"). InnerVolumeSpecName "kube-api-access-bkdgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:21.298315 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:21.298223    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16adca55-5cc2-4452-aba7-3e891c005f98-maas-api-service-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:21.298315 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:21.298261    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkdgk\" (UniqueName: \"kubernetes.io/projected/16adca55-5cc2-4452-aba7-3e891c005f98-kube-api-access-bkdgk\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:21.596682 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:21.596600    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5c45e5-29889915-zm4fd"
Oct 30 21:15:21.596682 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:21.596626    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5c45e5-29889915-zm4fd" event={"ID":"c3a522bc-746d-4cdf-940d-ccb562ade3c0","Type":"ContainerDied","Data":"aed3a280f06d4b5029de2fb16c29ceb74afba8d6cd7ea2dd68e775c8918b7099"}
Oct 30 21:15:21.597259 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:21.596692    5713 scope.go:122] "RemoveContainer" containerID="17433a2005cefd7c24eea71d6cde4729703f077f7f42bfc06c421df981786e6b"
Oct 30 21:15:21.601019 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:21.600994    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88da25-29889915-gf4hb" event={"ID":"16adca55-5cc2-4452-aba7-3e891c005f98","Type":"ContainerDied","Data":"8cbf8227dfcff041317b9c0a18ee1ad27b93dd0570bf27e7a1a878c5191e87b4"}
Oct 30 21:15:21.601232 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:21.601207    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88da25-29889915-gf4hb"
Oct 30 21:15:21.611243 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:21.611223    5713 scope.go:122] "RemoveContainer" containerID="b6add717ab64b5ac643e2852e41b13d3bb7cee241f8ba64699fa0b05d5b33e26"
Oct 30 21:15:21.681890 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:21.681833    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-677b45-29889915-4m4cj"]
Oct 30 21:15:21.782079 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:21.782018    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88da25-29889915-gf4hb"]
Oct 30 21:15:21.785747 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:21.785724    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-88da25-29889915-gf4hb"]
Oct 30 21:15:21.896685 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:21.896646    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5c45e5-29889915-zm4fd"]
Oct 30 21:15:21.901162 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:21.901133    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5c45e5-29889915-zm4fd"]
Oct 30 21:15:21.902298 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:21.902276    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-677b45-29889915-4m4cj"
Oct 30 21:15:22.001619 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:22.001582    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-682999-29889915-vzbj5"]
Oct 30 21:15:22.009664 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:22.008903    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5117983-c7b5-4679-8c1d-90ea13ebca99-kube-api-access-6whz8\" (UniqueName: \"kubernetes.io/projected/b5117983-c7b5-4679-8c1d-90ea13ebca99-kube-api-access-6whz8\") pod \"b5117983-c7b5-4679-8c1d-90ea13ebca99\" (UID: \"b5117983-c7b5-4679-8c1d-90ea13ebca99\") "
Oct 30 21:15:22.009664 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:22.009044    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5117983-c7b5-4679-8c1d-90ea13ebca99-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5117983-c7b5-4679-8c1d-90ea13ebca99-maas-api-service-ca\") pod \"b5117983-c7b5-4679-8c1d-90ea13ebca99\" (UID: \"b5117983-c7b5-4679-8c1d-90ea13ebca99\") "
Oct 30 21:15:22.009664 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:22.009407    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5117983-c7b5-4679-8c1d-90ea13ebca99-maas-api-service-ca" pod "b5117983-c7b5-4679-8c1d-90ea13ebca99" (UID: "b5117983-c7b5-4679-8c1d-90ea13ebca99"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:22.009948 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:22.009681    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5117983-c7b5-4679-8c1d-90ea13ebca99-maas-api-service-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:22.012001 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:22.011975    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5117983-c7b5-4679-8c1d-90ea13ebca99-kube-api-access-6whz8" pod "b5117983-c7b5-4679-8c1d-90ea13ebca99" (UID: "b5117983-c7b5-4679-8c1d-90ea13ebca99"). InnerVolumeSpecName "kube-api-access-6whz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:22.110779 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:22.110742    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6whz8\" (UniqueName: \"kubernetes.io/projected/b5117983-c7b5-4679-8c1d-90ea13ebca99-kube-api-access-6whz8\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:22.206616 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:22.206592    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-682999-29889915-vzbj5"
Oct 30 21:15:22.313983 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:22.313948    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/126910b1-789a-4e00-9e84-2f0a9c899abb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/126910b1-789a-4e00-9e84-2f0a9c899abb-maas-api-service-ca\") pod \"126910b1-789a-4e00-9e84-2f0a9c899abb\" (UID: \"126910b1-789a-4e00-9e84-2f0a9c899abb\") "
Oct 30 21:15:22.314198 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:22.314082    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/126910b1-789a-4e00-9e84-2f0a9c899abb-kube-api-access-wpzrc\" (UniqueName: \"kubernetes.io/projected/126910b1-789a-4e00-9e84-2f0a9c899abb-kube-api-access-wpzrc\") pod \"126910b1-789a-4e00-9e84-2f0a9c899abb\" (UID: \"126910b1-789a-4e00-9e84-2f0a9c899abb\") "
Oct 30 21:15:22.315309 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:22.315178    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/126910b1-789a-4e00-9e84-2f0a9c899abb-maas-api-service-ca" pod "126910b1-789a-4e00-9e84-2f0a9c899abb" (UID: "126910b1-789a-4e00-9e84-2f0a9c899abb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:22.316882 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:22.316851    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/126910b1-789a-4e00-9e84-2f0a9c899abb-kube-api-access-wpzrc" pod "126910b1-789a-4e00-9e84-2f0a9c899abb" (UID: "126910b1-789a-4e00-9e84-2f0a9c899abb"). InnerVolumeSpecName "kube-api-access-wpzrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:22.380718 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:22.380682    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f9f277-29889915-dbd9b"]
Oct 30 21:15:22.416591 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:22.416546    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpzrc\" (UniqueName: \"kubernetes.io/projected/126910b1-789a-4e00-9e84-2f0a9c899abb-kube-api-access-wpzrc\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:22.416591 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:22.416583    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/126910b1-789a-4e00-9e84-2f0a9c899abb-maas-api-service-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:22.607402 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:22.607371    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f9f277-29889915-dbd9b"
Oct 30 21:15:22.612802 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:22.612765    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-682999-29889915-vzbj5"
Oct 30 21:15:22.612938 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:22.612833    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-682999-29889915-vzbj5" event={"ID":"126910b1-789a-4e00-9e84-2f0a9c899abb","Type":"ContainerDied","Data":"51f21fb20b8c7f98b4b7edaa4a473ae4741a599792ce358c12c4cda15aaf86a8"}
Oct 30 21:15:22.612938 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:22.612901    5713 scope.go:122] "RemoveContainer" containerID="3fc56164212a377e3865a8d2ca10a82743ca5d0851e87d639f6df2c883bf1e33"
Oct 30 21:15:22.615903 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:22.615876    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f9f277-29889915-dbd9b" event={"ID":"9970e7c4-7661-47c5-a6df-4de2bea436e2","Type":"ContainerDied","Data":"db031a964d81dfd6850aa26fb5d9ce2e8ddbca1a618447f20b665787d2ac8c8a"}
Oct 30 21:15:22.616005 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:22.615992    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f9f277-29889915-dbd9b"
Oct 30 21:15:22.619553 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:22.619527    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-677b45-29889915-4m4cj"
Oct 30 21:15:22.619868 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:22.619837    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-677b45-29889915-4m4cj" event={"ID":"b5117983-c7b5-4679-8c1d-90ea13ebca99","Type":"ContainerDied","Data":"1178eda94691d33d074ad95e5142499d24f892fba2929d997e1ccf0566181a14"}
Oct 30 21:15:22.628616 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:22.628597    5713 scope.go:122] "RemoveContainer" containerID="08f4604e94056363333e54f16bd2a34389e65f3ad6c912d64ad5915b6ec5e356"
Oct 30 21:15:22.668508 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:22.668482    5713 scope.go:122] "RemoveContainer" containerID="28b55757ef5d4f15ed47a6c8d6f717044502f253e26d2a67cc36fe8a9dfe8b5a"
Oct 30 21:15:22.720462 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:22.720424    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9970e7c4-7661-47c5-a6df-4de2bea436e2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9970e7c4-7661-47c5-a6df-4de2bea436e2-maas-api-service-ca\") pod \"9970e7c4-7661-47c5-a6df-4de2bea436e2\" (UID: \"9970e7c4-7661-47c5-a6df-4de2bea436e2\") "
Oct 30 21:15:22.720667 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:22.720646    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9970e7c4-7661-47c5-a6df-4de2bea436e2-kube-api-access-8dmlh\" (UniqueName: \"kubernetes.io/projected/9970e7c4-7661-47c5-a6df-4de2bea436e2-kube-api-access-8dmlh\") pod \"9970e7c4-7661-47c5-a6df-4de2bea436e2\" (UID: \"9970e7c4-7661-47c5-a6df-4de2bea436e2\") "
Oct 30 21:15:22.720850 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:22.720828    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9970e7c4-7661-47c5-a6df-4de2bea436e2-maas-api-service-ca" pod "9970e7c4-7661-47c5-a6df-4de2bea436e2" (UID: "9970e7c4-7661-47c5-a6df-4de2bea436e2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:22.721200 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:22.721178    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9970e7c4-7661-47c5-a6df-4de2bea436e2-maas-api-service-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:22.723016 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:22.722990    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9970e7c4-7661-47c5-a6df-4de2bea436e2-kube-api-access-8dmlh" pod "9970e7c4-7661-47c5-a6df-4de2bea436e2" (UID: "9970e7c4-7661-47c5-a6df-4de2bea436e2"). InnerVolumeSpecName "kube-api-access-8dmlh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:22.823065 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:22.823030    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dmlh\" (UniqueName: \"kubernetes.io/projected/9970e7c4-7661-47c5-a6df-4de2bea436e2-kube-api-access-8dmlh\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:22.867378 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:22.867344    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-682999-29889915-vzbj5"]
Oct 30 21:15:22.871271 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:22.871242    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-682999-29889915-vzbj5"]
Oct 30 21:15:23.073550 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:23.073461    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="126910b1-789a-4e00-9e84-2f0a9c899abb" path="/var/lib/kubelet/pods/126910b1-789a-4e00-9e84-2f0a9c899abb/volumes"
Oct 30 21:15:23.074101 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:23.074083    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16adca55-5cc2-4452-aba7-3e891c005f98" path="/var/lib/kubelet/pods/16adca55-5cc2-4452-aba7-3e891c005f98/volumes"
Oct 30 21:15:23.074618 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:23.074603    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2184c91-9d42-4a01-bf05-dd962ff62dcb" path="/var/lib/kubelet/pods/c2184c91-9d42-4a01-bf05-dd962ff62dcb/volumes"
Oct 30 21:15:23.075628 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:23.075609    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3a522bc-746d-4cdf-940d-ccb562ade3c0" path="/var/lib/kubelet/pods/c3a522bc-746d-4cdf-940d-ccb562ade3c0/volumes"
Oct 30 21:15:23.076076 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:23.076052    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-677b45-29889915-4m4cj"]
Oct 30 21:15:23.076123 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:23.076102    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-677b45-29889915-4m4cj"]
Oct 30 21:15:23.161818 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:23.161740    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f9f277-29889915-dbd9b"]
Oct 30 21:15:23.165736 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:23.165710    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f9f277-29889915-dbd9b"]
Oct 30 21:15:24.460238 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:15:24.460201    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:15:25.066293 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:25.066258    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9970e7c4-7661-47c5-a6df-4de2bea436e2" path="/var/lib/kubelet/pods/9970e7c4-7661-47c5-a6df-4de2bea436e2/volumes"
Oct 30 21:15:25.066888 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:25.066865    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5117983-c7b5-4679-8c1d-90ea13ebca99" path="/var/lib/kubelet/pods/b5117983-c7b5-4679-8c1d-90ea13ebca99/volumes"
Oct 30 21:15:25.483250 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:25.483210    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-7eaf74-b6mtd"]
Oct 30 21:15:25.532496 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:25.532462    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-7eaf74-b6mtd"]
Oct 30 21:15:27.066154 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:27.066115    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9364652a-228c-4e15-ac17-6d26d75befad" path="/var/lib/kubelet/pods/9364652a-228c-4e15-ac17-6d26d75befad/volumes"
Oct 30 21:15:27.669993 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:27.669955    5713 generic.go:356] "Generic (PLEG): container finished" podID="94f9b9e7-7097-43d9-870e-98b67f32d0e9" containerID="b38d889798e8ba005656c437fe5daa846aacd07bb171168a11444965343e6eab" exitCode=0
Oct 30 21:15:27.670230 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:27.670035    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g" event={"ID":"94f9b9e7-7097-43d9-870e-98b67f32d0e9","Type":"ContainerDied","Data":"b38d889798e8ba005656c437fe5daa846aacd07bb171168a11444965343e6eab"}
Oct 30 21:15:27.670531 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:27.670514    5713 scope.go:122] "RemoveContainer" containerID="b38d889798e8ba005656c437fe5daa846aacd07bb171168a11444965343e6eab"
Oct 30 21:15:27.672303 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:27.672274    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:15:27.672370 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:27.672343    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:15:27.672411 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:27.672379    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:15:28.680802 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:28.680751    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g" event={"ID":"94f9b9e7-7097-43d9-870e-98b67f32d0e9","Type":"ContainerStarted","Data":"bdedc3f83a88ecf8efa43b98fdbc64d2cbfe0d670af59a0a8d403e3b35ef9d0e"}
Oct 30 21:15:28.818742 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:28.818706    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g"
Oct 30 21:15:28.820056 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:28.820032    5713 patch_prober.go:29] interesting pod/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 30 21:15:28.820198 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:28.820096    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g" podUID="94f9b9e7-7097-43d9-870e-98b67f32d0e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 30 21:15:29.078531 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:29.078443    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g"
Oct 30 21:15:29.819038 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:29.819005    5713 patch_prober.go:29] interesting pod/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 30 21:15:29.819455 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:29.819055    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g" podUID="94f9b9e7-7097-43d9-870e-98b67f32d0e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 30 21:15:30.818874 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:30.818842    5713 patch_prober.go:29] interesting pod/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 30 21:15:30.819056 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:30.818914    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g" podUID="94f9b9e7-7097-43d9-870e-98b67f32d0e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 30 21:15:31.818936 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:31.818905    5713 patch_prober.go:29] interesting pod/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 30 21:15:31.819382 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:31.818956    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g" podUID="94f9b9e7-7097-43d9-870e-98b67f32d0e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 30 21:15:32.818577 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:32.818544    5713 patch_prober.go:29] interesting pod/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 30 21:15:32.818778 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:32.818609    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g" podUID="94f9b9e7-7097-43d9-870e-98b67f32d0e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 30 21:15:33.097323 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:33.097230    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf"]
Oct 30 21:15:33.098136 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:33.098103    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf" podUID="c59144f6-c298-4a9d-a3c3-7a2e81bb8967" containerName="istio-proxy" containerID="cri-o://0fcb783b7658411186f3e3c318cf06459e180201e511a6ef12564779e28124fe" gracePeriod=30
Oct 30 21:15:33.119360 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:33.119329    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4"]
Oct 30 21:15:33.119579 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:33.119556    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4" podUID="0b85e68b-c6a9-4362-a6c0-93af13623e67" containerName="istio-proxy" containerID="cri-o://a79c0d0d0481c3988fda6012603ca6917b6fb98b676c2ae18668cddb5d157a2b" gracePeriod=30
Oct 30 21:15:33.123090 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:33.123065    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz"]
Oct 30 21:15:33.123271 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:33.123251    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz" podUID="d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f" containerName="istio-proxy" containerID="cri-o://cdd26ac7c03c95d666f43e2e53a5b8e81a441eb72f86201ea2b8f3da1d35b591" gracePeriod=30
Oct 30 21:15:33.299817 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:33.299121    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-390b71-5698dc795b-2hp76"]
Oct 30 21:15:33.367135 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:33.367045    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-390b71-7ff679864c-ghxkc"]
Oct 30 21:15:33.818611 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:33.818578    5713 patch_prober.go:29] interesting pod/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 30 21:15:33.818822 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:33.818631    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g" podUID="94f9b9e7-7097-43d9-870e-98b67f32d0e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 30 21:15:34.851842 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:34.851434    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g"
Oct 30 21:15:34.858920 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:34.857998    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g"
Oct 30 21:15:35.423768 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.423736    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh"]
Oct 30 21:15:35.424362 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424347    5713 cpu_manager.go:414] "containerMap: removing container" podUID="a8d5eebe-b80b-4b99-ace8-7e87724decaf" containerName="cleanup"
Oct 30 21:15:35.424410 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424373    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8d5eebe-b80b-4b99-ace8-7e87724decaf" containerName="cleanup"
Oct 30 21:15:35.424410 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424390    5713 cpu_manager.go:414] "containerMap: removing container" podUID="f59ff9d3-2927-4021-9103-fd2f60f7b655" containerName="cleanup"
Oct 30 21:15:35.424410 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424395    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f59ff9d3-2927-4021-9103-fd2f60f7b655" containerName="cleanup"
Oct 30 21:15:35.424410 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424406    5713 cpu_manager.go:414] "containerMap: removing container" podUID="16adca55-5cc2-4452-aba7-3e891c005f98" containerName="cleanup"
Oct 30 21:15:35.424410 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424410    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16adca55-5cc2-4452-aba7-3e891c005f98" containerName="cleanup"
Oct 30 21:15:35.424567 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424419    5713 cpu_manager.go:414] "containerMap: removing container" podUID="16adca55-5cc2-4452-aba7-3e891c005f98" containerName="cleanup"
Oct 30 21:15:35.424567 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424424    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16adca55-5cc2-4452-aba7-3e891c005f98" containerName="cleanup"
Oct 30 21:15:35.424567 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424430    5713 cpu_manager.go:414] "containerMap: removing container" podUID="c3a522bc-746d-4cdf-940d-ccb562ade3c0" containerName="cleanup"
Oct 30 21:15:35.424567 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424435    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3a522bc-746d-4cdf-940d-ccb562ade3c0" containerName="cleanup"
Oct 30 21:15:35.424567 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424441    5713 cpu_manager.go:414] "containerMap: removing container" podUID="9970e7c4-7661-47c5-a6df-4de2bea436e2" containerName="cleanup"
Oct 30 21:15:35.424567 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424445    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9970e7c4-7661-47c5-a6df-4de2bea436e2" containerName="cleanup"
Oct 30 21:15:35.424567 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424458    5713 cpu_manager.go:414] "containerMap: removing container" podUID="a8d5eebe-b80b-4b99-ace8-7e87724decaf" containerName="cleanup"
Oct 30 21:15:35.424567 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424463    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8d5eebe-b80b-4b99-ace8-7e87724decaf" containerName="cleanup"
Oct 30 21:15:35.424567 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424469    5713 cpu_manager.go:414] "containerMap: removing container" podUID="aa46459a-d46e-4af4-bdfc-d6872367429d" containerName="cleanup"
Oct 30 21:15:35.424567 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424473    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa46459a-d46e-4af4-bdfc-d6872367429d" containerName="cleanup"
Oct 30 21:15:35.424567 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424484    5713 cpu_manager.go:414] "containerMap: removing container" podUID="a08925fb-aee2-425a-84da-c407fb8cfb7f" containerName="payload-processing"
Oct 30 21:15:35.424567 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424488    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a08925fb-aee2-425a-84da-c407fb8cfb7f" containerName="payload-processing"
Oct 30 21:15:35.424567 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424497    5713 cpu_manager.go:414] "containerMap: removing container" podUID="aa46459a-d46e-4af4-bdfc-d6872367429d" containerName="cleanup"
Oct 30 21:15:35.424567 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424502    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa46459a-d46e-4af4-bdfc-d6872367429d" containerName="cleanup"
Oct 30 21:15:35.424567 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424511    5713 cpu_manager.go:414] "containerMap: removing container" podUID="e1255405-8724-4977-854d-11f13d8a847a" containerName="istio-proxy"
Oct 30 21:15:35.424567 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424515    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1255405-8724-4977-854d-11f13d8a847a" containerName="istio-proxy"
Oct 30 21:15:35.424567 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424522    5713 cpu_manager.go:414] "containerMap: removing container" podUID="126910b1-789a-4e00-9e84-2f0a9c899abb" containerName="cleanup"
Oct 30 21:15:35.424567 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424526    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="126910b1-789a-4e00-9e84-2f0a9c899abb" containerName="cleanup"
Oct 30 21:15:35.424567 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424533    5713 cpu_manager.go:414] "containerMap: removing container" podUID="aa46459a-d46e-4af4-bdfc-d6872367429d" containerName="cleanup"
Oct 30 21:15:35.424567 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424537    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa46459a-d46e-4af4-bdfc-d6872367429d" containerName="cleanup"
Oct 30 21:15:35.424567 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424545    5713 cpu_manager.go:414] "containerMap: removing container" podUID="b5117983-c7b5-4679-8c1d-90ea13ebca99" containerName="cleanup"
Oct 30 21:15:35.424567 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424549    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5117983-c7b5-4679-8c1d-90ea13ebca99" containerName="cleanup"
Oct 30 21:15:35.424567 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424559    5713 cpu_manager.go:414] "containerMap: removing container" podUID="b47eb722-c69a-4565-b0ae-1b46f99090df" containerName="payload-pre-processing"
Oct 30 21:15:35.424567 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424563    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b47eb722-c69a-4565-b0ae-1b46f99090df" containerName="payload-pre-processing"
Oct 30 21:15:35.424567 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424569    5713 cpu_manager.go:414] "containerMap: removing container" podUID="b5117983-c7b5-4679-8c1d-90ea13ebca99" containerName="cleanup"
Oct 30 21:15:35.424567 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424575    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5117983-c7b5-4679-8c1d-90ea13ebca99" containerName="cleanup"
Oct 30 21:15:35.425375 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424582    5713 cpu_manager.go:414] "containerMap: removing container" podUID="126910b1-789a-4e00-9e84-2f0a9c899abb" containerName="cleanup"
Oct 30 21:15:35.425375 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424587    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="126910b1-789a-4e00-9e84-2f0a9c899abb" containerName="cleanup"
Oct 30 21:15:35.425375 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424593    5713 cpu_manager.go:414] "containerMap: removing container" podUID="c3a522bc-746d-4cdf-940d-ccb562ade3c0" containerName="cleanup"
Oct 30 21:15:35.425375 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424597    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3a522bc-746d-4cdf-940d-ccb562ade3c0" containerName="cleanup"
Oct 30 21:15:35.425375 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424603    5713 cpu_manager.go:414] "containerMap: removing container" podUID="b5117983-c7b5-4679-8c1d-90ea13ebca99" containerName="cleanup"
Oct 30 21:15:35.425375 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424607    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5117983-c7b5-4679-8c1d-90ea13ebca99" containerName="cleanup"
Oct 30 21:15:35.425375 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424615    5713 cpu_manager.go:414] "containerMap: removing container" podUID="3faef396-57dc-4ff5-af9f-c1f940cce913" containerName="istio-proxy"
Oct 30 21:15:35.425375 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424619    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3faef396-57dc-4ff5-af9f-c1f940cce913" containerName="istio-proxy"
Oct 30 21:15:35.425375 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424625    5713 cpu_manager.go:414] "containerMap: removing container" podUID="f59ff9d3-2927-4021-9103-fd2f60f7b655" containerName="cleanup"
Oct 30 21:15:35.425375 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424629    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f59ff9d3-2927-4021-9103-fd2f60f7b655" containerName="cleanup"
Oct 30 21:15:35.425375 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424634    5713 cpu_manager.go:414] "containerMap: removing container" podUID="c2184c91-9d42-4a01-bf05-dd962ff62dcb" containerName="cleanup"
Oct 30 21:15:35.425375 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424638    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2184c91-9d42-4a01-bf05-dd962ff62dcb" containerName="cleanup"
Oct 30 21:15:35.425375 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424644    5713 cpu_manager.go:414] "containerMap: removing container" podUID="126910b1-789a-4e00-9e84-2f0a9c899abb" containerName="cleanup"
Oct 30 21:15:35.425375 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424648    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="126910b1-789a-4e00-9e84-2f0a9c899abb" containerName="cleanup"
Oct 30 21:15:35.425375 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424656    5713 cpu_manager.go:414] "containerMap: removing container" podUID="c3a522bc-746d-4cdf-940d-ccb562ade3c0" containerName="cleanup"
Oct 30 21:15:35.425375 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424660    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3a522bc-746d-4cdf-940d-ccb562ade3c0" containerName="cleanup"
Oct 30 21:15:35.425375 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424670    5713 cpu_manager.go:414] "containerMap: removing container" podUID="c2184c91-9d42-4a01-bf05-dd962ff62dcb" containerName="cleanup"
Oct 30 21:15:35.425375 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424674    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2184c91-9d42-4a01-bf05-dd962ff62dcb" containerName="cleanup"
Oct 30 21:15:35.425375 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424680    5713 cpu_manager.go:414] "containerMap: removing container" podUID="3543ab8c-8863-4207-ae05-d8094dbd8cb9" containerName="cleanup"
Oct 30 21:15:35.425375 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424684    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3543ab8c-8863-4207-ae05-d8094dbd8cb9" containerName="cleanup"
Oct 30 21:15:35.425375 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424691    5713 cpu_manager.go:414] "containerMap: removing container" podUID="c2184c91-9d42-4a01-bf05-dd962ff62dcb" containerName="cleanup"
Oct 30 21:15:35.425375 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424695    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2184c91-9d42-4a01-bf05-dd962ff62dcb" containerName="cleanup"
Oct 30 21:15:35.425375 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424701    5713 cpu_manager.go:414] "containerMap: removing container" podUID="9970e7c4-7661-47c5-a6df-4de2bea436e2" containerName="cleanup"
Oct 30 21:15:35.425375 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424704    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9970e7c4-7661-47c5-a6df-4de2bea436e2" containerName="cleanup"
Oct 30 21:15:35.425375 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424711    5713 cpu_manager.go:414] "containerMap: removing container" podUID="3543ab8c-8863-4207-ae05-d8094dbd8cb9" containerName="cleanup"
Oct 30 21:15:35.425375 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424715    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3543ab8c-8863-4207-ae05-d8094dbd8cb9" containerName="cleanup"
Oct 30 21:15:35.425375 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424721    5713 cpu_manager.go:414] "containerMap: removing container" podUID="3543ab8c-8863-4207-ae05-d8094dbd8cb9" containerName="cleanup"
Oct 30 21:15:35.425375 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424725    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3543ab8c-8863-4207-ae05-d8094dbd8cb9" containerName="cleanup"
Oct 30 21:15:35.425375 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424829    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1255405-8724-4977-854d-11f13d8a847a" containerName="istio-proxy"
Oct 30 21:15:35.425375 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424840    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="126910b1-789a-4e00-9e84-2f0a9c899abb" containerName="cleanup"
Oct 30 21:15:35.425375 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424847    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa46459a-d46e-4af4-bdfc-d6872367429d" containerName="cleanup"
Oct 30 21:15:35.425375 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424852    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5117983-c7b5-4679-8c1d-90ea13ebca99" containerName="cleanup"
Oct 30 21:15:35.425375 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424859    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="b47eb722-c69a-4565-b0ae-1b46f99090df" containerName="payload-pre-processing"
Oct 30 21:15:35.425375 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424866    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="f59ff9d3-2927-4021-9103-fd2f60f7b655" containerName="cleanup"
Oct 30 21:15:35.425375 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424871    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5117983-c7b5-4679-8c1d-90ea13ebca99" containerName="cleanup"
Oct 30 21:15:35.425375 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424877    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="126910b1-789a-4e00-9e84-2f0a9c899abb" containerName="cleanup"
Oct 30 21:15:35.425375 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424883    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3a522bc-746d-4cdf-940d-ccb562ade3c0" containerName="cleanup"
Oct 30 21:15:35.425375 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424888    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5117983-c7b5-4679-8c1d-90ea13ebca99" containerName="cleanup"
Oct 30 21:15:35.425375 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424893    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="3faef396-57dc-4ff5-af9f-c1f940cce913" containerName="istio-proxy"
Oct 30 21:15:35.425375 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424898    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="9970e7c4-7661-47c5-a6df-4de2bea436e2" containerName="cleanup"
Oct 30 21:15:35.425375 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424902    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="f59ff9d3-2927-4021-9103-fd2f60f7b655" containerName="cleanup"
Oct 30 21:15:35.425375 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424907    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2184c91-9d42-4a01-bf05-dd962ff62dcb" containerName="cleanup"
Oct 30 21:15:35.425375 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424913    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="126910b1-789a-4e00-9e84-2f0a9c899abb" containerName="cleanup"
Oct 30 21:15:35.425375 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424919    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3a522bc-746d-4cdf-940d-ccb562ade3c0" containerName="cleanup"
Oct 30 21:15:35.425375 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424924    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2184c91-9d42-4a01-bf05-dd962ff62dcb" containerName="cleanup"
Oct 30 21:15:35.425375 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424929    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="3543ab8c-8863-4207-ae05-d8094dbd8cb9" containerName="cleanup"
Oct 30 21:15:35.425375 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424935    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2184c91-9d42-4a01-bf05-dd962ff62dcb" containerName="cleanup"
Oct 30 21:15:35.425375 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424940    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="9970e7c4-7661-47c5-a6df-4de2bea436e2" containerName="cleanup"
Oct 30 21:15:35.425375 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424944    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="3543ab8c-8863-4207-ae05-d8094dbd8cb9" containerName="cleanup"
Oct 30 21:15:35.425375 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424950    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="3543ab8c-8863-4207-ae05-d8094dbd8cb9" containerName="cleanup"
Oct 30 21:15:35.425375 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424955    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8d5eebe-b80b-4b99-ace8-7e87724decaf" containerName="cleanup"
Oct 30 21:15:35.425375 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424961    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="f59ff9d3-2927-4021-9103-fd2f60f7b655" containerName="cleanup"
Oct 30 21:15:35.425375 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424967    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="16adca55-5cc2-4452-aba7-3e891c005f98" containerName="cleanup"
Oct 30 21:15:35.425375 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424971    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="16adca55-5cc2-4452-aba7-3e891c005f98" containerName="cleanup"
Oct 30 21:15:35.425375 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424976    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8d5eebe-b80b-4b99-ace8-7e87724decaf" containerName="cleanup"
Oct 30 21:15:35.425375 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424983    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8d5eebe-b80b-4b99-ace8-7e87724decaf" containerName="cleanup"
Oct 30 21:15:35.425375 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424988    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa46459a-d46e-4af4-bdfc-d6872367429d" containerName="cleanup"
Oct 30 21:15:35.425375 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.424993    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="a08925fb-aee2-425a-84da-c407fb8cfb7f" containerName="payload-processing"
Oct 30 21:15:35.428695 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.428679    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh"
Oct 30 21:15:35.485278 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.485252    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-d1b889bd-openshift-default-dockercfg-h9jb2\""
Oct 30 21:15:35.500685 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.500649    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh"]
Oct 30 21:15:35.501583 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.501554    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d34434b-cc8a-4ee2-a774-cacdea5465de-istio-data\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh\" (UID: \"4d34434b-cc8a-4ee2-a774-cacdea5465de\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh"
Oct 30 21:15:35.501665 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.501605    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d34434b-cc8a-4ee2-a774-cacdea5465de-istiod-ca-cert\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh\" (UID: \"4d34434b-cc8a-4ee2-a774-cacdea5465de\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh"
Oct 30 21:15:35.501747 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.501725    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d34434b-cc8a-4ee2-a774-cacdea5465de-istio-token\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh\" (UID: \"4d34434b-cc8a-4ee2-a774-cacdea5465de\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh"
Oct 30 21:15:35.501916 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.501891    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d34434b-cc8a-4ee2-a774-cacdea5465de-istio-podinfo\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh\" (UID: \"4d34434b-cc8a-4ee2-a774-cacdea5465de\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh"
Oct 30 21:15:35.502016 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.501960    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d34434b-cc8a-4ee2-a774-cacdea5465de-istio-envoy\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh\" (UID: \"4d34434b-cc8a-4ee2-a774-cacdea5465de\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh"
Oct 30 21:15:35.502097 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.502073    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jblvf\" (UniqueName: \"kubernetes.io/projected/4d34434b-cc8a-4ee2-a774-cacdea5465de-kube-api-access-jblvf\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh\" (UID: \"4d34434b-cc8a-4ee2-a774-cacdea5465de\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh"
Oct 30 21:15:35.502180 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.502162    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d34434b-cc8a-4ee2-a774-cacdea5465de-workload-socket\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh\" (UID: \"4d34434b-cc8a-4ee2-a774-cacdea5465de\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh"
Oct 30 21:15:35.502300 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.502279    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d34434b-cc8a-4ee2-a774-cacdea5465de-credential-socket\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh\" (UID: \"4d34434b-cc8a-4ee2-a774-cacdea5465de\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh"
Oct 30 21:15:35.502434 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.502401    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d34434b-cc8a-4ee2-a774-cacdea5465de-workload-certs\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh\" (UID: \"4d34434b-cc8a-4ee2-a774-cacdea5465de\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh"
Oct 30 21:15:35.603591 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.603558    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d34434b-cc8a-4ee2-a774-cacdea5465de-workload-certs\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh\" (UID: \"4d34434b-cc8a-4ee2-a774-cacdea5465de\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh"
Oct 30 21:15:35.603770 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.603614    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d34434b-cc8a-4ee2-a774-cacdea5465de-istio-data\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh\" (UID: \"4d34434b-cc8a-4ee2-a774-cacdea5465de\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh"
Oct 30 21:15:35.603770 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.603730    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d34434b-cc8a-4ee2-a774-cacdea5465de-istiod-ca-cert\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh\" (UID: \"4d34434b-cc8a-4ee2-a774-cacdea5465de\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh"
Oct 30 21:15:35.603929 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.603778    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d34434b-cc8a-4ee2-a774-cacdea5465de-istio-token\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh\" (UID: \"4d34434b-cc8a-4ee2-a774-cacdea5465de\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh"
Oct 30 21:15:35.603929 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.603913    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d34434b-cc8a-4ee2-a774-cacdea5465de-istio-podinfo\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh\" (UID: \"4d34434b-cc8a-4ee2-a774-cacdea5465de\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh"
Oct 30 21:15:35.604039 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.603971    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d34434b-cc8a-4ee2-a774-cacdea5465de-istio-envoy\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh\" (UID: \"4d34434b-cc8a-4ee2-a774-cacdea5465de\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh"
Oct 30 21:15:35.604039 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.604024    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d34434b-cc8a-4ee2-a774-cacdea5465de-workload-certs\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh\" (UID: \"4d34434b-cc8a-4ee2-a774-cacdea5465de\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh"
Oct 30 21:15:35.604158 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.603973    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d34434b-cc8a-4ee2-a774-cacdea5465de-istio-data\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh\" (UID: \"4d34434b-cc8a-4ee2-a774-cacdea5465de\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh"
Oct 30 21:15:35.604158 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.604052    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jblvf\" (UniqueName: \"kubernetes.io/projected/4d34434b-cc8a-4ee2-a774-cacdea5465de-kube-api-access-jblvf\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh\" (UID: \"4d34434b-cc8a-4ee2-a774-cacdea5465de\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh"
Oct 30 21:15:35.604158 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.604119    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d34434b-cc8a-4ee2-a774-cacdea5465de-workload-socket\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh\" (UID: \"4d34434b-cc8a-4ee2-a774-cacdea5465de\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh"
Oct 30 21:15:35.604324 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.604201    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d34434b-cc8a-4ee2-a774-cacdea5465de-credential-socket\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh\" (UID: \"4d34434b-cc8a-4ee2-a774-cacdea5465de\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh"
Oct 30 21:15:35.604433 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.604397    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d34434b-cc8a-4ee2-a774-cacdea5465de-istiod-ca-cert\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh\" (UID: \"4d34434b-cc8a-4ee2-a774-cacdea5465de\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh"
Oct 30 21:15:35.604565 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.604440    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d34434b-cc8a-4ee2-a774-cacdea5465de-workload-socket\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh\" (UID: \"4d34434b-cc8a-4ee2-a774-cacdea5465de\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh"
Oct 30 21:15:35.604565 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.604510    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d34434b-cc8a-4ee2-a774-cacdea5465de-credential-socket\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh\" (UID: \"4d34434b-cc8a-4ee2-a774-cacdea5465de\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh"
Oct 30 21:15:35.606427 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.606406    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d34434b-cc8a-4ee2-a774-cacdea5465de-istio-envoy\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh\" (UID: \"4d34434b-cc8a-4ee2-a774-cacdea5465de\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh"
Oct 30 21:15:35.606540 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.606522    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d34434b-cc8a-4ee2-a774-cacdea5465de-istio-podinfo\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh\" (UID: \"4d34434b-cc8a-4ee2-a774-cacdea5465de\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh"
Oct 30 21:15:35.663480 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.663449    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d34434b-cc8a-4ee2-a774-cacdea5465de-istio-token\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh\" (UID: \"4d34434b-cc8a-4ee2-a774-cacdea5465de\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh"
Oct 30 21:15:35.669455 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.669427    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jblvf\" (UniqueName: \"kubernetes.io/projected/4d34434b-cc8a-4ee2-a774-cacdea5465de-kube-api-access-jblvf\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh\" (UID: \"4d34434b-cc8a-4ee2-a774-cacdea5465de\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh"
Oct 30 21:15:35.744546 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:35.744511    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh"
Oct 30 21:15:36.193361 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:15:36.193327    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d34434b_cc8a_4ee2_a774_cacdea5465de.slice/crio-65372081de886ca60ef75c1d96951e6c51810c9a657090b96dd954ead10acefb WatchSource:0}: Error finding container 65372081de886ca60ef75c1d96951e6c51810c9a657090b96dd954ead10acefb: Status 404 returned error can't find the container with id 65372081de886ca60ef75c1d96951e6c51810c9a657090b96dd954ead10acefb
Oct 30 21:15:36.193361 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:36.193354    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh"]
Oct 30 21:15:36.196208 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:36.196179    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:15:36.196312 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:36.196238    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:15:36.196312 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:36.196287    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:15:36.761403 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:36.761364    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh" event={"ID":"4d34434b-cc8a-4ee2-a774-cacdea5465de","Type":"ContainerStarted","Data":"e00836866e297848f37cc5fd9f64d6e203b7203567da64847fd01764c6023368"}
Oct 30 21:15:36.761616 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:36.761436    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh" event={"ID":"4d34434b-cc8a-4ee2-a774-cacdea5465de","Type":"ContainerStarted","Data":"65372081de886ca60ef75c1d96951e6c51810c9a657090b96dd954ead10acefb"}
Oct 30 21:15:36.880417 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:36.880350    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh" podStartSLOduration=1.88033487 podStartE2EDuration="1.88033487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:36.879110644 +0000 UTC m=+2219.943887114" watchObservedRunningTime="2026-10-30 21:15:36.88033487 +0000 UTC m=+2219.945111267"
Oct 30 21:15:37.745389 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:37.745358    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh"
Oct 30 21:15:37.751527 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:37.751503    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh"
Oct 30 21:15:37.771696 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:37.771666    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh"
Oct 30 21:15:37.772783 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:37.772761    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh"
Oct 30 21:15:37.841580 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:37.841548    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-677b45-9b4nw"]
Oct 30 21:15:37.859099 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:37.859070    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-677b45-9b4nw"]
Oct 30 21:15:38.299737 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.299501    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-390b71-5698dc795b-2hp76" podUID="4a4313b7-0413-4625-9907-e12d701bd8fd" containerName="payload-processing" containerID="cri-o://c634c3ba4780c2513c5f992c3c41a9bb803071a75c5c71bf99eea35ddf376c5e" gracePeriod=25
Oct 30 21:15:38.367950 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.367850    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-390b71-7ff679864c-ghxkc" podUID="f1dcf961-808f-43d1-a978-3f9a8762006c" containerName="payload-pre-processing" containerID="cri-o://c1b4ac90d78ab627f7a0a228774a118950b0622e2bed50396d9fd52669239ff1" gracePeriod=25
Oct 30 21:15:38.484904 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.484879    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4"
Oct 30 21:15:38.489611 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.489587    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf"
Oct 30 21:15:38.494090 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.494064    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz"
Oct 30 21:15:38.582287 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.582249    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b85e68b-c6a9-4362-a6c0-93af13623e67-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b85e68b-c6a9-4362-a6c0-93af13623e67-workload-certs\") pod \"0b85e68b-c6a9-4362-a6c0-93af13623e67\" (UID: \"0b85e68b-c6a9-4362-a6c0-93af13623e67\") "
Oct 30 21:15:38.582493 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.582298    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b85e68b-c6a9-4362-a6c0-93af13623e67-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b85e68b-c6a9-4362-a6c0-93af13623e67-workload-socket\") pod \"0b85e68b-c6a9-4362-a6c0-93af13623e67\" (UID: \"0b85e68b-c6a9-4362-a6c0-93af13623e67\") "
Oct 30 21:15:38.582493 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.582329    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f-credential-socket\") pod \"d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f\" (UID: \"d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f\") "
Oct 30 21:15:38.582493 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.582369    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c59144f6-c298-4a9d-a3c3-7a2e81bb8967-istio-token\" (UniqueName: \"kubernetes.io/projected/c59144f6-c298-4a9d-a3c3-7a2e81bb8967-istio-token\") pod \"c59144f6-c298-4a9d-a3c3-7a2e81bb8967\" (UID: \"c59144f6-c298-4a9d-a3c3-7a2e81bb8967\") "
Oct 30 21:15:38.582493 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.582410    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b85e68b-c6a9-4362-a6c0-93af13623e67-kube-api-access-tdc2n\" (UniqueName: \"kubernetes.io/projected/0b85e68b-c6a9-4362-a6c0-93af13623e67-kube-api-access-tdc2n\") pod \"0b85e68b-c6a9-4362-a6c0-93af13623e67\" (UID: \"0b85e68b-c6a9-4362-a6c0-93af13623e67\") "
Oct 30 21:15:38.582493 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.582441    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b85e68b-c6a9-4362-a6c0-93af13623e67-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b85e68b-c6a9-4362-a6c0-93af13623e67-istio-envoy\") pod \"0b85e68b-c6a9-4362-a6c0-93af13623e67\" (UID: \"0b85e68b-c6a9-4362-a6c0-93af13623e67\") "
Oct 30 21:15:38.582493 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.582470    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f-istio-data\") pod \"d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f\" (UID: \"d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f\") "
Oct 30 21:15:38.582857 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.582512    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c59144f6-c298-4a9d-a3c3-7a2e81bb8967-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c59144f6-c298-4a9d-a3c3-7a2e81bb8967-istiod-ca-cert\") pod \"c59144f6-c298-4a9d-a3c3-7a2e81bb8967\" (UID: \"c59144f6-c298-4a9d-a3c3-7a2e81bb8967\") "
Oct 30 21:15:38.582857 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.582540    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b85e68b-c6a9-4362-a6c0-93af13623e67-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b85e68b-c6a9-4362-a6c0-93af13623e67-credential-socket\") pod \"0b85e68b-c6a9-4362-a6c0-93af13623e67\" (UID: \"0b85e68b-c6a9-4362-a6c0-93af13623e67\") "
Oct 30 21:15:38.582857 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.582569    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b85e68b-c6a9-4362-a6c0-93af13623e67-workload-certs" pod "0b85e68b-c6a9-4362-a6c0-93af13623e67" (UID: "0b85e68b-c6a9-4362-a6c0-93af13623e67"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:15:38.582857 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.582585    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f-istio-podinfo\") pod \"d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f\" (UID: \"d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f\") "
Oct 30 21:15:38.582857 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.582628    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c59144f6-c298-4a9d-a3c3-7a2e81bb8967-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c59144f6-c298-4a9d-a3c3-7a2e81bb8967-istio-data\") pod \"c59144f6-c298-4a9d-a3c3-7a2e81bb8967\" (UID: \"c59144f6-c298-4a9d-a3c3-7a2e81bb8967\") "
Oct 30 21:15:38.582857 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.582674    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b85e68b-c6a9-4362-a6c0-93af13623e67-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b85e68b-c6a9-4362-a6c0-93af13623e67-istiod-ca-cert\") pod \"0b85e68b-c6a9-4362-a6c0-93af13623e67\" (UID: \"0b85e68b-c6a9-4362-a6c0-93af13623e67\") "
Oct 30 21:15:38.582857 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.582707    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c59144f6-c298-4a9d-a3c3-7a2e81bb8967-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c59144f6-c298-4a9d-a3c3-7a2e81bb8967-istio-envoy\") pod \"c59144f6-c298-4a9d-a3c3-7a2e81bb8967\" (UID: \"c59144f6-c298-4a9d-a3c3-7a2e81bb8967\") "
Oct 30 21:15:38.582857 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.582738    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0b85e68b-c6a9-4362-a6c0-93af13623e67-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b85e68b-c6a9-4362-a6c0-93af13623e67-istio-podinfo\") pod \"0b85e68b-c6a9-4362-a6c0-93af13623e67\" (UID: \"0b85e68b-c6a9-4362-a6c0-93af13623e67\") "
Oct 30 21:15:38.582857 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.582775    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c59144f6-c298-4a9d-a3c3-7a2e81bb8967-kube-api-access-7v5vv\" (UniqueName: \"kubernetes.io/projected/c59144f6-c298-4a9d-a3c3-7a2e81bb8967-kube-api-access-7v5vv\") pod \"c59144f6-c298-4a9d-a3c3-7a2e81bb8967\" (UID: \"c59144f6-c298-4a9d-a3c3-7a2e81bb8967\") "
Oct 30 21:15:38.583410 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.582920    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f-workload-certs\") pod \"d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f\" (UID: \"d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f\") "
Oct 30 21:15:38.583410 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.583004    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f-istiod-ca-cert\") pod \"d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f\" (UID: \"d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f\") "
Oct 30 21:15:38.583410 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.583121    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c59144f6-c298-4a9d-a3c3-7a2e81bb8967-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c59144f6-c298-4a9d-a3c3-7a2e81bb8967-workload-socket\") pod \"c59144f6-c298-4a9d-a3c3-7a2e81bb8967\" (UID: \"c59144f6-c298-4a9d-a3c3-7a2e81bb8967\") "
Oct 30 21:15:38.583410 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.583136    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f-workload-certs" pod "d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f" (UID: "d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:15:38.583410 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.583192    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f-istio-token\" (UniqueName: \"kubernetes.io/projected/d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f-istio-token\") pod \"d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f\" (UID: \"d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f\") "
Oct 30 21:15:38.583410 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.583278    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f-kube-api-access-v9m6b\" (UniqueName: \"kubernetes.io/projected/d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f-kube-api-access-v9m6b\") pod \"d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f\" (UID: \"d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f\") "
Oct 30 21:15:38.583410 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.583316    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f-istio-envoy\") pod \"d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f\" (UID: \"d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f\") "
Oct 30 21:15:38.583410 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.583354    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c59144f6-c298-4a9d-a3c3-7a2e81bb8967-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c59144f6-c298-4a9d-a3c3-7a2e81bb8967-credential-socket\") pod \"c59144f6-c298-4a9d-a3c3-7a2e81bb8967\" (UID: \"c59144f6-c298-4a9d-a3c3-7a2e81bb8967\") "
Oct 30 21:15:38.583410 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.583388    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b85e68b-c6a9-4362-a6c0-93af13623e67-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b85e68b-c6a9-4362-a6c0-93af13623e67-istio-data\") pod \"0b85e68b-c6a9-4362-a6c0-93af13623e67\" (UID: \"0b85e68b-c6a9-4362-a6c0-93af13623e67\") "
Oct 30 21:15:38.583930 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.583540    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f-workload-socket\") pod \"d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f\" (UID: \"d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f\") "
Oct 30 21:15:38.583930 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.583631    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c59144f6-c298-4a9d-a3c3-7a2e81bb8967-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c59144f6-c298-4a9d-a3c3-7a2e81bb8967-istio-podinfo\") pod \"c59144f6-c298-4a9d-a3c3-7a2e81bb8967\" (UID: \"c59144f6-c298-4a9d-a3c3-7a2e81bb8967\") "
Oct 30 21:15:38.583930 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.583681    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b85e68b-c6a9-4362-a6c0-93af13623e67-istio-token\" (UniqueName: \"kubernetes.io/projected/0b85e68b-c6a9-4362-a6c0-93af13623e67-istio-token\") pod \"0b85e68b-c6a9-4362-a6c0-93af13623e67\" (UID: \"0b85e68b-c6a9-4362-a6c0-93af13623e67\") "
Oct 30 21:15:38.583930 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.583730    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c59144f6-c298-4a9d-a3c3-7a2e81bb8967-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c59144f6-c298-4a9d-a3c3-7a2e81bb8967-workload-certs\") pod \"c59144f6-c298-4a9d-a3c3-7a2e81bb8967\" (UID: \"c59144f6-c298-4a9d-a3c3-7a2e81bb8967\") "
Oct 30 21:15:38.584467 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.584445    5713 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f-workload-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:38.584556 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.584473    5713 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b85e68b-c6a9-4362-a6c0-93af13623e67-workload-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:38.585740 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.584956    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c59144f6-c298-4a9d-a3c3-7a2e81bb8967-workload-socket" pod "c59144f6-c298-4a9d-a3c3-7a2e81bb8967" (UID: "c59144f6-c298-4a9d-a3c3-7a2e81bb8967"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:15:38.585740 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.585096    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c59144f6-c298-4a9d-a3c3-7a2e81bb8967-istio-data" pod "c59144f6-c298-4a9d-a3c3-7a2e81bb8967" (UID: "c59144f6-c298-4a9d-a3c3-7a2e81bb8967"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:15:38.585740 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.585220    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f-credential-socket" pod "d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f" (UID: "d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:15:38.585740 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.585312    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f-istio-data" pod "d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f" (UID: "d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:15:38.585740 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.585337    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f-istiod-ca-cert" pod "d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f" (UID: "d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:38.585740 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.585452    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b85e68b-c6a9-4362-a6c0-93af13623e67-workload-socket" pod "0b85e68b-c6a9-4362-a6c0-93af13623e67" (UID: "0b85e68b-c6a9-4362-a6c0-93af13623e67"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:15:38.585740 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.585503    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c59144f6-c298-4a9d-a3c3-7a2e81bb8967-workload-certs" pod "c59144f6-c298-4a9d-a3c3-7a2e81bb8967" (UID: "c59144f6-c298-4a9d-a3c3-7a2e81bb8967"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:15:38.585740 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.585661    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b85e68b-c6a9-4362-a6c0-93af13623e67-credential-socket" pod "0b85e68b-c6a9-4362-a6c0-93af13623e67" (UID: "0b85e68b-c6a9-4362-a6c0-93af13623e67"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:15:38.585740 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.585695    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c59144f6-c298-4a9d-a3c3-7a2e81bb8967-istiod-ca-cert" pod "c59144f6-c298-4a9d-a3c3-7a2e81bb8967" (UID: "c59144f6-c298-4a9d-a3c3-7a2e81bb8967"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:38.585740 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.585734    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c59144f6-c298-4a9d-a3c3-7a2e81bb8967-credential-socket" pod "c59144f6-c298-4a9d-a3c3-7a2e81bb8967" (UID: "c59144f6-c298-4a9d-a3c3-7a2e81bb8967"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:15:38.586581 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.585908    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f-workload-socket" pod "d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f" (UID: "d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:15:38.586581 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.586081    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b85e68b-c6a9-4362-a6c0-93af13623e67-istio-data" pod "0b85e68b-c6a9-4362-a6c0-93af13623e67" (UID: "0b85e68b-c6a9-4362-a6c0-93af13623e67"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:15:38.587111 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.587080    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b85e68b-c6a9-4362-a6c0-93af13623e67-kube-api-access-tdc2n" pod "0b85e68b-c6a9-4362-a6c0-93af13623e67" (UID: "0b85e68b-c6a9-4362-a6c0-93af13623e67"). InnerVolumeSpecName "kube-api-access-tdc2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:38.587583 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.587555    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b85e68b-c6a9-4362-a6c0-93af13623e67-istio-envoy" pod "0b85e68b-c6a9-4362-a6c0-93af13623e67" (UID: "0b85e68b-c6a9-4362-a6c0-93af13623e67"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:15:38.587814 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.587759    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f-istio-envoy" pod "d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f" (UID: "d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:15:38.587950 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.587929    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b85e68b-c6a9-4362-a6c0-93af13623e67-istiod-ca-cert" pod "0b85e68b-c6a9-4362-a6c0-93af13623e67" (UID: "0b85e68b-c6a9-4362-a6c0-93af13623e67"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:38.589607 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.589580    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f-istio-token" pod "d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f" (UID: "d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:38.590919 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.590847    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c59144f6-c298-4a9d-a3c3-7a2e81bb8967-istio-token" pod "c59144f6-c298-4a9d-a3c3-7a2e81bb8967" (UID: "c59144f6-c298-4a9d-a3c3-7a2e81bb8967"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:38.592485 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.592458    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c59144f6-c298-4a9d-a3c3-7a2e81bb8967-istio-envoy" pod "c59144f6-c298-4a9d-a3c3-7a2e81bb8967" (UID: "c59144f6-c298-4a9d-a3c3-7a2e81bb8967"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:15:38.592575 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.592482    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b85e68b-c6a9-4362-a6c0-93af13623e67-istio-podinfo" pod "0b85e68b-c6a9-4362-a6c0-93af13623e67" (UID: "0b85e68b-c6a9-4362-a6c0-93af13623e67"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:15:38.592723 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.592705    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b85e68b-c6a9-4362-a6c0-93af13623e67-istio-token" pod "0b85e68b-c6a9-4362-a6c0-93af13623e67" (UID: "0b85e68b-c6a9-4362-a6c0-93af13623e67"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:38.601934 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.601904    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f-kube-api-access-v9m6b" pod "d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f" (UID: "d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f"). InnerVolumeSpecName "kube-api-access-v9m6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:38.602085 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.602049    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c59144f6-c298-4a9d-a3c3-7a2e81bb8967-istio-podinfo" pod "c59144f6-c298-4a9d-a3c3-7a2e81bb8967" (UID: "c59144f6-c298-4a9d-a3c3-7a2e81bb8967"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:15:38.602182 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.602165    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c59144f6-c298-4a9d-a3c3-7a2e81bb8967-kube-api-access-7v5vv" pod "c59144f6-c298-4a9d-a3c3-7a2e81bb8967" (UID: "c59144f6-c298-4a9d-a3c3-7a2e81bb8967"). InnerVolumeSpecName "kube-api-access-7v5vv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:38.603340 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.603314    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f-istio-podinfo" pod "d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f" (UID: "d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:15:38.644838 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.639477    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-390b71-5698dc795b-2hp76"
Oct 30 21:15:38.685844 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.685807    5713 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f-workload-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:38.686019 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.685848    5713 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c59144f6-c298-4a9d-a3c3-7a2e81bb8967-istio-podinfo\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:38.686019 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.685867    5713 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b85e68b-c6a9-4362-a6c0-93af13623e67-istio-token\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:38.686019 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.685881    5713 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c59144f6-c298-4a9d-a3c3-7a2e81bb8967-workload-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:38.686019 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.685894    5713 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b85e68b-c6a9-4362-a6c0-93af13623e67-workload-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:38.686019 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.685907    5713 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f-credential-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:38.686019 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.685922    5713 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c59144f6-c298-4a9d-a3c3-7a2e81bb8967-istio-token\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:38.686019 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.685935    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdc2n\" (UniqueName: \"kubernetes.io/projected/0b85e68b-c6a9-4362-a6c0-93af13623e67-kube-api-access-tdc2n\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:38.686019 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.685950    5713 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b85e68b-c6a9-4362-a6c0-93af13623e67-istio-envoy\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:38.686019 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.685963    5713 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f-istio-data\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:38.686019 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.685977    5713 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c59144f6-c298-4a9d-a3c3-7a2e81bb8967-istiod-ca-cert\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:38.686019 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.685991    5713 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b85e68b-c6a9-4362-a6c0-93af13623e67-credential-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:38.686019 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.686004    5713 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f-istio-podinfo\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:38.686019 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.686017    5713 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c59144f6-c298-4a9d-a3c3-7a2e81bb8967-istio-data\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:38.686757 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.686030    5713 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b85e68b-c6a9-4362-a6c0-93af13623e67-istiod-ca-cert\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:38.686757 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.686045    5713 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c59144f6-c298-4a9d-a3c3-7a2e81bb8967-istio-envoy\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:38.686757 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.686058    5713 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b85e68b-c6a9-4362-a6c0-93af13623e67-istio-podinfo\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:38.686757 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.686070    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v5vv\" (UniqueName: \"kubernetes.io/projected/c59144f6-c298-4a9d-a3c3-7a2e81bb8967-kube-api-access-7v5vv\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:38.686757 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.686086    5713 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f-istiod-ca-cert\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:38.686757 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.686100    5713 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c59144f6-c298-4a9d-a3c3-7a2e81bb8967-workload-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:38.686757 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.686114    5713 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f-istio-token\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:38.686757 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.686128    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9m6b\" (UniqueName: \"kubernetes.io/projected/d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f-kube-api-access-v9m6b\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:38.686757 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.686143    5713 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f-istio-envoy\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:38.686757 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.686158    5713 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c59144f6-c298-4a9d-a3c3-7a2e81bb8967-credential-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:38.686757 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.686173    5713 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b85e68b-c6a9-4362-a6c0-93af13623e67-istio-data\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:38.688879 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.688850    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-390b71-7ff679864c-ghxkc"
Oct 30 21:15:38.783968 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.783928    5713 generic.go:356] "Generic (PLEG): container finished" podID="f1dcf961-808f-43d1-a978-3f9a8762006c" containerID="c1b4ac90d78ab627f7a0a228774a118950b0622e2bed50396d9fd52669239ff1" exitCode=0
Oct 30 21:15:38.784425 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.783986    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-390b71-7ff679864c-ghxkc"
Oct 30 21:15:38.784425 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.784035    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-390b71-7ff679864c-ghxkc" event={"ID":"f1dcf961-808f-43d1-a978-3f9a8762006c","Type":"ContainerDied","Data":"c1b4ac90d78ab627f7a0a228774a118950b0622e2bed50396d9fd52669239ff1"}
Oct 30 21:15:38.784425 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.784113    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-390b71-7ff679864c-ghxkc" event={"ID":"f1dcf961-808f-43d1-a978-3f9a8762006c","Type":"ContainerDied","Data":"58ccd16b064fbd72050fed86190f808c7fb1203c617e1911d712733ad5d46941"}
Oct 30 21:15:38.784425 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.784149    5713 scope.go:122] "RemoveContainer" containerID="c1b4ac90d78ab627f7a0a228774a118950b0622e2bed50396d9fd52669239ff1"
Oct 30 21:15:38.786669 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.786638    5713 generic.go:356] "Generic (PLEG): container finished" podID="d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f" containerID="cdd26ac7c03c95d666f43e2e53a5b8e81a441eb72f86201ea2b8f3da1d35b591" exitCode=0
Oct 30 21:15:38.786842 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.786680    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz" event={"ID":"d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f","Type":"ContainerDied","Data":"cdd26ac7c03c95d666f43e2e53a5b8e81a441eb72f86201ea2b8f3da1d35b591"}
Oct 30 21:15:38.786842 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.786769    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz" event={"ID":"d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f","Type":"ContainerDied","Data":"8dc38cde0b3ba97470fbe62b504ee8e6d60d150e2a4c1b59d1609416d0ed94c1"}
Oct 30 21:15:38.786950 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.786934    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz"
Oct 30 21:15:38.787102 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.787083    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a4313b7-0413-4625-9907-e12d701bd8fd-kube-api-access-ptjfc\" (UniqueName: \"kubernetes.io/projected/4a4313b7-0413-4625-9907-e12d701bd8fd-kube-api-access-ptjfc\") pod \"4a4313b7-0413-4625-9907-e12d701bd8fd\" (UID: \"4a4313b7-0413-4625-9907-e12d701bd8fd\") "
Oct 30 21:15:38.787491 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.787335    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a4313b7-0413-4625-9907-e12d701bd8fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a4313b7-0413-4625-9907-e12d701bd8fd-plugins-config-volume\") pod \"4a4313b7-0413-4625-9907-e12d701bd8fd\" (UID: \"4a4313b7-0413-4625-9907-e12d701bd8fd\") "
Oct 30 21:15:38.787491 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.787414    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1dcf961-808f-43d1-a978-3f9a8762006c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1dcf961-808f-43d1-a978-3f9a8762006c-plugins-config-volume\") pod \"f1dcf961-808f-43d1-a978-3f9a8762006c\" (UID: \"f1dcf961-808f-43d1-a978-3f9a8762006c\") "
Oct 30 21:15:38.787491 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.787447    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1dcf961-808f-43d1-a978-3f9a8762006c-kube-api-access-b5zc5\" (UniqueName: \"kubernetes.io/projected/f1dcf961-808f-43d1-a978-3f9a8762006c-kube-api-access-b5zc5\") pod \"f1dcf961-808f-43d1-a978-3f9a8762006c\" (UID: \"f1dcf961-808f-43d1-a978-3f9a8762006c\") "
Oct 30 21:15:38.788359 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.788306    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a4313b7-0413-4625-9907-e12d701bd8fd-plugins-config-volume" pod "4a4313b7-0413-4625-9907-e12d701bd8fd" (UID: "4a4313b7-0413-4625-9907-e12d701bd8fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:38.788359 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.788325    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1dcf961-808f-43d1-a978-3f9a8762006c-plugins-config-volume" pod "f1dcf961-808f-43d1-a978-3f9a8762006c" (UID: "f1dcf961-808f-43d1-a978-3f9a8762006c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:38.790141 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.790111    5713 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a4313b7-0413-4625-9907-e12d701bd8fd-plugins-config-volume\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:38.790223 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.790148    5713 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1dcf961-808f-43d1-a978-3f9a8762006c-plugins-config-volume\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:38.790289 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.790244    5713 generic.go:356] "Generic (PLEG): container finished" podID="c59144f6-c298-4a9d-a3c3-7a2e81bb8967" containerID="0fcb783b7658411186f3e3c318cf06459e180201e511a6ef12564779e28124fe" exitCode=0
Oct 30 21:15:38.790601 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.790579    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf"
Oct 30 21:15:38.791948 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.791898    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf" event={"ID":"c59144f6-c298-4a9d-a3c3-7a2e81bb8967","Type":"ContainerDied","Data":"0fcb783b7658411186f3e3c318cf06459e180201e511a6ef12564779e28124fe"}
Oct 30 21:15:38.792115 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.791976    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf" event={"ID":"c59144f6-c298-4a9d-a3c3-7a2e81bb8967","Type":"ContainerDied","Data":"89c29e1944e8726a040ff32d992f7784032cdb388c6e47ae84b3f91706f6ea4a"}
Oct 30 21:15:38.795986 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.795958    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a4313b7-0413-4625-9907-e12d701bd8fd-kube-api-access-ptjfc" pod "4a4313b7-0413-4625-9907-e12d701bd8fd" (UID: "4a4313b7-0413-4625-9907-e12d701bd8fd"). InnerVolumeSpecName "kube-api-access-ptjfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:38.796099 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.796001    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1dcf961-808f-43d1-a978-3f9a8762006c-kube-api-access-b5zc5" pod "f1dcf961-808f-43d1-a978-3f9a8762006c" (UID: "f1dcf961-808f-43d1-a978-3f9a8762006c"). InnerVolumeSpecName "kube-api-access-b5zc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:38.797600 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.797421    5713 generic.go:356] "Generic (PLEG): container finished" podID="4a4313b7-0413-4625-9907-e12d701bd8fd" containerID="c634c3ba4780c2513c5f992c3c41a9bb803071a75c5c71bf99eea35ddf376c5e" exitCode=0
Oct 30 21:15:38.797600 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.797489    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-390b71-5698dc795b-2hp76"
Oct 30 21:15:38.797764 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.797610    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-390b71-5698dc795b-2hp76" event={"ID":"4a4313b7-0413-4625-9907-e12d701bd8fd","Type":"ContainerDied","Data":"c634c3ba4780c2513c5f992c3c41a9bb803071a75c5c71bf99eea35ddf376c5e"}
Oct 30 21:15:38.797764 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.797658    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-390b71-5698dc795b-2hp76" event={"ID":"4a4313b7-0413-4625-9907-e12d701bd8fd","Type":"ContainerDied","Data":"6ca761c9e7162594d5abd6d2f5a99af217c0aa4b2df6db3ad221ee67285aba38"}
Oct 30 21:15:38.799827 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.799769    5713 generic.go:356] "Generic (PLEG): container finished" podID="0b85e68b-c6a9-4362-a6c0-93af13623e67" containerID="a79c0d0d0481c3988fda6012603ca6917b6fb98b676c2ae18668cddb5d157a2b" exitCode=0
Oct 30 21:15:38.800062 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.799974    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4" event={"ID":"0b85e68b-c6a9-4362-a6c0-93af13623e67","Type":"ContainerDied","Data":"a79c0d0d0481c3988fda6012603ca6917b6fb98b676c2ae18668cddb5d157a2b"}
Oct 30 21:15:38.800062 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.800030    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4" event={"ID":"0b85e68b-c6a9-4362-a6c0-93af13623e67","Type":"ContainerDied","Data":"444e694acdb929f8b45a6ea551b99d32c34b77297857885aa1ac73368a5cfee0"}
Oct 30 21:15:38.800299 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.800280    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4"
Oct 30 21:15:38.805978 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.805934    5713 scope.go:122] "RemoveContainer" containerID="c1b4ac90d78ab627f7a0a228774a118950b0622e2bed50396d9fd52669239ff1"
Oct 30 21:15:38.806336 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:15:38.806307    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1b4ac90d78ab627f7a0a228774a118950b0622e2bed50396d9fd52669239ff1\": container with ID starting with c1b4ac90d78ab627f7a0a228774a118950b0622e2bed50396d9fd52669239ff1 not found: ID does not exist" containerID="c1b4ac90d78ab627f7a0a228774a118950b0622e2bed50396d9fd52669239ff1"
Oct 30 21:15:38.806453 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.806349    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1b4ac90d78ab627f7a0a228774a118950b0622e2bed50396d9fd52669239ff1"} err="failed to get container status \"c1b4ac90d78ab627f7a0a228774a118950b0622e2bed50396d9fd52669239ff1\": rpc error: code = NotFound desc = could not find container \"c1b4ac90d78ab627f7a0a228774a118950b0622e2bed50396d9fd52669239ff1\": container with ID starting with c1b4ac90d78ab627f7a0a228774a118950b0622e2bed50396d9fd52669239ff1 not found: ID does not exist"
Oct 30 21:15:38.806453 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.806371    5713 scope.go:122] "RemoveContainer" containerID="cdd26ac7c03c95d666f43e2e53a5b8e81a441eb72f86201ea2b8f3da1d35b591"
Oct 30 21:15:38.822284 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.822260    5713 scope.go:122] "RemoveContainer" containerID="cdd26ac7c03c95d666f43e2e53a5b8e81a441eb72f86201ea2b8f3da1d35b591"
Oct 30 21:15:38.822675 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:15:38.822650    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdd26ac7c03c95d666f43e2e53a5b8e81a441eb72f86201ea2b8f3da1d35b591\": container with ID starting with cdd26ac7c03c95d666f43e2e53a5b8e81a441eb72f86201ea2b8f3da1d35b591 not found: ID does not exist" containerID="cdd26ac7c03c95d666f43e2e53a5b8e81a441eb72f86201ea2b8f3da1d35b591"
Oct 30 21:15:38.822768 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.822686    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdd26ac7c03c95d666f43e2e53a5b8e81a441eb72f86201ea2b8f3da1d35b591"} err="failed to get container status \"cdd26ac7c03c95d666f43e2e53a5b8e81a441eb72f86201ea2b8f3da1d35b591\": rpc error: code = NotFound desc = could not find container \"cdd26ac7c03c95d666f43e2e53a5b8e81a441eb72f86201ea2b8f3da1d35b591\": container with ID starting with cdd26ac7c03c95d666f43e2e53a5b8e81a441eb72f86201ea2b8f3da1d35b591 not found: ID does not exist"
Oct 30 21:15:38.822768 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.822710    5713 scope.go:122] "RemoveContainer" containerID="0fcb783b7658411186f3e3c318cf06459e180201e511a6ef12564779e28124fe"
Oct 30 21:15:38.834250 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.834230    5713 scope.go:122] "RemoveContainer" containerID="0fcb783b7658411186f3e3c318cf06459e180201e511a6ef12564779e28124fe"
Oct 30 21:15:38.834536 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:15:38.834516    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fcb783b7658411186f3e3c318cf06459e180201e511a6ef12564779e28124fe\": container with ID starting with 0fcb783b7658411186f3e3c318cf06459e180201e511a6ef12564779e28124fe not found: ID does not exist" containerID="0fcb783b7658411186f3e3c318cf06459e180201e511a6ef12564779e28124fe"
Oct 30 21:15:38.834598 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.834545    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fcb783b7658411186f3e3c318cf06459e180201e511a6ef12564779e28124fe"} err="failed to get container status \"0fcb783b7658411186f3e3c318cf06459e180201e511a6ef12564779e28124fe\": rpc error: code = NotFound desc = could not find container \"0fcb783b7658411186f3e3c318cf06459e180201e511a6ef12564779e28124fe\": container with ID starting with 0fcb783b7658411186f3e3c318cf06459e180201e511a6ef12564779e28124fe not found: ID does not exist"
Oct 30 21:15:38.834598 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.834563    5713 scope.go:122] "RemoveContainer" containerID="c634c3ba4780c2513c5f992c3c41a9bb803071a75c5c71bf99eea35ddf376c5e"
Oct 30 21:15:38.845172 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.845151    5713 scope.go:122] "RemoveContainer" containerID="c634c3ba4780c2513c5f992c3c41a9bb803071a75c5c71bf99eea35ddf376c5e"
Oct 30 21:15:38.845493 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:15:38.845471    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c634c3ba4780c2513c5f992c3c41a9bb803071a75c5c71bf99eea35ddf376c5e\": container with ID starting with c634c3ba4780c2513c5f992c3c41a9bb803071a75c5c71bf99eea35ddf376c5e not found: ID does not exist" containerID="c634c3ba4780c2513c5f992c3c41a9bb803071a75c5c71bf99eea35ddf376c5e"
Oct 30 21:15:38.845590 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.845501    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c634c3ba4780c2513c5f992c3c41a9bb803071a75c5c71bf99eea35ddf376c5e"} err="failed to get container status \"c634c3ba4780c2513c5f992c3c41a9bb803071a75c5c71bf99eea35ddf376c5e\": rpc error: code = NotFound desc = could not find container \"c634c3ba4780c2513c5f992c3c41a9bb803071a75c5c71bf99eea35ddf376c5e\": container with ID starting with c634c3ba4780c2513c5f992c3c41a9bb803071a75c5c71bf99eea35ddf376c5e not found: ID does not exist"
Oct 30 21:15:38.845590 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.845520    5713 scope.go:122] "RemoveContainer" containerID="a79c0d0d0481c3988fda6012603ca6917b6fb98b676c2ae18668cddb5d157a2b"
Oct 30 21:15:38.857279 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.857260    5713 scope.go:122] "RemoveContainer" containerID="a79c0d0d0481c3988fda6012603ca6917b6fb98b676c2ae18668cddb5d157a2b"
Oct 30 21:15:38.857531 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:15:38.857514    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a79c0d0d0481c3988fda6012603ca6917b6fb98b676c2ae18668cddb5d157a2b\": container with ID starting with a79c0d0d0481c3988fda6012603ca6917b6fb98b676c2ae18668cddb5d157a2b not found: ID does not exist" containerID="a79c0d0d0481c3988fda6012603ca6917b6fb98b676c2ae18668cddb5d157a2b"
Oct 30 21:15:38.857591 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.857543    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a79c0d0d0481c3988fda6012603ca6917b6fb98b676c2ae18668cddb5d157a2b"} err="failed to get container status \"a79c0d0d0481c3988fda6012603ca6917b6fb98b676c2ae18668cddb5d157a2b\": rpc error: code = NotFound desc = could not find container \"a79c0d0d0481c3988fda6012603ca6917b6fb98b676c2ae18668cddb5d157a2b\": container with ID starting with a79c0d0d0481c3988fda6012603ca6917b6fb98b676c2ae18668cddb5d157a2b not found: ID does not exist"
Oct 30 21:15:38.891765 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.891740    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5zc5\" (UniqueName: \"kubernetes.io/projected/f1dcf961-808f-43d1-a978-3f9a8762006c-kube-api-access-b5zc5\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:38.892056 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:38.891769    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptjfc\" (UniqueName: \"kubernetes.io/projected/4a4313b7-0413-4625-9907-e12d701bd8fd-kube-api-access-ptjfc\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:39.067228 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.067195    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b868dd4-50b6-44d9-b296-cf01ca7d1d92" path="/var/lib/kubelet/pods/9b868dd4-50b6-44d9-b296-cf01ca7d1d92/volumes"
Oct 30 21:15:39.102615 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.102582    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-390b71-5698dc795b-2hp76"]
Oct 30 21:15:39.119268 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.119224    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-390b71-5698dc795b-2hp76"]
Oct 30 21:15:39.203096 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.203062    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5"]
Oct 30 21:15:39.203662 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.203648    5713 cpu_manager.go:414] "containerMap: removing container" podUID="f59ff9d3-2927-4021-9103-fd2f60f7b655" containerName="cleanup"
Oct 30 21:15:39.203727 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.203665    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f59ff9d3-2927-4021-9103-fd2f60f7b655" containerName="cleanup"
Oct 30 21:15:39.203727 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.203681    5713 cpu_manager.go:414] "containerMap: removing container" podUID="16adca55-5cc2-4452-aba7-3e891c005f98" containerName="cleanup"
Oct 30 21:15:39.203727 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.203687    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16adca55-5cc2-4452-aba7-3e891c005f98" containerName="cleanup"
Oct 30 21:15:39.203727 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.203697    5713 cpu_manager.go:414] "containerMap: removing container" podUID="9970e7c4-7661-47c5-a6df-4de2bea436e2" containerName="cleanup"
Oct 30 21:15:39.203727 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.203702    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9970e7c4-7661-47c5-a6df-4de2bea436e2" containerName="cleanup"
Oct 30 21:15:39.203727 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.203719    5713 cpu_manager.go:414] "containerMap: removing container" podUID="4a4313b7-0413-4625-9907-e12d701bd8fd" containerName="payload-processing"
Oct 30 21:15:39.203727 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.203724    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a4313b7-0413-4625-9907-e12d701bd8fd" containerName="payload-processing"
Oct 30 21:15:39.204133 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.203734    5713 cpu_manager.go:414] "containerMap: removing container" podUID="0b85e68b-c6a9-4362-a6c0-93af13623e67" containerName="istio-proxy"
Oct 30 21:15:39.204133 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.203739    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b85e68b-c6a9-4362-a6c0-93af13623e67" containerName="istio-proxy"
Oct 30 21:15:39.204133 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.203764    5713 cpu_manager.go:414] "containerMap: removing container" podUID="c59144f6-c298-4a9d-a3c3-7a2e81bb8967" containerName="istio-proxy"
Oct 30 21:15:39.204133 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.203770    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c59144f6-c298-4a9d-a3c3-7a2e81bb8967" containerName="istio-proxy"
Oct 30 21:15:39.204133 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.203782    5713 cpu_manager.go:414] "containerMap: removing container" podUID="d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f" containerName="istio-proxy"
Oct 30 21:15:39.204133 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.203807    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f" containerName="istio-proxy"
Oct 30 21:15:39.204133 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.203822    5713 cpu_manager.go:414] "containerMap: removing container" podUID="a8d5eebe-b80b-4b99-ace8-7e87724decaf" containerName="cleanup"
Oct 30 21:15:39.204133 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.203827    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8d5eebe-b80b-4b99-ace8-7e87724decaf" containerName="cleanup"
Oct 30 21:15:39.204133 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.203836    5713 cpu_manager.go:414] "containerMap: removing container" podUID="f1dcf961-808f-43d1-a978-3f9a8762006c" containerName="payload-pre-processing"
Oct 30 21:15:39.204133 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.203841    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1dcf961-808f-43d1-a978-3f9a8762006c" containerName="payload-pre-processing"
Oct 30 21:15:39.204133 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.203942    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1dcf961-808f-43d1-a978-3f9a8762006c" containerName="payload-pre-processing"
Oct 30 21:15:39.204133 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.203952    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="16adca55-5cc2-4452-aba7-3e891c005f98" containerName="cleanup"
Oct 30 21:15:39.204133 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.203959    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa46459a-d46e-4af4-bdfc-d6872367429d" containerName="cleanup"
Oct 30 21:15:39.204133 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.203966    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a4313b7-0413-4625-9907-e12d701bd8fd" containerName="payload-processing"
Oct 30 21:15:39.204133 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.203971    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b85e68b-c6a9-4362-a6c0-93af13623e67" containerName="istio-proxy"
Oct 30 21:15:39.204133 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.203979    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="9970e7c4-7661-47c5-a6df-4de2bea436e2" containerName="cleanup"
Oct 30 21:15:39.204133 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.203986    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3a522bc-746d-4cdf-940d-ccb562ade3c0" containerName="cleanup"
Oct 30 21:15:39.204133 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.203993    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="c59144f6-c298-4a9d-a3c3-7a2e81bb8967" containerName="istio-proxy"
Oct 30 21:15:39.204133 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.203999    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f" containerName="istio-proxy"
Oct 30 21:15:39.210533 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.210512    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5"
Oct 30 21:15:39.261423 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.261389    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz"]
Oct 30 21:15:39.276124 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.276093    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5"]
Oct 30 21:15:39.280648 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.280613    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-bz7rz"]
Oct 30 21:15:39.298698 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.298664    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a-workload-socket\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5\" (UID: \"fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5"
Oct 30 21:15:39.298908 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.298817    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a-istio-token\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5\" (UID: \"fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5"
Oct 30 21:15:39.298908 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.298869    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a-credential-socket\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5\" (UID: \"fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5"
Oct 30 21:15:39.299047 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.298964    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a-istio-podinfo\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5\" (UID: \"fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5"
Oct 30 21:15:39.299047 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.298992    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv6wc\" (UniqueName: \"kubernetes.io/projected/fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a-kube-api-access-wv6wc\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5\" (UID: \"fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5"
Oct 30 21:15:39.299163 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.299066    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a-workload-certs\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5\" (UID: \"fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5"
Oct 30 21:15:39.299163 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.299104    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a-istio-data\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5\" (UID: \"fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5"
Oct 30 21:15:39.299163 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.299140    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a-istio-envoy\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5\" (UID: \"fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5"
Oct 30 21:15:39.299346 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.299239    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a-istiod-ca-cert\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5\" (UID: \"fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5"
Oct 30 21:15:39.400705 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.400609    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a-istio-token\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5\" (UID: \"fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5"
Oct 30 21:15:39.400967 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.400701    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a-credential-socket\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5\" (UID: \"fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5"
Oct 30 21:15:39.400967 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.400762    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a-istio-podinfo\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5\" (UID: \"fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5"
Oct 30 21:15:39.400967 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.400829    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv6wc\" (UniqueName: \"kubernetes.io/projected/fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a-kube-api-access-wv6wc\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5\" (UID: \"fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5"
Oct 30 21:15:39.400967 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.400904    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a-workload-certs\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5\" (UID: \"fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5"
Oct 30 21:15:39.400967 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.400939    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a-istio-data\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5\" (UID: \"fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5"
Oct 30 21:15:39.401258 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.400978    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a-istio-envoy\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5\" (UID: \"fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5"
Oct 30 21:15:39.401258 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.401067    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a-istiod-ca-cert\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5\" (UID: \"fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5"
Oct 30 21:15:39.401258 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.401132    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a-workload-socket\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5\" (UID: \"fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5"
Oct 30 21:15:39.401258 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.401217    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a-credential-socket\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5\" (UID: \"fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5"
Oct 30 21:15:39.401459 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.401275    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a-workload-certs\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5\" (UID: \"fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5"
Oct 30 21:15:39.402114 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.402082    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a-istio-data\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5\" (UID: \"fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5"
Oct 30 21:15:39.402603 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.402260    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a-workload-socket\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5\" (UID: \"fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5"
Oct 30 21:15:39.402603 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.402265    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a-istiod-ca-cert\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5\" (UID: \"fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5"
Oct 30 21:15:39.403775 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.403746    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a-istio-envoy\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5\" (UID: \"fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5"
Oct 30 21:15:39.404228 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.404209    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a-istio-podinfo\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5\" (UID: \"fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5"
Oct 30 21:15:39.457671 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.457634    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a-istio-token\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5\" (UID: \"fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5"
Oct 30 21:15:39.458869 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.458851    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv6wc\" (UniqueName: \"kubernetes.io/projected/fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a-kube-api-access-wv6wc\") pod \"e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5\" (UID: \"fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a\") " pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5"
Oct 30 21:15:39.514666 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.514626    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf"]
Oct 30 21:15:39.517961 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.517935    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-x6hwf"]
Oct 30 21:15:39.524854 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.524838    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5"
Oct 30 21:15:39.643468 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.643422    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4"]
Oct 30 21:15:39.649622 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.649002    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-7eaf74-openshift-default-69c8f48765-pjnt4"]
Oct 30 21:15:39.873286 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.873241    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-390b71-7ff679864c-ghxkc"]
Oct 30 21:15:39.879706 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.879665    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-390b71-7ff679864c-ghxkc"]
Oct 30 21:15:39.913189 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.913150    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5"]
Oct 30 21:15:39.914066 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:15:39.914041    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd2ddd20_e0d4_4bbd_80c3_f953e9e8025a.slice/crio-6934c44b2b92d9a67ec17e544173e660fc1843488d81848b10bdc3dee5adbea3 WatchSource:0}: Error finding container 6934c44b2b92d9a67ec17e544173e660fc1843488d81848b10bdc3dee5adbea3: Status 404 returned error can't find the container with id 6934c44b2b92d9a67ec17e544173e660fc1843488d81848b10bdc3dee5adbea3
Oct 30 21:15:39.916962 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.916923    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:15:39.917051 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.917035    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:15:39.917097 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:39.917082    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:15:40.822196 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:40.822158    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5" event={"ID":"fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a","Type":"ContainerStarted","Data":"8708f005a53b67ee2c563425cef2b4d8b9b8db85761dbfde3607262558f686c5"}
Oct 30 21:15:40.822394 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:40.822222    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5" event={"ID":"fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a","Type":"ContainerStarted","Data":"6934c44b2b92d9a67ec17e544173e660fc1843488d81848b10bdc3dee5adbea3"}
Oct 30 21:15:41.068880 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:41.068847    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b85e68b-c6a9-4362-a6c0-93af13623e67" path="/var/lib/kubelet/pods/0b85e68b-c6a9-4362-a6c0-93af13623e67/volumes"
Oct 30 21:15:41.069433 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:41.069413    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a4313b7-0413-4625-9907-e12d701bd8fd" path="/var/lib/kubelet/pods/4a4313b7-0413-4625-9907-e12d701bd8fd/volumes"
Oct 30 21:15:41.069977 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:41.069959    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c59144f6-c298-4a9d-a3c3-7a2e81bb8967" path="/var/lib/kubelet/pods/c59144f6-c298-4a9d-a3c3-7a2e81bb8967/volumes"
Oct 30 21:15:41.070756 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:41.070741    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f" path="/var/lib/kubelet/pods/d34f59c3-1aa3-4673-a9ac-4a00b5bfd23f/volumes"
Oct 30 21:15:41.071169 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:41.071155    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1dcf961-808f-43d1-a978-3f9a8762006c" path="/var/lib/kubelet/pods/f1dcf961-808f-43d1-a978-3f9a8762006c/volumes"
Oct 30 21:15:41.525958 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:41.525923    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5"
Oct 30 21:15:41.527712 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:41.527683    5713 patch_prober.go:29] interesting pod/e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body=
Oct 30 21:15:41.527867 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:41.527753    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5" podUID="fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused"
Oct 30 21:15:42.525423 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:42.525386    5713 patch_prober.go:29] interesting pod/e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body=
Oct 30 21:15:42.525891 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:42.525440    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5" podUID="fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused"
Oct 30 21:15:43.526034 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:43.525991    5713 patch_prober.go:29] interesting pod/e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body=
Oct 30 21:15:43.526534 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:43.526069    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5" podUID="fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused"
Oct 30 21:15:44.175813 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:44.174289    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5" podStartSLOduration=5.17426465 podStartE2EDuration="5.17426465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:40.950096767 +0000 UTC m=+2224.014873165" watchObservedRunningTime="2026-10-30 21:15:44.17426465 +0000 UTC m=+2227.239041049"
Oct 30 21:15:44.176815 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:44.176075    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d1b889bd-bb457476c-9nz29"]
Oct 30 21:15:44.187812 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:44.184906    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d1b889bd-bb457476c-9nz29"
Oct 30 21:15:44.240820 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:44.238521    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d1b889bd-bb457476c-9nz29"]
Oct 30 21:15:44.240820 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:44.238798    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa5ce369-e5ae-4099-b215-20dad86b5c3d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d1b889bd-bb457476c-9nz29\" (UID: \"aa5ce369-e5ae-4099-b215-20dad86b5c3d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d1b889bd-bb457476c-9nz29"
Oct 30 21:15:44.240820 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:44.239058    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqf9l\" (UniqueName: \"kubernetes.io/projected/aa5ce369-e5ae-4099-b215-20dad86b5c3d-kube-api-access-zqf9l\") pod \"payload-pre-processing-e2e-mt-d1b889bd-bb457476c-9nz29\" (UID: \"aa5ce369-e5ae-4099-b215-20dad86b5c3d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d1b889bd-bb457476c-9nz29"
Oct 30 21:15:44.315571 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:44.313129    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d1b889bd-7d887b46cb-kvshd"]
Oct 30 21:15:44.315571 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:44.315560    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-d1b889bd\""
Oct 30 21:15:44.319910 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:44.319869    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-d1b889bd-dockercfg-ffdl4\""
Oct 30 21:15:44.323773 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:44.323730    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d1b889bd-7d887b46cb-kvshd"
Oct 30 21:15:44.376035 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:44.355824    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa5ce369-e5ae-4099-b215-20dad86b5c3d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d1b889bd-bb457476c-9nz29\" (UID: \"aa5ce369-e5ae-4099-b215-20dad86b5c3d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d1b889bd-bb457476c-9nz29"
Oct 30 21:15:44.376035 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:44.362463    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d1b889bd-7d887b46cb-kvshd"]
Oct 30 21:15:44.376035 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:44.374220    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa5ce369-e5ae-4099-b215-20dad86b5c3d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d1b889bd-bb457476c-9nz29\" (UID: \"aa5ce369-e5ae-4099-b215-20dad86b5c3d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d1b889bd-bb457476c-9nz29"
Oct 30 21:15:44.376035 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:44.374528    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqf9l\" (UniqueName: \"kubernetes.io/projected/aa5ce369-e5ae-4099-b215-20dad86b5c3d-kube-api-access-zqf9l\") pod \"payload-pre-processing-e2e-mt-d1b889bd-bb457476c-9nz29\" (UID: \"aa5ce369-e5ae-4099-b215-20dad86b5c3d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d1b889bd-bb457476c-9nz29"
Oct 30 21:15:44.432355 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:44.432327    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqf9l\" (UniqueName: \"kubernetes.io/projected/aa5ce369-e5ae-4099-b215-20dad86b5c3d-kube-api-access-zqf9l\") pod \"payload-pre-processing-e2e-mt-d1b889bd-bb457476c-9nz29\" (UID: \"aa5ce369-e5ae-4099-b215-20dad86b5c3d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d1b889bd-bb457476c-9nz29"
Oct 30 21:15:44.476248 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:44.476210    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a5cc9df-af25-4c17-8105-5bc748364a4d-plugins-config-volume\") pod \"payload-processing-e2e-mt-d1b889bd-7d887b46cb-kvshd\" (UID: \"2a5cc9df-af25-4c17-8105-5bc748364a4d\") " pod="openshift-ingress/payload-processing-e2e-mt-d1b889bd-7d887b46cb-kvshd"
Oct 30 21:15:44.476436 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:44.476382    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t664\" (UniqueName: \"kubernetes.io/projected/2a5cc9df-af25-4c17-8105-5bc748364a4d-kube-api-access-6t664\") pod \"payload-processing-e2e-mt-d1b889bd-7d887b46cb-kvshd\" (UID: \"2a5cc9df-af25-4c17-8105-5bc748364a4d\") " pod="openshift-ingress/payload-processing-e2e-mt-d1b889bd-7d887b46cb-kvshd"
Oct 30 21:15:44.525947 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:44.525914    5713 patch_prober.go:29] interesting pod/e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body=
Oct 30 21:15:44.526149 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:44.525964    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5" podUID="fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused"
Oct 30 21:15:44.531184 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:44.531161    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d1b889bd-bb457476c-9nz29"
Oct 30 21:15:44.578031 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:44.577997    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a5cc9df-af25-4c17-8105-5bc748364a4d-plugins-config-volume\") pod \"payload-processing-e2e-mt-d1b889bd-7d887b46cb-kvshd\" (UID: \"2a5cc9df-af25-4c17-8105-5bc748364a4d\") " pod="openshift-ingress/payload-processing-e2e-mt-d1b889bd-7d887b46cb-kvshd"
Oct 30 21:15:44.578217 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:44.578179    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t664\" (UniqueName: \"kubernetes.io/projected/2a5cc9df-af25-4c17-8105-5bc748364a4d-kube-api-access-6t664\") pod \"payload-processing-e2e-mt-d1b889bd-7d887b46cb-kvshd\" (UID: \"2a5cc9df-af25-4c17-8105-5bc748364a4d\") " pod="openshift-ingress/payload-processing-e2e-mt-d1b889bd-7d887b46cb-kvshd"
Oct 30 21:15:44.579078 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:44.579050    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a5cc9df-af25-4c17-8105-5bc748364a4d-plugins-config-volume\") pod \"payload-processing-e2e-mt-d1b889bd-7d887b46cb-kvshd\" (UID: \"2a5cc9df-af25-4c17-8105-5bc748364a4d\") " pod="openshift-ingress/payload-processing-e2e-mt-d1b889bd-7d887b46cb-kvshd"
Oct 30 21:15:44.668410 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:44.668371    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t664\" (UniqueName: \"kubernetes.io/projected/2a5cc9df-af25-4c17-8105-5bc748364a4d-kube-api-access-6t664\") pod \"payload-processing-e2e-mt-d1b889bd-7d887b46cb-kvshd\" (UID: \"2a5cc9df-af25-4c17-8105-5bc748364a4d\") " pod="openshift-ingress/payload-processing-e2e-mt-d1b889bd-7d887b46cb-kvshd"
Oct 30 21:15:44.679615 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:44.679580    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d1b889bd-7d887b46cb-kvshd"
Oct 30 21:15:45.000656 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:45.000626    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d1b889bd-bb457476c-9nz29"]
Oct 30 21:15:45.001232 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:15:45.001201    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa5ce369_e5ae_4099_b215_20dad86b5c3d.slice/crio-47119c473c33e2a9eebee5db49c5dc47abb9652d5bd6e503feb794891c989626 WatchSource:0}: Error finding container 47119c473c33e2a9eebee5db49c5dc47abb9652d5bd6e503feb794891c989626: Status 404 returned error can't find the container with id 47119c473c33e2a9eebee5db49c5dc47abb9652d5bd6e503feb794891c989626
Oct 30 21:15:45.113306 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:45.113275    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d1b889bd-7d887b46cb-kvshd"]
Oct 30 21:15:45.114083 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:15:45.114059    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a5cc9df_af25_4c17_8105_5bc748364a4d.slice/crio-cc5ce48f44ac8f18b8c6bf489e67088b1968b3ae48a261748e0bfc82813b24d4 WatchSource:0}: Error finding container cc5ce48f44ac8f18b8c6bf489e67088b1968b3ae48a261748e0bfc82813b24d4: Status 404 returned error can't find the container with id cc5ce48f44ac8f18b8c6bf489e67088b1968b3ae48a261748e0bfc82813b24d4
Oct 30 21:15:45.525600 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:45.525158    5713 patch_prober.go:29] interesting pod/e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body=
Oct 30 21:15:45.525600 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:45.525216    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5" podUID="fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused"
Oct 30 21:15:45.659881 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:45.658060    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk"]
Oct 30 21:15:45.659881 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:45.658302    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk" podUID="ad23561f-a24f-4c13-9df7-0ecce2596034" containerName="istio-proxy" containerID="cri-o://f43f383d2ee675de7ecee69c246a89878d85c124ccb234cbee9817a50f1fba8d" gracePeriod=30
Oct 30 21:15:45.988398 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:45.988360    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d1b889bd-7d887b46cb-kvshd" event={"ID":"2a5cc9df-af25-4c17-8105-5bc748364a4d","Type":"ContainerStarted","Data":"d8c62126591999403edc64fb78fae1670c920e90b5f5749528a878f2602dc49b"}
Oct 30 21:15:45.990222 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:45.989951    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-d1b889bd-7d887b46cb-kvshd"
Oct 30 21:15:45.990435 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:45.990407    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d1b889bd-7d887b46cb-kvshd" event={"ID":"2a5cc9df-af25-4c17-8105-5bc748364a4d","Type":"ContainerStarted","Data":"cc5ce48f44ac8f18b8c6bf489e67088b1968b3ae48a261748e0bfc82813b24d4"}
Oct 30 21:15:45.990584 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:45.990567    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d1b889bd-bb457476c-9nz29" event={"ID":"aa5ce369-e5ae-4099-b215-20dad86b5c3d","Type":"ContainerStarted","Data":"5da55b990290a85a970e4b2644c5773ad7d24b7a2855c66c6e3792453ccf2071"}
Oct 30 21:15:45.990697 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:45.990685    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d1b889bd-bb457476c-9nz29" event={"ID":"aa5ce369-e5ae-4099-b215-20dad86b5c3d","Type":"ContainerStarted","Data":"47119c473c33e2a9eebee5db49c5dc47abb9652d5bd6e503feb794891c989626"}
Oct 30 21:15:45.990807 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:45.990780    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d1b889bd-bb457476c-9nz29"
Oct 30 21:15:46.307855 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:46.307702    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-d1b889bd-bb457476c-9nz29" podStartSLOduration=2.307687469 podStartE2EDuration="2.307687469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:46.306141306 +0000 UTC m=+2229.370917700" watchObservedRunningTime="2026-10-30 21:15:46.307687469 +0000 UTC m=+2229.372463868"
Oct 30 21:15:46.308842 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:46.308772    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-d1b889bd-7d887b46cb-kvshd" podStartSLOduration=2.308756163 podStartE2EDuration="2.308756163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:46.139995882 +0000 UTC m=+2229.204772281" watchObservedRunningTime="2026-10-30 21:15:46.308756163 +0000 UTC m=+2229.373532564"
Oct 30 21:15:46.528809 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:46.528223    5713 patch_prober.go:29] interesting pod/e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body=
Oct 30 21:15:46.528809 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:46.528305    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5" podUID="fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused"
Oct 30 21:15:47.871505 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:47.871460    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5"
Oct 30 21:15:47.872050 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:47.871923    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5"
Oct 30 21:15:47.872725 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:47.872708    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5"
Oct 30 21:15:51.075355 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:51.075320    5713 generic.go:356] "Generic (PLEG): container finished" podID="ad23561f-a24f-4c13-9df7-0ecce2596034" containerID="f43f383d2ee675de7ecee69c246a89878d85c124ccb234cbee9817a50f1fba8d" exitCode=0
Oct 30 21:15:51.075938 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:51.075392    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk" event={"ID":"ad23561f-a24f-4c13-9df7-0ecce2596034","Type":"ContainerDied","Data":"f43f383d2ee675de7ecee69c246a89878d85c124ccb234cbee9817a50f1fba8d"}
Oct 30 21:15:51.075938 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:51.075466    5713 scope.go:122] "RemoveContainer" containerID="4157e3bb57ce1a7d4978d8fa5903723c81ad1f43ca0a805cd809861f4e54edcc"
Oct 30 21:15:51.228871 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:51.228844    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk"
Oct 30 21:15:51.404303 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:51.404269    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad23561f-a24f-4c13-9df7-0ecce2596034-istio-token\" (UniqueName: \"kubernetes.io/projected/ad23561f-a24f-4c13-9df7-0ecce2596034-istio-token\") pod \"ad23561f-a24f-4c13-9df7-0ecce2596034\" (UID: \"ad23561f-a24f-4c13-9df7-0ecce2596034\") "
Oct 30 21:15:51.404505 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:51.404325    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad23561f-a24f-4c13-9df7-0ecce2596034-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad23561f-a24f-4c13-9df7-0ecce2596034-credential-socket\") pod \"ad23561f-a24f-4c13-9df7-0ecce2596034\" (UID: \"ad23561f-a24f-4c13-9df7-0ecce2596034\") "
Oct 30 21:15:51.404505 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:51.404438    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad23561f-a24f-4c13-9df7-0ecce2596034-kube-api-access-vbdw9\" (UniqueName: \"kubernetes.io/projected/ad23561f-a24f-4c13-9df7-0ecce2596034-kube-api-access-vbdw9\") pod \"ad23561f-a24f-4c13-9df7-0ecce2596034\" (UID: \"ad23561f-a24f-4c13-9df7-0ecce2596034\") "
Oct 30 21:15:51.404505 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:51.404465    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad23561f-a24f-4c13-9df7-0ecce2596034-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad23561f-a24f-4c13-9df7-0ecce2596034-istio-envoy\") pod \"ad23561f-a24f-4c13-9df7-0ecce2596034\" (UID: \"ad23561f-a24f-4c13-9df7-0ecce2596034\") "
Oct 30 21:15:51.404685 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:51.404545    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad23561f-a24f-4c13-9df7-0ecce2596034-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad23561f-a24f-4c13-9df7-0ecce2596034-istiod-ca-cert\") pod \"ad23561f-a24f-4c13-9df7-0ecce2596034\" (UID: \"ad23561f-a24f-4c13-9df7-0ecce2596034\") "
Oct 30 21:15:51.404685 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:51.404615    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad23561f-a24f-4c13-9df7-0ecce2596034-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad23561f-a24f-4c13-9df7-0ecce2596034-istio-data\") pod \"ad23561f-a24f-4c13-9df7-0ecce2596034\" (UID: \"ad23561f-a24f-4c13-9df7-0ecce2596034\") "
Oct 30 21:15:51.404685 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:51.404651    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad23561f-a24f-4c13-9df7-0ecce2596034-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad23561f-a24f-4c13-9df7-0ecce2596034-workload-certs\") pod \"ad23561f-a24f-4c13-9df7-0ecce2596034\" (UID: \"ad23561f-a24f-4c13-9df7-0ecce2596034\") "
Oct 30 21:15:51.404886 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:51.404672    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad23561f-a24f-4c13-9df7-0ecce2596034-credential-socket" pod "ad23561f-a24f-4c13-9df7-0ecce2596034" (UID: "ad23561f-a24f-4c13-9df7-0ecce2596034"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:15:51.404886 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:51.404778    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ad23561f-a24f-4c13-9df7-0ecce2596034-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad23561f-a24f-4c13-9df7-0ecce2596034-istio-podinfo\") pod \"ad23561f-a24f-4c13-9df7-0ecce2596034\" (UID: \"ad23561f-a24f-4c13-9df7-0ecce2596034\") "
Oct 30 21:15:51.405007 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:51.404891    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad23561f-a24f-4c13-9df7-0ecce2596034-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad23561f-a24f-4c13-9df7-0ecce2596034-workload-socket\") pod \"ad23561f-a24f-4c13-9df7-0ecce2596034\" (UID: \"ad23561f-a24f-4c13-9df7-0ecce2596034\") "
Oct 30 21:15:51.405007 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:51.404923    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad23561f-a24f-4c13-9df7-0ecce2596034-workload-certs" pod "ad23561f-a24f-4c13-9df7-0ecce2596034" (UID: "ad23561f-a24f-4c13-9df7-0ecce2596034"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:15:51.405125 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:51.405008    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad23561f-a24f-4c13-9df7-0ecce2596034-istio-data" pod "ad23561f-a24f-4c13-9df7-0ecce2596034" (UID: "ad23561f-a24f-4c13-9df7-0ecce2596034"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:15:51.405234 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:51.405204    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad23561f-a24f-4c13-9df7-0ecce2596034-workload-socket" pod "ad23561f-a24f-4c13-9df7-0ecce2596034" (UID: "ad23561f-a24f-4c13-9df7-0ecce2596034"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:15:51.405694 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:51.405603    5713 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad23561f-a24f-4c13-9df7-0ecce2596034-credential-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:51.405694 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:51.405629    5713 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad23561f-a24f-4c13-9df7-0ecce2596034-istio-data\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:51.405694 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:51.405643    5713 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad23561f-a24f-4c13-9df7-0ecce2596034-workload-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:51.405694 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:51.405657    5713 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad23561f-a24f-4c13-9df7-0ecce2596034-workload-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:51.405990 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:51.405966    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad23561f-a24f-4c13-9df7-0ecce2596034-istiod-ca-cert" pod "ad23561f-a24f-4c13-9df7-0ecce2596034" (UID: "ad23561f-a24f-4c13-9df7-0ecce2596034"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:51.407422 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:51.407396    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad23561f-a24f-4c13-9df7-0ecce2596034-istio-token" pod "ad23561f-a24f-4c13-9df7-0ecce2596034" (UID: "ad23561f-a24f-4c13-9df7-0ecce2596034"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:51.407422 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:51.407412    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad23561f-a24f-4c13-9df7-0ecce2596034-istio-envoy" pod "ad23561f-a24f-4c13-9df7-0ecce2596034" (UID: "ad23561f-a24f-4c13-9df7-0ecce2596034"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:15:51.407682 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:51.407664    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad23561f-a24f-4c13-9df7-0ecce2596034-kube-api-access-vbdw9" pod "ad23561f-a24f-4c13-9df7-0ecce2596034" (UID: "ad23561f-a24f-4c13-9df7-0ecce2596034"). InnerVolumeSpecName "kube-api-access-vbdw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:51.407963 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:51.407944    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad23561f-a24f-4c13-9df7-0ecce2596034-istio-podinfo" pod "ad23561f-a24f-4c13-9df7-0ecce2596034" (UID: "ad23561f-a24f-4c13-9df7-0ecce2596034"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:15:51.506826 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:51.506699    5713 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad23561f-a24f-4c13-9df7-0ecce2596034-istio-podinfo\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:51.506826 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:51.506733    5713 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad23561f-a24f-4c13-9df7-0ecce2596034-istio-token\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:51.506826 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:51.506743    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbdw9\" (UniqueName: \"kubernetes.io/projected/ad23561f-a24f-4c13-9df7-0ecce2596034-kube-api-access-vbdw9\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:51.506826 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:51.506754    5713 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad23561f-a24f-4c13-9df7-0ecce2596034-istio-envoy\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:51.506826 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:51.506762    5713 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad23561f-a24f-4c13-9df7-0ecce2596034-istiod-ca-cert\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:15:52.084639 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:52.084607    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk"
Oct 30 21:15:52.085157 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:52.084638    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk" event={"ID":"ad23561f-a24f-4c13-9df7-0ecce2596034","Type":"ContainerDied","Data":"080cf27cf9517f4edf56f7c309334ca693f071836c64293a8161f556b1753c00"}
Oct 30 21:15:52.085157 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:52.084699    5713 scope.go:122] "RemoveContainer" containerID="f43f383d2ee675de7ecee69c246a89878d85c124ccb234cbee9817a50f1fba8d"
Oct 30 21:15:52.268970 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:52.268933    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk"]
Oct 30 21:15:52.319084 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:52.319044    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-677b45-openshift-default-59786bb9df-q87bk"]
Oct 30 21:15:53.069881 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:53.069836    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad23561f-a24f-4c13-9df7-0ecce2596034" path="/var/lib/kubelet/pods/ad23561f-a24f-4c13-9df7-0ecce2596034/volumes"
Oct 30 21:15:54.462349 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:15:54.462306    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:15:54.650981 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:54.650945    5713 patch_prober.go:29] interesting pod/e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:15:54.651186 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:54.651021    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg" podUID="d0dff592-d5a1-4832-8104-46cc5fd973c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:55.067667 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:55.067633    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d061d0-sq9mk"]
Oct 30 21:15:55.117198 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:55.117162    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d061d0-sq9mk"]
Oct 30 21:15:57.064199 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:57.064127    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecf576f4-8582-4ddc-8768-8c04888e5e4b" path="/var/lib/kubelet/pods/ecf576f4-8582-4ddc-8768-8c04888e5e4b/volumes"
Oct 30 21:15:57.159176 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:57.159070    5713 generic.go:356] "Generic (PLEG): container finished" podID="d0dff592-d5a1-4832-8104-46cc5fd973c5" containerID="607a33906cd32389d6ea58d710f74b7d5db210e30b2d21241e6e873832713ff8" exitCode=0
Oct 30 21:15:57.159176 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:57.159119    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg" event={"ID":"d0dff592-d5a1-4832-8104-46cc5fd973c5","Type":"ContainerDied","Data":"607a33906cd32389d6ea58d710f74b7d5db210e30b2d21241e6e873832713ff8"}
Oct 30 21:15:57.159682 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:57.159661    5713 scope.go:122] "RemoveContainer" containerID="607a33906cd32389d6ea58d710f74b7d5db210e30b2d21241e6e873832713ff8"
Oct 30 21:15:57.161466 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:57.161436    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:15:57.161571 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:57.161507    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:15:57.161571 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:57.161539    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:15:58.019006 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:58.018963    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d1b889bd-bb457476c-9nz29"
Oct 30 21:15:58.020186 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:58.020165    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-d1b889bd-7d887b46cb-kvshd"
Oct 30 21:15:58.177045 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:58.177008    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg" event={"ID":"d0dff592-d5a1-4832-8104-46cc5fd973c5","Type":"ContainerStarted","Data":"eb0fe2865c319539aa0fcaba4ec26a0846a4830c675c707d4ee10ebde1983082"}
Oct 30 21:15:58.359121 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:58.359019    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg"
Oct 30 21:15:58.359333 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:58.359304    5713 patch_prober.go:29] interesting pod/e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Oct 30 21:15:58.359416 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:58.359363    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg" podUID="d0dff592-d5a1-4832-8104-46cc5fd973c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Oct 30 21:15:59.361290 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:59.361248    5713 patch_prober.go:29] interesting pod/e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Oct 30 21:15:59.361978 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:15:59.361945    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg" podUID="d0dff592-d5a1-4832-8104-46cc5fd973c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Oct 30 21:16:00.359290 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:00.359241    5713 patch_prober.go:29] interesting pod/e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Oct 30 21:16:00.359505 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:00.359316    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg" podUID="d0dff592-d5a1-4832-8104-46cc5fd973c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Oct 30 21:16:01.359208 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:01.359170    5713 patch_prober.go:29] interesting pod/e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Oct 30 21:16:01.359643 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:01.359231    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg" podUID="d0dff592-d5a1-4832-8104-46cc5fd973c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Oct 30 21:16:02.359266 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:02.359224    5713 patch_prober.go:29] interesting pod/e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Oct 30 21:16:02.359854 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:02.359303    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg" podUID="d0dff592-d5a1-4832-8104-46cc5fd973c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Oct 30 21:16:02.490267 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:02.490214    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm"]
Oct 30 21:16:02.490478 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:02.490456    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm" podUID="e4e643cf-8167-42e3-a104-03ac5f7bd133" containerName="istio-proxy" containerID="cri-o://06cf657bc4c28717d86d3c09fd43b27935bceb0b8e9c9d563fa4a76e3ab76cd1" gracePeriod=30
Oct 30 21:16:02.509780 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:02.509746    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5"]
Oct 30 21:16:02.510044 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:02.510017    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5" podUID="fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a" containerName="istio-proxy" containerID="cri-o://8708f005a53b67ee2c563425cef2b4d8b9b8db85761dbfde3607262558f686c5" gracePeriod=30
Oct 30 21:16:02.874478 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:02.874440    5713 patch_prober.go:29] interesting pod/e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:16:02.874655 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:02.874520    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5" podUID="fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:16:03.359257 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:03.359216    5713 patch_prober.go:29] interesting pod/e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Oct 30 21:16:03.359451 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:03.359287    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg" podUID="d0dff592-d5a1-4832-8104-46cc5fd973c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Oct 30 21:16:04.362931 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:04.362898    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg"
Oct 30 21:16:04.363354 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:04.363215    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg"
Oct 30 21:16:04.363875 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:04.363859    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg"
Oct 30 21:16:06.203497 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:06.203466    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x"]
Oct 30 21:16:06.204168 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:06.204151    5713 cpu_manager.go:414] "containerMap: removing container" podUID="ad23561f-a24f-4c13-9df7-0ecce2596034" containerName="istio-proxy"
Oct 30 21:16:06.204225 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:06.204170    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad23561f-a24f-4c13-9df7-0ecce2596034" containerName="istio-proxy"
Oct 30 21:16:06.204225 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:06.204183    5713 cpu_manager.go:414] "containerMap: removing container" podUID="ad23561f-a24f-4c13-9df7-0ecce2596034" containerName="istio-proxy"
Oct 30 21:16:06.204225 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:06.204188    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad23561f-a24f-4c13-9df7-0ecce2596034" containerName="istio-proxy"
Oct 30 21:16:06.204326 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:06.204278    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad23561f-a24f-4c13-9df7-0ecce2596034" containerName="istio-proxy"
Oct 30 21:16:06.204326 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:06.204286    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad23561f-a24f-4c13-9df7-0ecce2596034" containerName="istio-proxy"
Oct 30 21:16:06.207911 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:06.207893    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x"
Oct 30 21:16:06.280464 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:06.280428    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x"]
Oct 30 21:16:06.293114 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:06.293083    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5n78\" (UniqueName: \"kubernetes.io/projected/fbb39bfd-2716-4ded-9e69-6131383afdb9-kube-api-access-n5n78\") pod \"e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x\" (UID: \"fbb39bfd-2716-4ded-9e69-6131383afdb9\") " pod="openshift-ingress/e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x"
Oct 30 21:16:06.293289 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:06.293167    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbb39bfd-2716-4ded-9e69-6131383afdb9-istio-token\") pod \"e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x\" (UID: \"fbb39bfd-2716-4ded-9e69-6131383afdb9\") " pod="openshift-ingress/e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x"
Oct 30 21:16:06.293289 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:06.293233    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb39bfd-2716-4ded-9e69-6131383afdb9-workload-socket\") pod \"e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x\" (UID: \"fbb39bfd-2716-4ded-9e69-6131383afdb9\") " pod="openshift-ingress/e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x"
Oct 30 21:16:06.293449 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:06.293419    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbb39bfd-2716-4ded-9e69-6131383afdb9-istiod-ca-cert\") pod \"e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x\" (UID: \"fbb39bfd-2716-4ded-9e69-6131383afdb9\") " pod="openshift-ingress/e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x"
Oct 30 21:16:06.293509 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:06.293473    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbb39bfd-2716-4ded-9e69-6131383afdb9-istio-data\") pod \"e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x\" (UID: \"fbb39bfd-2716-4ded-9e69-6131383afdb9\") " pod="openshift-ingress/e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x"
Oct 30 21:16:06.293568 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:06.293550    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbb39bfd-2716-4ded-9e69-6131383afdb9-istio-envoy\") pod \"e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x\" (UID: \"fbb39bfd-2716-4ded-9e69-6131383afdb9\") " pod="openshift-ingress/e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x"
Oct 30 21:16:06.293628 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:06.293602    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbb39bfd-2716-4ded-9e69-6131383afdb9-workload-certs\") pod \"e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x\" (UID: \"fbb39bfd-2716-4ded-9e69-6131383afdb9\") " pod="openshift-ingress/e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x"
Oct 30 21:16:06.293729 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:06.293704    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb39bfd-2716-4ded-9e69-6131383afdb9-credential-socket\") pod \"e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x\" (UID: \"fbb39bfd-2716-4ded-9e69-6131383afdb9\") " pod="openshift-ingress/e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x"
Oct 30 21:16:06.293774 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:06.293760    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbb39bfd-2716-4ded-9e69-6131383afdb9-istio-podinfo\") pod \"e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x\" (UID: \"fbb39bfd-2716-4ded-9e69-6131383afdb9\") " pod="openshift-ingress/e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x"
Oct 30 21:16:06.313501 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:06.313470    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-08df98-openshift-default-dockercfg-4mpbn\""
Oct 30 21:16:06.395101 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:06.395056    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbb39bfd-2716-4ded-9e69-6131383afdb9-istiod-ca-cert\") pod \"e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x\" (UID: \"fbb39bfd-2716-4ded-9e69-6131383afdb9\") " pod="openshift-ingress/e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x"
Oct 30 21:16:06.395101 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:06.395099    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbb39bfd-2716-4ded-9e69-6131383afdb9-istio-data\") pod \"e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x\" (UID: \"fbb39bfd-2716-4ded-9e69-6131383afdb9\") " pod="openshift-ingress/e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x"
Oct 30 21:16:06.395382 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:06.395121    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbb39bfd-2716-4ded-9e69-6131383afdb9-istio-envoy\") pod \"e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x\" (UID: \"fbb39bfd-2716-4ded-9e69-6131383afdb9\") " pod="openshift-ingress/e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x"
Oct 30 21:16:06.395382 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:06.395155    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbb39bfd-2716-4ded-9e69-6131383afdb9-workload-certs\") pod \"e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x\" (UID: \"fbb39bfd-2716-4ded-9e69-6131383afdb9\") " pod="openshift-ingress/e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x"
Oct 30 21:16:06.395382 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:06.395180    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb39bfd-2716-4ded-9e69-6131383afdb9-credential-socket\") pod \"e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x\" (UID: \"fbb39bfd-2716-4ded-9e69-6131383afdb9\") " pod="openshift-ingress/e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x"
Oct 30 21:16:06.395382 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:06.395216    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbb39bfd-2716-4ded-9e69-6131383afdb9-istio-podinfo\") pod \"e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x\" (UID: \"fbb39bfd-2716-4ded-9e69-6131383afdb9\") " pod="openshift-ingress/e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x"
Oct 30 21:16:06.395382 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:06.395332    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5n78\" (UniqueName: \"kubernetes.io/projected/fbb39bfd-2716-4ded-9e69-6131383afdb9-kube-api-access-n5n78\") pod \"e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x\" (UID: \"fbb39bfd-2716-4ded-9e69-6131383afdb9\") " pod="openshift-ingress/e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x"
Oct 30 21:16:06.395678 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:06.395447    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbb39bfd-2716-4ded-9e69-6131383afdb9-istio-token\") pod \"e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x\" (UID: \"fbb39bfd-2716-4ded-9e69-6131383afdb9\") " pod="openshift-ingress/e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x"
Oct 30 21:16:06.395678 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:06.395550    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb39bfd-2716-4ded-9e69-6131383afdb9-workload-socket\") pod \"e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x\" (UID: \"fbb39bfd-2716-4ded-9e69-6131383afdb9\") " pod="openshift-ingress/e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x"
Oct 30 21:16:06.395678 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:06.395579    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbb39bfd-2716-4ded-9e69-6131383afdb9-workload-certs\") pod \"e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x\" (UID: \"fbb39bfd-2716-4ded-9e69-6131383afdb9\") " pod="openshift-ingress/e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x"
Oct 30 21:16:06.395678 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:06.395634    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbb39bfd-2716-4ded-9e69-6131383afdb9-istio-data\") pod \"e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x\" (UID: \"fbb39bfd-2716-4ded-9e69-6131383afdb9\") " pod="openshift-ingress/e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x"
Oct 30 21:16:06.395678 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:06.395677    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb39bfd-2716-4ded-9e69-6131383afdb9-credential-socket\") pod \"e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x\" (UID: \"fbb39bfd-2716-4ded-9e69-6131383afdb9\") " pod="openshift-ingress/e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x"
Oct 30 21:16:06.396030 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:06.395921    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbb39bfd-2716-4ded-9e69-6131383afdb9-istiod-ca-cert\") pod \"e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x\" (UID: \"fbb39bfd-2716-4ded-9e69-6131383afdb9\") " pod="openshift-ingress/e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x"
Oct 30 21:16:06.396030 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:06.395953    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb39bfd-2716-4ded-9e69-6131383afdb9-workload-socket\") pod \"e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x\" (UID: \"fbb39bfd-2716-4ded-9e69-6131383afdb9\") " pod="openshift-ingress/e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x"
Oct 30 21:16:06.397819 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:06.397779    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbb39bfd-2716-4ded-9e69-6131383afdb9-istio-envoy\") pod \"e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x\" (UID: \"fbb39bfd-2716-4ded-9e69-6131383afdb9\") " pod="openshift-ingress/e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x"
Oct 30 21:16:06.398259 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:06.398242    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbb39bfd-2716-4ded-9e69-6131383afdb9-istio-podinfo\") pod \"e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x\" (UID: \"fbb39bfd-2716-4ded-9e69-6131383afdb9\") " pod="openshift-ingress/e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x"
Oct 30 21:16:06.459193 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:06.459119    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbb39bfd-2716-4ded-9e69-6131383afdb9-istio-token\") pod \"e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x\" (UID: \"fbb39bfd-2716-4ded-9e69-6131383afdb9\") " pod="openshift-ingress/e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x"
Oct 30 21:16:06.459193 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:06.459127    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5n78\" (UniqueName: \"kubernetes.io/projected/fbb39bfd-2716-4ded-9e69-6131383afdb9-kube-api-access-n5n78\") pod \"e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x\" (UID: \"fbb39bfd-2716-4ded-9e69-6131383afdb9\") " pod="openshift-ingress/e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x"
Oct 30 21:16:06.522664 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:06.522621    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x"
Oct 30 21:16:06.957965 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:06.957893    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x"]
Oct 30 21:16:06.958402 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:16:06.958380    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbb39bfd_2716_4ded_9e69_6131383afdb9.slice/crio-725f37863f4271aceddeaa7bad0a2cb14c194845c0b17376144cea81d200874f WatchSource:0}: Error finding container 725f37863f4271aceddeaa7bad0a2cb14c194845c0b17376144cea81d200874f: Status 404 returned error can't find the container with id 725f37863f4271aceddeaa7bad0a2cb14c194845c0b17376144cea81d200874f
Oct 30 21:16:06.961162 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:06.961133    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:16:06.961246 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:06.961194    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:16:06.961246 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:06.961220    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:16:07.313234 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:07.313144    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x" event={"ID":"fbb39bfd-2716-4ded-9e69-6131383afdb9","Type":"ContainerStarted","Data":"ae817ded885d8d06555408698bfad5fe2bb80a8ef99e711f5330eda0f7763afe"}
Oct 30 21:16:07.313234 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:07.313208    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x" event={"ID":"fbb39bfd-2716-4ded-9e69-6131383afdb9","Type":"ContainerStarted","Data":"725f37863f4271aceddeaa7bad0a2cb14c194845c0b17376144cea81d200874f"}
Oct 30 21:16:07.436337 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:07.436286    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x" podStartSLOduration=1.436272918 podStartE2EDuration="1.436272918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:07.435526708 +0000 UTC m=+2250.500303143" watchObservedRunningTime="2026-10-30 21:16:07.436272918 +0000 UTC m=+2250.501049369"
Oct 30 21:16:07.522971 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:07.522943    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x"
Oct 30 21:16:07.786375 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:07.786349    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm"
Oct 30 21:16:07.815603 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:07.815578    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5"
Oct 30 21:16:07.917495 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:07.917405    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a-istio-data\") pod \"fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a\" (UID: \"fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a\") "
Oct 30 21:16:07.917495 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:07.917468    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4e643cf-8167-42e3-a104-03ac5f7bd133-kube-api-access-8vsq9\" (UniqueName: \"kubernetes.io/projected/e4e643cf-8167-42e3-a104-03ac5f7bd133-kube-api-access-8vsq9\") pod \"e4e643cf-8167-42e3-a104-03ac5f7bd133\" (UID: \"e4e643cf-8167-42e3-a104-03ac5f7bd133\") "
Oct 30 21:16:07.917495 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:07.917497    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4e643cf-8167-42e3-a104-03ac5f7bd133-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e643cf-8167-42e3-a104-03ac5f7bd133-workload-socket\") pod \"e4e643cf-8167-42e3-a104-03ac5f7bd133\" (UID: \"e4e643cf-8167-42e3-a104-03ac5f7bd133\") "
Oct 30 21:16:07.917845 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:07.917526    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a-istio-envoy\") pod \"fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a\" (UID: \"fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a\") "
Oct 30 21:16:07.917845 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:07.917566    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e4e643cf-8167-42e3-a104-03ac5f7bd133-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4e643cf-8167-42e3-a104-03ac5f7bd133-istio-podinfo\") pod \"e4e643cf-8167-42e3-a104-03ac5f7bd133\" (UID: \"e4e643cf-8167-42e3-a104-03ac5f7bd133\") "
Oct 30 21:16:07.917845 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:07.917629    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4e643cf-8167-42e3-a104-03ac5f7bd133-istio-token\" (UniqueName: \"kubernetes.io/projected/e4e643cf-8167-42e3-a104-03ac5f7bd133-istio-token\") pod \"e4e643cf-8167-42e3-a104-03ac5f7bd133\" (UID: \"e4e643cf-8167-42e3-a104-03ac5f7bd133\") "
Oct 30 21:16:07.917845 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:07.917667    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a-kube-api-access-wv6wc\" (UniqueName: \"kubernetes.io/projected/fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a-kube-api-access-wv6wc\") pod \"fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a\" (UID: \"fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a\") "
Oct 30 21:16:07.917845 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:07.917748    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4e643cf-8167-42e3-a104-03ac5f7bd133-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4e643cf-8167-42e3-a104-03ac5f7bd133-istiod-ca-cert\") pod \"e4e643cf-8167-42e3-a104-03ac5f7bd133\" (UID: \"e4e643cf-8167-42e3-a104-03ac5f7bd133\") "
Oct 30 21:16:07.917845 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:07.917822    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a-istiod-ca-cert\") pod \"fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a\" (UID: \"fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a\") "
Oct 30 21:16:07.918191 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:07.917875    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4e643cf-8167-42e3-a104-03ac5f7bd133-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e643cf-8167-42e3-a104-03ac5f7bd133-credential-socket\") pod \"e4e643cf-8167-42e3-a104-03ac5f7bd133\" (UID: \"e4e643cf-8167-42e3-a104-03ac5f7bd133\") "
Oct 30 21:16:07.918191 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:07.918009    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4e643cf-8167-42e3-a104-03ac5f7bd133-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4e643cf-8167-42e3-a104-03ac5f7bd133-workload-certs\") pod \"e4e643cf-8167-42e3-a104-03ac5f7bd133\" (UID: \"e4e643cf-8167-42e3-a104-03ac5f7bd133\") "
Oct 30 21:16:07.918191 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:07.918037    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a-credential-socket\") pod \"fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a\" (UID: \"fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a\") "
Oct 30 21:16:07.918191 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:07.918061    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a-workload-certs\") pod \"fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a\" (UID: \"fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a\") "
Oct 30 21:16:07.918191 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:07.918106    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a-workload-socket\") pod \"fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a\" (UID: \"fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a\") "
Oct 30 21:16:07.918191 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:07.918169    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4e643cf-8167-42e3-a104-03ac5f7bd133-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4e643cf-8167-42e3-a104-03ac5f7bd133-istio-data\") pod \"e4e643cf-8167-42e3-a104-03ac5f7bd133\" (UID: \"e4e643cf-8167-42e3-a104-03ac5f7bd133\") "
Oct 30 21:16:07.918518 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:07.918220    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a-istio-token\" (UniqueName: \"kubernetes.io/projected/fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a-istio-token\") pod \"fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a\" (UID: \"fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a\") "
Oct 30 21:16:07.918518 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:07.917662    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a-istio-data" pod "fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a" (UID: "fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:16:07.918518 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:07.917867    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4e643cf-8167-42e3-a104-03ac5f7bd133-workload-socket" pod "e4e643cf-8167-42e3-a104-03ac5f7bd133" (UID: "e4e643cf-8167-42e3-a104-03ac5f7bd133"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:16:07.918518 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:07.918213    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4e643cf-8167-42e3-a104-03ac5f7bd133-credential-socket" pod "e4e643cf-8167-42e3-a104-03ac5f7bd133" (UID: "e4e643cf-8167-42e3-a104-03ac5f7bd133"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:16:07.918518 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:07.918237    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4e643cf-8167-42e3-a104-03ac5f7bd133-istiod-ca-cert" pod "e4e643cf-8167-42e3-a104-03ac5f7bd133" (UID: "e4e643cf-8167-42e3-a104-03ac5f7bd133"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:16:07.918518 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:07.918291    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a-istio-podinfo\") pod \"fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a\" (UID: \"fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a\") "
Oct 30 21:16:07.918518 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:07.918376    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4e643cf-8167-42e3-a104-03ac5f7bd133-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4e643cf-8167-42e3-a104-03ac5f7bd133-istio-envoy\") pod \"e4e643cf-8167-42e3-a104-03ac5f7bd133\" (UID: \"e4e643cf-8167-42e3-a104-03ac5f7bd133\") "
Oct 30 21:16:07.918518 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:07.918468    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a-workload-certs" pod "fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a" (UID: "fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:16:07.919144 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:07.918644    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4e643cf-8167-42e3-a104-03ac5f7bd133-workload-certs" pod "e4e643cf-8167-42e3-a104-03ac5f7bd133" (UID: "e4e643cf-8167-42e3-a104-03ac5f7bd133"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:16:07.919144 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:07.918700    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a-istiod-ca-cert" pod "fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a" (UID: "fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:16:07.919144 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:07.918832    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a-credential-socket" pod "fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a" (UID: "fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:16:07.919144 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:07.918987    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a-workload-socket" pod "fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a" (UID: "fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:16:07.919380 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:07.919293    5713 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4e643cf-8167-42e3-a104-03ac5f7bd133-workload-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:16:07.919380 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:07.919314    5713 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a-credential-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:16:07.919380 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:07.919327    5713 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a-workload-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:16:07.919380 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:07.919340    5713 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a-workload-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:16:07.919380 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:07.919351    5713 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a-istio-data\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:16:07.919380 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:07.919363    5713 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e643cf-8167-42e3-a104-03ac5f7bd133-workload-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:16:07.919380 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:07.919375    5713 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4e643cf-8167-42e3-a104-03ac5f7bd133-istiod-ca-cert\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:16:07.919774 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:07.919387    5713 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a-istiod-ca-cert\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:16:07.919774 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:07.919400    5713 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e643cf-8167-42e3-a104-03ac5f7bd133-credential-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:16:07.919774 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:07.919602    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4e643cf-8167-42e3-a104-03ac5f7bd133-istio-data" pod "e4e643cf-8167-42e3-a104-03ac5f7bd133" (UID: "e4e643cf-8167-42e3-a104-03ac5f7bd133"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:16:07.923858 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:07.920997    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4e643cf-8167-42e3-a104-03ac5f7bd133-kube-api-access-8vsq9" pod "e4e643cf-8167-42e3-a104-03ac5f7bd133" (UID: "e4e643cf-8167-42e3-a104-03ac5f7bd133"). InnerVolumeSpecName "kube-api-access-8vsq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:16:07.923858 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:07.921230    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4e643cf-8167-42e3-a104-03ac5f7bd133-istio-token" pod "e4e643cf-8167-42e3-a104-03ac5f7bd133" (UID: "e4e643cf-8167-42e3-a104-03ac5f7bd133"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:16:07.923858 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:07.921452    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a-istio-podinfo" pod "fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a" (UID: "fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:16:07.923858 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:07.922198    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a-kube-api-access-wv6wc" pod "fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a" (UID: "fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a"). InnerVolumeSpecName "kube-api-access-wv6wc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:16:07.923858 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:07.922259    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a-istio-token" pod "fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a" (UID: "fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:16:07.923858 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:07.922560    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4e643cf-8167-42e3-a104-03ac5f7bd133-istio-envoy" pod "e4e643cf-8167-42e3-a104-03ac5f7bd133" (UID: "e4e643cf-8167-42e3-a104-03ac5f7bd133"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:16:07.923858 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:07.922654    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a-istio-envoy" pod "fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a" (UID: "fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:16:07.923858 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:07.923688    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e4e643cf-8167-42e3-a104-03ac5f7bd133-istio-podinfo" pod "e4e643cf-8167-42e3-a104-03ac5f7bd133" (UID: "e4e643cf-8167-42e3-a104-03ac5f7bd133"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:16:08.021043 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:08.021007    5713 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a-istio-podinfo\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:16:08.021043 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:08.021041    5713 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4e643cf-8167-42e3-a104-03ac5f7bd133-istio-envoy\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:16:08.021287 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:08.021056    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vsq9\" (UniqueName: \"kubernetes.io/projected/e4e643cf-8167-42e3-a104-03ac5f7bd133-kube-api-access-8vsq9\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:16:08.021287 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:08.021071    5713 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a-istio-envoy\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:16:08.021287 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:08.021085    5713 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4e643cf-8167-42e3-a104-03ac5f7bd133-istio-podinfo\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:16:08.021287 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:08.021102    5713 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4e643cf-8167-42e3-a104-03ac5f7bd133-istio-token\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:16:08.021287 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:08.021116    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wv6wc\" (UniqueName: \"kubernetes.io/projected/fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a-kube-api-access-wv6wc\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:16:08.021287 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:08.021131    5713 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4e643cf-8167-42e3-a104-03ac5f7bd133-istio-data\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:16:08.021287 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:08.021144    5713 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a-istio-token\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:16:08.321952 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:08.321911    5713 generic.go:356] "Generic (PLEG): container finished" podID="fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a" containerID="8708f005a53b67ee2c563425cef2b4d8b9b8db85761dbfde3607262558f686c5" exitCode=0
Oct 30 21:16:08.322455 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:08.321985    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5"
Oct 30 21:16:08.322455 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:08.321996    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5" event={"ID":"fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a","Type":"ContainerDied","Data":"8708f005a53b67ee2c563425cef2b4d8b9b8db85761dbfde3607262558f686c5"}
Oct 30 21:16:08.322455 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:08.322081    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5" event={"ID":"fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a","Type":"ContainerDied","Data":"6934c44b2b92d9a67ec17e544173e660fc1843488d81848b10bdc3dee5adbea3"}
Oct 30 21:16:08.322455 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:08.322103    5713 scope.go:122] "RemoveContainer" containerID="8708f005a53b67ee2c563425cef2b4d8b9b8db85761dbfde3607262558f686c5"
Oct 30 21:16:08.324267 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:08.324234    5713 generic.go:356] "Generic (PLEG): container finished" podID="e4e643cf-8167-42e3-a104-03ac5f7bd133" containerID="06cf657bc4c28717d86d3c09fd43b27935bceb0b8e9c9d563fa4a76e3ab76cd1" exitCode=0
Oct 30 21:16:08.324509 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:08.324488    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm"
Oct 30 21:16:08.324628 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:08.324511    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm" event={"ID":"e4e643cf-8167-42e3-a104-03ac5f7bd133","Type":"ContainerDied","Data":"06cf657bc4c28717d86d3c09fd43b27935bceb0b8e9c9d563fa4a76e3ab76cd1"}
Oct 30 21:16:08.324628 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:08.324581    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm" event={"ID":"e4e643cf-8167-42e3-a104-03ac5f7bd133","Type":"ContainerDied","Data":"9e3b401e1ac0a5bcc5e63972e1e222545a39ae2279c53af8fc2089e92f1b0a41"}
Oct 30 21:16:08.335610 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:08.335586    5713 scope.go:122] "RemoveContainer" containerID="8708f005a53b67ee2c563425cef2b4d8b9b8db85761dbfde3607262558f686c5"
Oct 30 21:16:08.335959 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:16:08.335935    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8708f005a53b67ee2c563425cef2b4d8b9b8db85761dbfde3607262558f686c5\": container with ID starting with 8708f005a53b67ee2c563425cef2b4d8b9b8db85761dbfde3607262558f686c5 not found: ID does not exist" containerID="8708f005a53b67ee2c563425cef2b4d8b9b8db85761dbfde3607262558f686c5"
Oct 30 21:16:08.336025 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:08.335970    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8708f005a53b67ee2c563425cef2b4d8b9b8db85761dbfde3607262558f686c5"} err="failed to get container status \"8708f005a53b67ee2c563425cef2b4d8b9b8db85761dbfde3607262558f686c5\": rpc error: code = NotFound desc = could not find container \"8708f005a53b67ee2c563425cef2b4d8b9b8db85761dbfde3607262558f686c5\": container with ID starting with 8708f005a53b67ee2c563425cef2b4d8b9b8db85761dbfde3607262558f686c5 not found: ID does not exist"
Oct 30 21:16:08.336025 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:08.335998    5713 scope.go:122] "RemoveContainer" containerID="06cf657bc4c28717d86d3c09fd43b27935bceb0b8e9c9d563fa4a76e3ab76cd1"
Oct 30 21:16:08.347661 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:08.347640    5713 scope.go:122] "RemoveContainer" containerID="e885c7f726a81161a04012f02368d308828df70aa2ed6e28ab7870842505ce8a"
Oct 30 21:16:08.357592 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:08.357574    5713 scope.go:122] "RemoveContainer" containerID="06cf657bc4c28717d86d3c09fd43b27935bceb0b8e9c9d563fa4a76e3ab76cd1"
Oct 30 21:16:08.357923 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:16:08.357894    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06cf657bc4c28717d86d3c09fd43b27935bceb0b8e9c9d563fa4a76e3ab76cd1\": container with ID starting with 06cf657bc4c28717d86d3c09fd43b27935bceb0b8e9c9d563fa4a76e3ab76cd1 not found: ID does not exist" containerID="06cf657bc4c28717d86d3c09fd43b27935bceb0b8e9c9d563fa4a76e3ab76cd1"
Oct 30 21:16:08.357986 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:08.357923    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06cf657bc4c28717d86d3c09fd43b27935bceb0b8e9c9d563fa4a76e3ab76cd1"} err="failed to get container status \"06cf657bc4c28717d86d3c09fd43b27935bceb0b8e9c9d563fa4a76e3ab76cd1\": rpc error: code = NotFound desc = could not find container \"06cf657bc4c28717d86d3c09fd43b27935bceb0b8e9c9d563fa4a76e3ab76cd1\": container with ID starting with 06cf657bc4c28717d86d3c09fd43b27935bceb0b8e9c9d563fa4a76e3ab76cd1 not found: ID does not exist"
Oct 30 21:16:08.357986 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:08.357941    5713 scope.go:122] "RemoveContainer" containerID="e885c7f726a81161a04012f02368d308828df70aa2ed6e28ab7870842505ce8a"
Oct 30 21:16:08.358220 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:16:08.358201    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e885c7f726a81161a04012f02368d308828df70aa2ed6e28ab7870842505ce8a\": container with ID starting with e885c7f726a81161a04012f02368d308828df70aa2ed6e28ab7870842505ce8a not found: ID does not exist" containerID="e885c7f726a81161a04012f02368d308828df70aa2ed6e28ab7870842505ce8a"
Oct 30 21:16:08.358260 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:08.358228    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e885c7f726a81161a04012f02368d308828df70aa2ed6e28ab7870842505ce8a"} err="failed to get container status \"e885c7f726a81161a04012f02368d308828df70aa2ed6e28ab7870842505ce8a\": rpc error: code = NotFound desc = could not find container \"e885c7f726a81161a04012f02368d308828df70aa2ed6e28ab7870842505ce8a\": container with ID starting with e885c7f726a81161a04012f02368d308828df70aa2ed6e28ab7870842505ce8a not found: ID does not exist"
Oct 30 21:16:08.490269 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:08.490234    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5"]
Oct 30 21:16:08.529800 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:08.529756    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x"
Oct 30 21:16:08.539814 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:08.539775    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-5h8d5"]
Oct 30 21:16:08.606818 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:08.606732    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm"]
Oct 30 21:16:08.611362 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:08.611339    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-d061d0-openshift-default-86b87f495-jd6rm"]
Oct 30 21:16:09.064991 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:09.064963    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4e643cf-8167-42e3-a104-03ac5f7bd133" path="/var/lib/kubelet/pods/e4e643cf-8167-42e3-a104-03ac5f7bd133/volumes"
Oct 30 21:16:09.065454 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:09.065440    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a" path="/var/lib/kubelet/pods/fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a/volumes"
Oct 30 21:16:09.334906 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:09.334832    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x"
Oct 30 21:16:09.335951 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:09.335931    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x"
Oct 30 21:16:14.008680 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:14.008645    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-08df98-699dbdd999-t8wvt"]
Oct 30 21:16:14.009491 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:14.009465    5713 cpu_manager.go:414] "containerMap: removing container" podUID="e4e643cf-8167-42e3-a104-03ac5f7bd133" containerName="istio-proxy"
Oct 30 21:16:14.009607 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:14.009493    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4e643cf-8167-42e3-a104-03ac5f7bd133" containerName="istio-proxy"
Oct 30 21:16:14.009607 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:14.009521    5713 cpu_manager.go:414] "containerMap: removing container" podUID="fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a" containerName="istio-proxy"
Oct 30 21:16:14.009607 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:14.009530    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a" containerName="istio-proxy"
Oct 30 21:16:14.009758 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:14.009728    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4e643cf-8167-42e3-a104-03ac5f7bd133" containerName="istio-proxy"
Oct 30 21:16:14.009758 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:14.009749    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd2ddd20-e0d4-4bbd-80c3-f953e9e8025a" containerName="istio-proxy"
Oct 30 21:16:14.015177 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:14.015153    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-08df98-699dbdd999-t8wvt"
Oct 30 21:16:14.070437 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:14.070404    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-08df98-699dbdd999-t8wvt"]
Oct 30 21:16:14.120981 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:14.115190    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-08df98-dockercfg-qpzm5\""
Oct 30 21:16:14.120981 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:14.115296    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-08df98\""
Oct 30 21:16:14.145819 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:14.128721    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aec20547-eca3-464b-9c2d-974ff1673459-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-08df98-699dbdd999-t8wvt\" (UID: \"aec20547-eca3-464b-9c2d-974ff1673459\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-08df98-699dbdd999-t8wvt"
Oct 30 21:16:14.145819 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:14.129080    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h5nx\" (UniqueName: \"kubernetes.io/projected/aec20547-eca3-464b-9c2d-974ff1673459-kube-api-access-5h5nx\") pod \"payload-pre-processing-e2e-inf-b-08df98-699dbdd999-t8wvt\" (UID: \"aec20547-eca3-464b-9c2d-974ff1673459\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-08df98-699dbdd999-t8wvt"
Oct 30 21:16:14.231609 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:14.231561    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h5nx\" (UniqueName: \"kubernetes.io/projected/aec20547-eca3-464b-9c2d-974ff1673459-kube-api-access-5h5nx\") pod \"payload-pre-processing-e2e-inf-b-08df98-699dbdd999-t8wvt\" (UID: \"aec20547-eca3-464b-9c2d-974ff1673459\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-08df98-699dbdd999-t8wvt"
Oct 30 21:16:14.231997 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:14.231643    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aec20547-eca3-464b-9c2d-974ff1673459-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-08df98-699dbdd999-t8wvt\" (UID: \"aec20547-eca3-464b-9c2d-974ff1673459\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-08df98-699dbdd999-t8wvt"
Oct 30 21:16:14.234327 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:14.234292    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aec20547-eca3-464b-9c2d-974ff1673459-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-08df98-699dbdd999-t8wvt\" (UID: \"aec20547-eca3-464b-9c2d-974ff1673459\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-08df98-699dbdd999-t8wvt"
Oct 30 21:16:14.235189 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:14.235167    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-08df98-57c8c9bdbd-2nsdc"]
Oct 30 21:16:14.236263 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:14.236235    5713 cpu_manager.go:414] "containerMap: removing container" podUID="e4e643cf-8167-42e3-a104-03ac5f7bd133" containerName="istio-proxy"
Oct 30 21:16:14.236263 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:14.236262    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4e643cf-8167-42e3-a104-03ac5f7bd133" containerName="istio-proxy"
Oct 30 21:16:14.236478 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:14.236462    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4e643cf-8167-42e3-a104-03ac5f7bd133" containerName="istio-proxy"
Oct 30 21:16:14.241851 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:14.241771    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-08df98-57c8c9bdbd-2nsdc"]
Oct 30 21:16:14.242023 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:14.241883    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-08df98-57c8c9bdbd-2nsdc"
Oct 30 21:16:14.291045 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:14.290974    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h5nx\" (UniqueName: \"kubernetes.io/projected/aec20547-eca3-464b-9c2d-974ff1673459-kube-api-access-5h5nx\") pod \"payload-pre-processing-e2e-inf-b-08df98-699dbdd999-t8wvt\" (UID: \"aec20547-eca3-464b-9c2d-974ff1673459\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-08df98-699dbdd999-t8wvt"
Oct 30 21:16:14.332805 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:14.332764    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-08df98-699dbdd999-t8wvt"
Oct 30 21:16:14.333284 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:14.333259    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgsw4\" (UniqueName: \"kubernetes.io/projected/3b0becd4-be23-402c-8870-a49d40b04f90-kube-api-access-wgsw4\") pod \"payload-processing-e2e-inf-b-08df98-57c8c9bdbd-2nsdc\" (UID: \"3b0becd4-be23-402c-8870-a49d40b04f90\") " pod="openshift-ingress/payload-processing-e2e-inf-b-08df98-57c8c9bdbd-2nsdc"
Oct 30 21:16:14.333460 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:14.333438    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b0becd4-be23-402c-8870-a49d40b04f90-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-08df98-57c8c9bdbd-2nsdc\" (UID: \"3b0becd4-be23-402c-8870-a49d40b04f90\") " pod="openshift-ingress/payload-processing-e2e-inf-b-08df98-57c8c9bdbd-2nsdc"
Oct 30 21:16:14.435366 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:14.435335    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgsw4\" (UniqueName: \"kubernetes.io/projected/3b0becd4-be23-402c-8870-a49d40b04f90-kube-api-access-wgsw4\") pod \"payload-processing-e2e-inf-b-08df98-57c8c9bdbd-2nsdc\" (UID: \"3b0becd4-be23-402c-8870-a49d40b04f90\") " pod="openshift-ingress/payload-processing-e2e-inf-b-08df98-57c8c9bdbd-2nsdc"
Oct 30 21:16:14.435537 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:14.435502    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b0becd4-be23-402c-8870-a49d40b04f90-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-08df98-57c8c9bdbd-2nsdc\" (UID: \"3b0becd4-be23-402c-8870-a49d40b04f90\") " pod="openshift-ingress/payload-processing-e2e-inf-b-08df98-57c8c9bdbd-2nsdc"
Oct 30 21:16:14.436467 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:14.436442    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b0becd4-be23-402c-8870-a49d40b04f90-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-08df98-57c8c9bdbd-2nsdc\" (UID: \"3b0becd4-be23-402c-8870-a49d40b04f90\") " pod="openshift-ingress/payload-processing-e2e-inf-b-08df98-57c8c9bdbd-2nsdc"
Oct 30 21:16:14.492826 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:14.492766    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgsw4\" (UniqueName: \"kubernetes.io/projected/3b0becd4-be23-402c-8870-a49d40b04f90-kube-api-access-wgsw4\") pod \"payload-processing-e2e-inf-b-08df98-57c8c9bdbd-2nsdc\" (UID: \"3b0becd4-be23-402c-8870-a49d40b04f90\") " pod="openshift-ingress/payload-processing-e2e-inf-b-08df98-57c8c9bdbd-2nsdc"
Oct 30 21:16:14.557497 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:14.557427    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-08df98-57c8c9bdbd-2nsdc"
Oct 30 21:16:14.715639 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:16:14.715610    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaec20547_eca3_464b_9c2d_974ff1673459.slice/crio-61dc6907418f316f391f91df3206b6b365b852c83ba6da6b6835bdd99df23db7 WatchSource:0}: Error finding container 61dc6907418f316f391f91df3206b6b365b852c83ba6da6b6835bdd99df23db7: Status 404 returned error can't find the container with id 61dc6907418f316f391f91df3206b6b365b852c83ba6da6b6835bdd99df23db7
Oct 30 21:16:14.716027 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:14.716000    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-08df98-699dbdd999-t8wvt"]
Oct 30 21:16:14.970249 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:14.970196    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-08df98-57c8c9bdbd-2nsdc"]
Oct 30 21:16:14.976372 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:16:14.976300    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b0becd4_be23_402c_8870_a49d40b04f90.slice/crio-8e2c45a305b6ff0e52b724bfbc0098c254215a30e1704d60d25513bea0072f06 WatchSource:0}: Error finding container 8e2c45a305b6ff0e52b724bfbc0098c254215a30e1704d60d25513bea0072f06: Status 404 returned error can't find the container with id 8e2c45a305b6ff0e52b724bfbc0098c254215a30e1704d60d25513bea0072f06
Oct 30 21:16:15.383550 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:15.383450    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-08df98-699dbdd999-t8wvt" event={"ID":"aec20547-eca3-464b-9c2d-974ff1673459","Type":"ContainerStarted","Data":"a175d0a592d30634c21dc1848654e188d7c1d47fdf5585e6b389fed68d368dc5"}
Oct 30 21:16:15.383550 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:15.383517    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-08df98-699dbdd999-t8wvt" event={"ID":"aec20547-eca3-464b-9c2d-974ff1673459","Type":"ContainerStarted","Data":"61dc6907418f316f391f91df3206b6b365b852c83ba6da6b6835bdd99df23db7"}
Oct 30 21:16:15.384124 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:15.383642    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-08df98-699dbdd999-t8wvt"
Oct 30 21:16:15.385260 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:15.385234    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-08df98-57c8c9bdbd-2nsdc" event={"ID":"3b0becd4-be23-402c-8870-a49d40b04f90","Type":"ContainerStarted","Data":"8eb1414e7fd9205b027baafcf20ec8f8cffbbe46552c04a25e2ab3747b6d8a15"}
Oct 30 21:16:15.385381 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:15.385277    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-08df98-57c8c9bdbd-2nsdc" event={"ID":"3b0becd4-be23-402c-8870-a49d40b04f90","Type":"ContainerStarted","Data":"8e2c45a305b6ff0e52b724bfbc0098c254215a30e1704d60d25513bea0072f06"}
Oct 30 21:16:15.385381 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:15.385339    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-08df98-57c8c9bdbd-2nsdc"
Oct 30 21:16:15.500335 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:15.500278    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-08df98-699dbdd999-t8wvt" podStartSLOduration=2.500263122 podStartE2EDuration="2.500263122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:15.497974282 +0000 UTC m=+2258.562750691" watchObservedRunningTime="2026-10-30 21:16:15.500263122 +0000 UTC m=+2258.565039499"
Oct 30 21:16:15.621691 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:15.621623    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-08df98-57c8c9bdbd-2nsdc" podStartSLOduration=1.621600179 podStartE2EDuration="1.621600179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:15.615955293 +0000 UTC m=+2258.680731703" watchObservedRunningTime="2026-10-30 21:16:15.621600179 +0000 UTC m=+2258.686376578"
Oct 30 21:16:24.462415 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:16:24.462375    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:16:26.397878 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:26.397845    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-08df98-699dbdd999-t8wvt"
Oct 30 21:16:26.399768 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:26.399745    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-08df98-57c8c9bdbd-2nsdc"
Oct 30 21:16:29.593556 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:29.593520    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b888d0-trcw2"]
Oct 30 21:16:29.643141 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:29.643106    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b888d0-trcw2"]
Oct 30 21:16:31.063969 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:31.063932    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="059f5085-59af-4a89-97ac-890c70f071b1" path="/var/lib/kubelet/pods/059f5085-59af-4a89-97ac-890c70f071b1/volumes"
Oct 30 21:16:36.926798 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:36.926752    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz"]
Oct 30 21:16:36.927223 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:36.927022    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz" podUID="a26a6cf1-a0b9-4470-93ea-557cae471765" containerName="istio-proxy" containerID="cri-o://8503397f8a4b7f19b6aebdbf1a4ce1477d04d95f031db242087d95b1a1834e01" gracePeriod=30
Oct 30 21:16:36.939454 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:36.939430    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5"]
Oct 30 21:16:36.939657 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:36.939623    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5" podUID="e3000e9c-41b7-4667-b9e1-c90ecf6c9b63" containerName="istio-proxy" containerID="cri-o://0adf7a2bfc102d7a99bc4e81c775046c82ee85f9a7463bbc28fb6db8fe9760f2" gracePeriod=30
Oct 30 21:16:37.807552 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:37.807522    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cb6404-hzrp4"]
Oct 30 21:16:37.811908 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:37.811888    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cb6404-hzrp4"
Oct 30 21:16:37.865806 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:37.865766    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cb6404-hzrp4"]
Oct 30 21:16:37.882745 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:37.882711    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svbhq\" (UniqueName: \"kubernetes.io/projected/f2ecbf1c-d317-4e83-98ac-4aea45a351c3-kube-api-access-svbhq\") pod \"maas-api-revoke-keys-e2e-api-b-cb6404-hzrp4\" (UID: \"f2ecbf1c-d317-4e83-98ac-4aea45a351c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cb6404-hzrp4"
Oct 30 21:16:37.882947 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:37.882869    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2ecbf1c-d317-4e83-98ac-4aea45a351c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-cb6404-hzrp4\" (UID: \"f2ecbf1c-d317-4e83-98ac-4aea45a351c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cb6404-hzrp4"
Oct 30 21:16:37.883037 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:37.883020    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2ecbf1c-d317-4e83-98ac-4aea45a351c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-cb6404-hzrp4\" (UID: \"f2ecbf1c-d317-4e83-98ac-4aea45a351c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cb6404-hzrp4"
Oct 30 21:16:37.910882 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:37.910839    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:16:37.911017 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:37.910990    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:16:37.911017 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:37.910991    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:16:37.911107 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:37.911051    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:16:37.984848 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:37.984780    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2ecbf1c-d317-4e83-98ac-4aea45a351c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-cb6404-hzrp4\" (UID: \"f2ecbf1c-d317-4e83-98ac-4aea45a351c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cb6404-hzrp4"
Oct 30 21:16:37.985245 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:37.984961    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2ecbf1c-d317-4e83-98ac-4aea45a351c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-cb6404-hzrp4\" (UID: \"f2ecbf1c-d317-4e83-98ac-4aea45a351c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cb6404-hzrp4"
Oct 30 21:16:37.985245 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:37.985092    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svbhq\" (UniqueName: \"kubernetes.io/projected/f2ecbf1c-d317-4e83-98ac-4aea45a351c3-kube-api-access-svbhq\") pod \"maas-api-revoke-keys-e2e-api-b-cb6404-hzrp4\" (UID: \"f2ecbf1c-d317-4e83-98ac-4aea45a351c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cb6404-hzrp4"
Oct 30 21:16:37.985488 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:37.985467    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2ecbf1c-d317-4e83-98ac-4aea45a351c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-cb6404-hzrp4\" (UID: \"f2ecbf1c-d317-4e83-98ac-4aea45a351c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cb6404-hzrp4"
Oct 30 21:16:37.987558 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:37.987536    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2ecbf1c-d317-4e83-98ac-4aea45a351c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-cb6404-hzrp4\" (UID: \"f2ecbf1c-d317-4e83-98ac-4aea45a351c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cb6404-hzrp4"
Oct 30 21:16:38.041947 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:38.041920    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svbhq\" (UniqueName: \"kubernetes.io/projected/f2ecbf1c-d317-4e83-98ac-4aea45a351c3-kube-api-access-svbhq\") pod \"maas-api-revoke-keys-e2e-api-b-cb6404-hzrp4\" (UID: \"f2ecbf1c-d317-4e83-98ac-4aea45a351c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cb6404-hzrp4"
Oct 30 21:16:38.127997 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:38.127916    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cb6404-hzrp4"
Oct 30 21:16:38.255630 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:38.255595    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2b425d-hdhgj"]
Oct 30 21:16:38.260352 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:38.260327    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2b425d-hdhgj"
Oct 30 21:16:38.316712 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:38.316677    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2b425d-hdhgj"]
Oct 30 21:16:38.391545 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:38.391473    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94effd3f-fc12-433c-aad6-82f48b4ec4fa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-2b425d-hdhgj\" (UID: \"94effd3f-fc12-433c-aad6-82f48b4ec4fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2b425d-hdhgj"
Oct 30 21:16:38.391698 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:38.391637    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94effd3f-fc12-433c-aad6-82f48b4ec4fa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-2b425d-hdhgj\" (UID: \"94effd3f-fc12-433c-aad6-82f48b4ec4fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2b425d-hdhgj"
Oct 30 21:16:38.391842 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:38.391828    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdk8x\" (UniqueName: \"kubernetes.io/projected/94effd3f-fc12-433c-aad6-82f48b4ec4fa-kube-api-access-zdk8x\") pod \"maas-api-revoke-keys-e2e-worker-w5-2b425d-hdhgj\" (UID: \"94effd3f-fc12-433c-aad6-82f48b4ec4fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2b425d-hdhgj"
Oct 30 21:16:38.493363 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:38.493320    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdk8x\" (UniqueName: \"kubernetes.io/projected/94effd3f-fc12-433c-aad6-82f48b4ec4fa-kube-api-access-zdk8x\") pod \"maas-api-revoke-keys-e2e-worker-w5-2b425d-hdhgj\" (UID: \"94effd3f-fc12-433c-aad6-82f48b4ec4fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2b425d-hdhgj"
Oct 30 21:16:38.493570 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:38.493455    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94effd3f-fc12-433c-aad6-82f48b4ec4fa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-2b425d-hdhgj\" (UID: \"94effd3f-fc12-433c-aad6-82f48b4ec4fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2b425d-hdhgj"
Oct 30 21:16:38.493570 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:38.493567    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94effd3f-fc12-433c-aad6-82f48b4ec4fa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-2b425d-hdhgj\" (UID: \"94effd3f-fc12-433c-aad6-82f48b4ec4fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2b425d-hdhgj"
Oct 30 21:16:38.494241 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:38.494212    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94effd3f-fc12-433c-aad6-82f48b4ec4fa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-2b425d-hdhgj\" (UID: \"94effd3f-fc12-433c-aad6-82f48b4ec4fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2b425d-hdhgj"
Oct 30 21:16:38.495967 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:38.495945    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94effd3f-fc12-433c-aad6-82f48b4ec4fa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-2b425d-hdhgj\" (UID: \"94effd3f-fc12-433c-aad6-82f48b4ec4fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2b425d-hdhgj"
Oct 30 21:16:38.506774 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:38.506753    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cb6404-hzrp4"]
Oct 30 21:16:38.508705 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:16:38.508681    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2ecbf1c_d317_4e83_98ac_4aea45a351c3.slice/crio-32c3db39d51db91ffba0c3ded8ab2f9c37a991323b7c034d74a1c3ceb9603af2 WatchSource:0}: Error finding container 32c3db39d51db91ffba0c3ded8ab2f9c37a991323b7c034d74a1c3ceb9603af2: Status 404 returned error can't find the container with id 32c3db39d51db91ffba0c3ded8ab2f9c37a991323b7c034d74a1c3ceb9603af2
Oct 30 21:16:38.553660 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:38.553635    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdk8x\" (UniqueName: \"kubernetes.io/projected/94effd3f-fc12-433c-aad6-82f48b4ec4fa-kube-api-access-zdk8x\") pod \"maas-api-revoke-keys-e2e-worker-w5-2b425d-hdhgj\" (UID: \"94effd3f-fc12-433c-aad6-82f48b4ec4fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2b425d-hdhgj"
Oct 30 21:16:38.577841 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:38.577818    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2b425d-hdhgj"
Oct 30 21:16:38.595719 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:38.595686    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cb6404-hzrp4" event={"ID":"f2ecbf1c-d317-4e83-98ac-4aea45a351c3","Type":"ContainerStarted","Data":"32c3db39d51db91ffba0c3ded8ab2f9c37a991323b7c034d74a1c3ceb9603af2"}
Oct 30 21:16:38.961398 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:16:38.961356    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94effd3f_fc12_433c_aad6_82f48b4ec4fa.slice/crio-52f944a120d18bce33c8af3c44f04f36bc9754dea476f882d3290c4cf733f7aa WatchSource:0}: Error finding container 52f944a120d18bce33c8af3c44f04f36bc9754dea476f882d3290c4cf733f7aa: Status 404 returned error can't find the container with id 52f944a120d18bce33c8af3c44f04f36bc9754dea476f882d3290c4cf733f7aa
Oct 30 21:16:38.961537 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:38.961462    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2b425d-hdhgj"]
Oct 30 21:16:39.606160 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:39.606122    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2b425d-hdhgj" event={"ID":"94effd3f-fc12-433c-aad6-82f48b4ec4fa","Type":"ContainerStarted","Data":"88dd3ed8d9891d89f0acc3ecf3065dc73774069844d401cbbbea4280b1e6247d"}
Oct 30 21:16:39.606588 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:39.606186    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2b425d-hdhgj" event={"ID":"94effd3f-fc12-433c-aad6-82f48b4ec4fa","Type":"ContainerStarted","Data":"52f944a120d18bce33c8af3c44f04f36bc9754dea476f882d3290c4cf733f7aa"}
Oct 30 21:16:39.607939 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:39.607917    5713 generic.go:356] "Generic (PLEG): container finished" podID="f2ecbf1c-d317-4e83-98ac-4aea45a351c3" containerID="3fcd580fdec568aaf4816fe6d37463cb9eb543152672db9bd2deec13e1d11073" exitCode=0
Oct 30 21:16:39.608019 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:39.607953    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cb6404-hzrp4" event={"ID":"f2ecbf1c-d317-4e83-98ac-4aea45a351c3","Type":"ContainerDied","Data":"3fcd580fdec568aaf4816fe6d37463cb9eb543152672db9bd2deec13e1d11073"}
Oct 30 21:16:39.722098 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:39.722050    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2b425d-hdhgj" podStartSLOduration=1.722034753 podStartE2EDuration="1.722034753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:39.719090938 +0000 UTC m=+2282.783867337" watchObservedRunningTime="2026-10-30 21:16:39.722034753 +0000 UTC m=+2282.786811185"
Oct 30 21:16:40.618672 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:40.618634    5713 generic.go:356] "Generic (PLEG): container finished" podID="94effd3f-fc12-433c-aad6-82f48b4ec4fa" containerID="88dd3ed8d9891d89f0acc3ecf3065dc73774069844d401cbbbea4280b1e6247d" exitCode=0
Oct 30 21:16:40.619149 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:40.618773    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2b425d-hdhgj" event={"ID":"94effd3f-fc12-433c-aad6-82f48b4ec4fa","Type":"ContainerDied","Data":"88dd3ed8d9891d89f0acc3ecf3065dc73774069844d401cbbbea4280b1e6247d"}
Oct 30 21:16:40.772257 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:40.772226    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cb6404-hzrp4"
Oct 30 21:16:40.826976 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:40.826939    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2ecbf1c-d317-4e83-98ac-4aea45a351c3-kube-api-access-svbhq\" (UniqueName: \"kubernetes.io/projected/f2ecbf1c-d317-4e83-98ac-4aea45a351c3-kube-api-access-svbhq\") pod \"f2ecbf1c-d317-4e83-98ac-4aea45a351c3\" (UID: \"f2ecbf1c-d317-4e83-98ac-4aea45a351c3\") "
Oct 30 21:16:40.827138 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:40.827051    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2ecbf1c-d317-4e83-98ac-4aea45a351c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2ecbf1c-d317-4e83-98ac-4aea45a351c3-maas-api-xks-ca\") pod \"f2ecbf1c-d317-4e83-98ac-4aea45a351c3\" (UID: \"f2ecbf1c-d317-4e83-98ac-4aea45a351c3\") "
Oct 30 21:16:40.827138 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:40.827115    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2ecbf1c-d317-4e83-98ac-4aea45a351c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2ecbf1c-d317-4e83-98ac-4aea45a351c3-maas-api-openshift-ca\") pod \"f2ecbf1c-d317-4e83-98ac-4aea45a351c3\" (UID: \"f2ecbf1c-d317-4e83-98ac-4aea45a351c3\") "
Oct 30 21:16:40.827494 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:40.827464    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2ecbf1c-d317-4e83-98ac-4aea45a351c3-maas-api-openshift-ca" pod "f2ecbf1c-d317-4e83-98ac-4aea45a351c3" (UID: "f2ecbf1c-d317-4e83-98ac-4aea45a351c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:16:40.827769 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:40.827674    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2ecbf1c-d317-4e83-98ac-4aea45a351c3-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:16:40.829283 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:40.829257    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2ecbf1c-d317-4e83-98ac-4aea45a351c3-kube-api-access-svbhq" pod "f2ecbf1c-d317-4e83-98ac-4aea45a351c3" (UID: "f2ecbf1c-d317-4e83-98ac-4aea45a351c3"). InnerVolumeSpecName "kube-api-access-svbhq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:16:40.829417 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:40.829315    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2ecbf1c-d317-4e83-98ac-4aea45a351c3-maas-api-xks-ca" pod "f2ecbf1c-d317-4e83-98ac-4aea45a351c3" (UID: "f2ecbf1c-d317-4e83-98ac-4aea45a351c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:16:40.929342 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:40.929269    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svbhq\" (UniqueName: \"kubernetes.io/projected/f2ecbf1c-d317-4e83-98ac-4aea45a351c3-kube-api-access-svbhq\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:16:40.929342 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:40.929298    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2ecbf1c-d317-4e83-98ac-4aea45a351c3-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:16:41.628324 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:41.628294    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cb6404-hzrp4"
Oct 30 21:16:41.628728 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:41.628334    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cb6404-hzrp4" event={"ID":"f2ecbf1c-d317-4e83-98ac-4aea45a351c3","Type":"ContainerDied","Data":"32c3db39d51db91ffba0c3ded8ab2f9c37a991323b7c034d74a1c3ceb9603af2"}
Oct 30 21:16:41.628728 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:41.628418    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32c3db39d51db91ffba0c3ded8ab2f9c37a991323b7c034d74a1c3ceb9603af2"
Oct 30 21:16:41.785422 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:41.785398    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2b425d-hdhgj"
Oct 30 21:16:41.839899 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:41.839876    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94effd3f-fc12-433c-aad6-82f48b4ec4fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94effd3f-fc12-433c-aad6-82f48b4ec4fa-maas-api-xks-ca\") pod \"94effd3f-fc12-433c-aad6-82f48b4ec4fa\" (UID: \"94effd3f-fc12-433c-aad6-82f48b4ec4fa\") "
Oct 30 21:16:41.840088 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:41.840075    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94effd3f-fc12-433c-aad6-82f48b4ec4fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94effd3f-fc12-433c-aad6-82f48b4ec4fa-maas-api-openshift-ca\") pod \"94effd3f-fc12-433c-aad6-82f48b4ec4fa\" (UID: \"94effd3f-fc12-433c-aad6-82f48b4ec4fa\") "
Oct 30 21:16:41.840137 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:41.840110    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94effd3f-fc12-433c-aad6-82f48b4ec4fa-kube-api-access-zdk8x\" (UniqueName: \"kubernetes.io/projected/94effd3f-fc12-433c-aad6-82f48b4ec4fa-kube-api-access-zdk8x\") pod \"94effd3f-fc12-433c-aad6-82f48b4ec4fa\" (UID: \"94effd3f-fc12-433c-aad6-82f48b4ec4fa\") "
Oct 30 21:16:41.840399 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:41.840376    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94effd3f-fc12-433c-aad6-82f48b4ec4fa-maas-api-openshift-ca" pod "94effd3f-fc12-433c-aad6-82f48b4ec4fa" (UID: "94effd3f-fc12-433c-aad6-82f48b4ec4fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:16:41.840665 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:41.840617    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94effd3f-fc12-433c-aad6-82f48b4ec4fa-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:16:41.842146 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:41.842127    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94effd3f-fc12-433c-aad6-82f48b4ec4fa-maas-api-xks-ca" pod "94effd3f-fc12-433c-aad6-82f48b4ec4fa" (UID: "94effd3f-fc12-433c-aad6-82f48b4ec4fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:16:41.842562 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:41.842530    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94effd3f-fc12-433c-aad6-82f48b4ec4fa-kube-api-access-zdk8x" pod "94effd3f-fc12-433c-aad6-82f48b4ec4fa" (UID: "94effd3f-fc12-433c-aad6-82f48b4ec4fa"). InnerVolumeSpecName "kube-api-access-zdk8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:16:41.942269 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:41.942228    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdk8x\" (UniqueName: \"kubernetes.io/projected/94effd3f-fc12-433c-aad6-82f48b4ec4fa-kube-api-access-zdk8x\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:16:41.942269 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:41.942259    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94effd3f-fc12-433c-aad6-82f48b4ec4fa-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:16:42.203324 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.203249    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz"
Oct 30 21:16:42.250306 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.250283    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5"
Oct 30 21:16:42.347465 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.347428    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a26a6cf1-a0b9-4470-93ea-557cae471765-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a26a6cf1-a0b9-4470-93ea-557cae471765-workload-certs\") pod \"a26a6cf1-a0b9-4470-93ea-557cae471765\" (UID: \"a26a6cf1-a0b9-4470-93ea-557cae471765\") "
Oct 30 21:16:42.347658 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.347492    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a26a6cf1-a0b9-4470-93ea-557cae471765-istio-token\" (UniqueName: \"kubernetes.io/projected/a26a6cf1-a0b9-4470-93ea-557cae471765-istio-token\") pod \"a26a6cf1-a0b9-4470-93ea-557cae471765\" (UID: \"a26a6cf1-a0b9-4470-93ea-557cae471765\") "
Oct 30 21:16:42.347658 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.347538    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e3000e9c-41b7-4667-b9e1-c90ecf6c9b63-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3000e9c-41b7-4667-b9e1-c90ecf6c9b63-istio-podinfo\") pod \"e3000e9c-41b7-4667-b9e1-c90ecf6c9b63\" (UID: \"e3000e9c-41b7-4667-b9e1-c90ecf6c9b63\") "
Oct 30 21:16:42.347658 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.347584    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a26a6cf1-a0b9-4470-93ea-557cae471765-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a26a6cf1-a0b9-4470-93ea-557cae471765-istio-data\") pod \"a26a6cf1-a0b9-4470-93ea-557cae471765\" (UID: \"a26a6cf1-a0b9-4470-93ea-557cae471765\") "
Oct 30 21:16:42.347658 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.347632    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3000e9c-41b7-4667-b9e1-c90ecf6c9b63-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3000e9c-41b7-4667-b9e1-c90ecf6c9b63-istio-data\") pod \"e3000e9c-41b7-4667-b9e1-c90ecf6c9b63\" (UID: \"e3000e9c-41b7-4667-b9e1-c90ecf6c9b63\") "
Oct 30 21:16:42.347936 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.347675    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3000e9c-41b7-4667-b9e1-c90ecf6c9b63-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3000e9c-41b7-4667-b9e1-c90ecf6c9b63-workload-socket\") pod \"e3000e9c-41b7-4667-b9e1-c90ecf6c9b63\" (UID: \"e3000e9c-41b7-4667-b9e1-c90ecf6c9b63\") "
Oct 30 21:16:42.347936 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.347746    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a26a6cf1-a0b9-4470-93ea-557cae471765-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a26a6cf1-a0b9-4470-93ea-557cae471765-workload-socket\") pod \"a26a6cf1-a0b9-4470-93ea-557cae471765\" (UID: \"a26a6cf1-a0b9-4470-93ea-557cae471765\") "
Oct 30 21:16:42.347936 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.347775    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a26a6cf1-a0b9-4470-93ea-557cae471765-workload-certs" pod "a26a6cf1-a0b9-4470-93ea-557cae471765" (UID: "a26a6cf1-a0b9-4470-93ea-557cae471765"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:16:42.347936 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.347822    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a26a6cf1-a0b9-4470-93ea-557cae471765-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a26a6cf1-a0b9-4470-93ea-557cae471765-istio-envoy\") pod \"a26a6cf1-a0b9-4470-93ea-557cae471765\" (UID: \"a26a6cf1-a0b9-4470-93ea-557cae471765\") "
Oct 30 21:16:42.347936 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.347871    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3000e9c-41b7-4667-b9e1-c90ecf6c9b63-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3000e9c-41b7-4667-b9e1-c90ecf6c9b63-istiod-ca-cert\") pod \"e3000e9c-41b7-4667-b9e1-c90ecf6c9b63\" (UID: \"e3000e9c-41b7-4667-b9e1-c90ecf6c9b63\") "
Oct 30 21:16:42.348231 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.347961    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3000e9c-41b7-4667-b9e1-c90ecf6c9b63-kube-api-access-98z4s\" (UniqueName: \"kubernetes.io/projected/e3000e9c-41b7-4667-b9e1-c90ecf6c9b63-kube-api-access-98z4s\") pod \"e3000e9c-41b7-4667-b9e1-c90ecf6c9b63\" (UID: \"e3000e9c-41b7-4667-b9e1-c90ecf6c9b63\") "
Oct 30 21:16:42.348231 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.347991    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a26a6cf1-a0b9-4470-93ea-557cae471765-kube-api-access-l6tj2\" (UniqueName: \"kubernetes.io/projected/a26a6cf1-a0b9-4470-93ea-557cae471765-kube-api-access-l6tj2\") pod \"a26a6cf1-a0b9-4470-93ea-557cae471765\" (UID: \"a26a6cf1-a0b9-4470-93ea-557cae471765\") "
Oct 30 21:16:42.348231 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.347998    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a26a6cf1-a0b9-4470-93ea-557cae471765-istio-data" pod "a26a6cf1-a0b9-4470-93ea-557cae471765" (UID: "a26a6cf1-a0b9-4470-93ea-557cae471765"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:16:42.348231 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.348113    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3000e9c-41b7-4667-b9e1-c90ecf6c9b63-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3000e9c-41b7-4667-b9e1-c90ecf6c9b63-workload-certs\") pod \"e3000e9c-41b7-4667-b9e1-c90ecf6c9b63\" (UID: \"e3000e9c-41b7-4667-b9e1-c90ecf6c9b63\") "
Oct 30 21:16:42.348231 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.348152    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a26a6cf1-a0b9-4470-93ea-557cae471765-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a26a6cf1-a0b9-4470-93ea-557cae471765-credential-socket\") pod \"a26a6cf1-a0b9-4470-93ea-557cae471765\" (UID: \"a26a6cf1-a0b9-4470-93ea-557cae471765\") "
Oct 30 21:16:42.348231 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.348188    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3000e9c-41b7-4667-b9e1-c90ecf6c9b63-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3000e9c-41b7-4667-b9e1-c90ecf6c9b63-istio-envoy\") pod \"e3000e9c-41b7-4667-b9e1-c90ecf6c9b63\" (UID: \"e3000e9c-41b7-4667-b9e1-c90ecf6c9b63\") "
Oct 30 21:16:42.348231 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.348229    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a26a6cf1-a0b9-4470-93ea-557cae471765-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a26a6cf1-a0b9-4470-93ea-557cae471765-istiod-ca-cert\") pod \"a26a6cf1-a0b9-4470-93ea-557cae471765\" (UID: \"a26a6cf1-a0b9-4470-93ea-557cae471765\") "
Oct 30 21:16:42.348804 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.348278    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3000e9c-41b7-4667-b9e1-c90ecf6c9b63-istio-token\" (UniqueName: \"kubernetes.io/projected/e3000e9c-41b7-4667-b9e1-c90ecf6c9b63-istio-token\") pod \"e3000e9c-41b7-4667-b9e1-c90ecf6c9b63\" (UID: \"e3000e9c-41b7-4667-b9e1-c90ecf6c9b63\") "
Oct 30 21:16:42.348804 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.348320    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3000e9c-41b7-4667-b9e1-c90ecf6c9b63-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3000e9c-41b7-4667-b9e1-c90ecf6c9b63-credential-socket\") pod \"e3000e9c-41b7-4667-b9e1-c90ecf6c9b63\" (UID: \"e3000e9c-41b7-4667-b9e1-c90ecf6c9b63\") "
Oct 30 21:16:42.348804 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.348349    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a26a6cf1-a0b9-4470-93ea-557cae471765-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a26a6cf1-a0b9-4470-93ea-557cae471765-istio-podinfo\") pod \"a26a6cf1-a0b9-4470-93ea-557cae471765\" (UID: \"a26a6cf1-a0b9-4470-93ea-557cae471765\") "
Oct 30 21:16:42.348995 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.348111    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3000e9c-41b7-4667-b9e1-c90ecf6c9b63-workload-socket" pod "e3000e9c-41b7-4667-b9e1-c90ecf6c9b63" (UID: "e3000e9c-41b7-4667-b9e1-c90ecf6c9b63"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:16:42.348995 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.348427    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a26a6cf1-a0b9-4470-93ea-557cae471765-workload-socket" pod "a26a6cf1-a0b9-4470-93ea-557cae471765" (UID: "a26a6cf1-a0b9-4470-93ea-557cae471765"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:16:42.348995 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.348509    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3000e9c-41b7-4667-b9e1-c90ecf6c9b63-istiod-ca-cert" pod "e3000e9c-41b7-4667-b9e1-c90ecf6c9b63" (UID: "e3000e9c-41b7-4667-b9e1-c90ecf6c9b63"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:16:42.348995 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.348689    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3000e9c-41b7-4667-b9e1-c90ecf6c9b63-istio-data" pod "e3000e9c-41b7-4667-b9e1-c90ecf6c9b63" (UID: "e3000e9c-41b7-4667-b9e1-c90ecf6c9b63"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:16:42.349226 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.349024    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3000e9c-41b7-4667-b9e1-c90ecf6c9b63-credential-socket" pod "e3000e9c-41b7-4667-b9e1-c90ecf6c9b63" (UID: "e3000e9c-41b7-4667-b9e1-c90ecf6c9b63"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:16:42.349332 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.349258    5713 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a26a6cf1-a0b9-4470-93ea-557cae471765-workload-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:16:42.349445 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.349403    5713 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a26a6cf1-a0b9-4470-93ea-557cae471765-istio-data\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:16:42.349515 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.349296    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a26a6cf1-a0b9-4470-93ea-557cae471765-istiod-ca-cert" pod "a26a6cf1-a0b9-4470-93ea-557cae471765" (UID: "a26a6cf1-a0b9-4470-93ea-557cae471765"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:16:42.349515 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.349345    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3000e9c-41b7-4667-b9e1-c90ecf6c9b63-workload-certs" pod "e3000e9c-41b7-4667-b9e1-c90ecf6c9b63" (UID: "e3000e9c-41b7-4667-b9e1-c90ecf6c9b63"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:16:42.349627 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.349545    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a26a6cf1-a0b9-4470-93ea-557cae471765-credential-socket" pod "a26a6cf1-a0b9-4470-93ea-557cae471765" (UID: "a26a6cf1-a0b9-4470-93ea-557cae471765"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:16:42.351599 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.351562    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a26a6cf1-a0b9-4470-93ea-557cae471765-istio-envoy" pod "a26a6cf1-a0b9-4470-93ea-557cae471765" (UID: "a26a6cf1-a0b9-4470-93ea-557cae471765"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:16:42.351725 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.351586    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a26a6cf1-a0b9-4470-93ea-557cae471765-kube-api-access-l6tj2" pod "a26a6cf1-a0b9-4470-93ea-557cae471765" (UID: "a26a6cf1-a0b9-4470-93ea-557cae471765"). InnerVolumeSpecName "kube-api-access-l6tj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:16:42.352041 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.351995    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3000e9c-41b7-4667-b9e1-c90ecf6c9b63-istio-envoy" pod "e3000e9c-41b7-4667-b9e1-c90ecf6c9b63" (UID: "e3000e9c-41b7-4667-b9e1-c90ecf6c9b63"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:16:42.352041 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.352005    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3000e9c-41b7-4667-b9e1-c90ecf6c9b63-kube-api-access-98z4s" pod "e3000e9c-41b7-4667-b9e1-c90ecf6c9b63" (UID: "e3000e9c-41b7-4667-b9e1-c90ecf6c9b63"). InnerVolumeSpecName "kube-api-access-98z4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:16:42.352754 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.352667    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a26a6cf1-a0b9-4470-93ea-557cae471765-istio-token" pod "a26a6cf1-a0b9-4470-93ea-557cae471765" (UID: "a26a6cf1-a0b9-4470-93ea-557cae471765"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:16:42.353136 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.353112    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3000e9c-41b7-4667-b9e1-c90ecf6c9b63-istio-token" pod "e3000e9c-41b7-4667-b9e1-c90ecf6c9b63" (UID: "e3000e9c-41b7-4667-b9e1-c90ecf6c9b63"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:16:42.353267 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.353192    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a26a6cf1-a0b9-4470-93ea-557cae471765-istio-podinfo" pod "a26a6cf1-a0b9-4470-93ea-557cae471765" (UID: "a26a6cf1-a0b9-4470-93ea-557cae471765"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:16:42.353634 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.353609    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3000e9c-41b7-4667-b9e1-c90ecf6c9b63-istio-podinfo" pod "e3000e9c-41b7-4667-b9e1-c90ecf6c9b63" (UID: "e3000e9c-41b7-4667-b9e1-c90ecf6c9b63"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:16:42.451458 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.451424    5713 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3000e9c-41b7-4667-b9e1-c90ecf6c9b63-istio-data\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:16:42.451458 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.451461    5713 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3000e9c-41b7-4667-b9e1-c90ecf6c9b63-workload-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:16:42.451652 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.451473    5713 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a26a6cf1-a0b9-4470-93ea-557cae471765-workload-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:16:42.451652 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.451486    5713 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a26a6cf1-a0b9-4470-93ea-557cae471765-istio-envoy\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:16:42.451652 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.451499    5713 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3000e9c-41b7-4667-b9e1-c90ecf6c9b63-istiod-ca-cert\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:16:42.451652 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.451514    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98z4s\" (UniqueName: \"kubernetes.io/projected/e3000e9c-41b7-4667-b9e1-c90ecf6c9b63-kube-api-access-98z4s\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:16:42.451652 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.451529    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6tj2\" (UniqueName: \"kubernetes.io/projected/a26a6cf1-a0b9-4470-93ea-557cae471765-kube-api-access-l6tj2\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:16:42.451652 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.451544    5713 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3000e9c-41b7-4667-b9e1-c90ecf6c9b63-workload-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:16:42.451652 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.451557    5713 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a26a6cf1-a0b9-4470-93ea-557cae471765-credential-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:16:42.451652 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.451568    5713 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3000e9c-41b7-4667-b9e1-c90ecf6c9b63-istio-envoy\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:16:42.451652 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.451580    5713 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a26a6cf1-a0b9-4470-93ea-557cae471765-istiod-ca-cert\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:16:42.451652 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.451592    5713 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3000e9c-41b7-4667-b9e1-c90ecf6c9b63-istio-token\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:16:42.451652 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.451604    5713 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3000e9c-41b7-4667-b9e1-c90ecf6c9b63-credential-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:16:42.451652 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.451617    5713 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a26a6cf1-a0b9-4470-93ea-557cae471765-istio-podinfo\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:16:42.451652 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.451631    5713 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a26a6cf1-a0b9-4470-93ea-557cae471765-istio-token\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:16:42.451652 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.451643    5713 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3000e9c-41b7-4667-b9e1-c90ecf6c9b63-istio-podinfo\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:16:42.637110 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.637020    5713 generic.go:356] "Generic (PLEG): container finished" podID="a26a6cf1-a0b9-4470-93ea-557cae471765" containerID="8503397f8a4b7f19b6aebdbf1a4ce1477d04d95f031db242087d95b1a1834e01" exitCode=0
Oct 30 21:16:42.637110 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.637073    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz"
Oct 30 21:16:42.637667 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.637111    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz" event={"ID":"a26a6cf1-a0b9-4470-93ea-557cae471765","Type":"ContainerDied","Data":"8503397f8a4b7f19b6aebdbf1a4ce1477d04d95f031db242087d95b1a1834e01"}
Oct 30 21:16:42.637667 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.637166    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz" event={"ID":"a26a6cf1-a0b9-4470-93ea-557cae471765","Type":"ContainerDied","Data":"0e8d8cae54e5943d5c8047f811d2b73a37c3bca9d6f3e5d67bb921712e3233a3"}
Oct 30 21:16:42.637667 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.637194    5713 scope.go:122] "RemoveContainer" containerID="8503397f8a4b7f19b6aebdbf1a4ce1477d04d95f031db242087d95b1a1834e01"
Oct 30 21:16:42.639277 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.639254    5713 generic.go:356] "Generic (PLEG): container finished" podID="e3000e9c-41b7-4667-b9e1-c90ecf6c9b63" containerID="0adf7a2bfc102d7a99bc4e81c775046c82ee85f9a7463bbc28fb6db8fe9760f2" exitCode=0
Oct 30 21:16:42.639380 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.639293    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5" event={"ID":"e3000e9c-41b7-4667-b9e1-c90ecf6c9b63","Type":"ContainerDied","Data":"0adf7a2bfc102d7a99bc4e81c775046c82ee85f9a7463bbc28fb6db8fe9760f2"}
Oct 30 21:16:42.639380 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.639358    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5" event={"ID":"e3000e9c-41b7-4667-b9e1-c90ecf6c9b63","Type":"ContainerDied","Data":"47cb405db7a850b2dec06d05e817bdd6d83202f5d089033655351c6b263fd8ca"}
Oct 30 21:16:42.639484 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.639305    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5"
Oct 30 21:16:42.641152 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.641136    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2b425d-hdhgj"
Oct 30 21:16:42.641255 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.641234    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2b425d-hdhgj" event={"ID":"94effd3f-fc12-433c-aad6-82f48b4ec4fa","Type":"ContainerDied","Data":"52f944a120d18bce33c8af3c44f04f36bc9754dea476f882d3290c4cf733f7aa"}
Oct 30 21:16:42.641325 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.641279    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52f944a120d18bce33c8af3c44f04f36bc9754dea476f882d3290c4cf733f7aa"
Oct 30 21:16:42.651055 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.651039    5713 scope.go:122] "RemoveContainer" containerID="8503397f8a4b7f19b6aebdbf1a4ce1477d04d95f031db242087d95b1a1834e01"
Oct 30 21:16:42.651395 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:16:42.651372    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8503397f8a4b7f19b6aebdbf1a4ce1477d04d95f031db242087d95b1a1834e01\": container with ID starting with 8503397f8a4b7f19b6aebdbf1a4ce1477d04d95f031db242087d95b1a1834e01 not found: ID does not exist" containerID="8503397f8a4b7f19b6aebdbf1a4ce1477d04d95f031db242087d95b1a1834e01"
Oct 30 21:16:42.651455 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.651404    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8503397f8a4b7f19b6aebdbf1a4ce1477d04d95f031db242087d95b1a1834e01"} err="failed to get container status \"8503397f8a4b7f19b6aebdbf1a4ce1477d04d95f031db242087d95b1a1834e01\": rpc error: code = NotFound desc = could not find container \"8503397f8a4b7f19b6aebdbf1a4ce1477d04d95f031db242087d95b1a1834e01\": container with ID starting with 8503397f8a4b7f19b6aebdbf1a4ce1477d04d95f031db242087d95b1a1834e01 not found: ID does not exist"
Oct 30 21:16:42.651455 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.651427    5713 scope.go:122] "RemoveContainer" containerID="0adf7a2bfc102d7a99bc4e81c775046c82ee85f9a7463bbc28fb6db8fe9760f2"
Oct 30 21:16:42.662030 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.662012    5713 scope.go:122] "RemoveContainer" containerID="75d41fbab6583d1312c43e434c531e6b47564a0ab9ed3aacb7382a1705032798"
Oct 30 21:16:42.671917 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.671900    5713 scope.go:122] "RemoveContainer" containerID="0adf7a2bfc102d7a99bc4e81c775046c82ee85f9a7463bbc28fb6db8fe9760f2"
Oct 30 21:16:42.672191 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:16:42.672175    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0adf7a2bfc102d7a99bc4e81c775046c82ee85f9a7463bbc28fb6db8fe9760f2\": container with ID starting with 0adf7a2bfc102d7a99bc4e81c775046c82ee85f9a7463bbc28fb6db8fe9760f2 not found: ID does not exist" containerID="0adf7a2bfc102d7a99bc4e81c775046c82ee85f9a7463bbc28fb6db8fe9760f2"
Oct 30 21:16:42.672253 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.672201    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0adf7a2bfc102d7a99bc4e81c775046c82ee85f9a7463bbc28fb6db8fe9760f2"} err="failed to get container status \"0adf7a2bfc102d7a99bc4e81c775046c82ee85f9a7463bbc28fb6db8fe9760f2\": rpc error: code = NotFound desc = could not find container \"0adf7a2bfc102d7a99bc4e81c775046c82ee85f9a7463bbc28fb6db8fe9760f2\": container with ID starting with 0adf7a2bfc102d7a99bc4e81c775046c82ee85f9a7463bbc28fb6db8fe9760f2 not found: ID does not exist"
Oct 30 21:16:42.672253 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.672217    5713 scope.go:122] "RemoveContainer" containerID="75d41fbab6583d1312c43e434c531e6b47564a0ab9ed3aacb7382a1705032798"
Oct 30 21:16:42.672433 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:16:42.672418    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75d41fbab6583d1312c43e434c531e6b47564a0ab9ed3aacb7382a1705032798\": container with ID starting with 75d41fbab6583d1312c43e434c531e6b47564a0ab9ed3aacb7382a1705032798 not found: ID does not exist" containerID="75d41fbab6583d1312c43e434c531e6b47564a0ab9ed3aacb7382a1705032798"
Oct 30 21:16:42.672468 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.672437    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75d41fbab6583d1312c43e434c531e6b47564a0ab9ed3aacb7382a1705032798"} err="failed to get container status \"75d41fbab6583d1312c43e434c531e6b47564a0ab9ed3aacb7382a1705032798\": rpc error: code = NotFound desc = could not find container \"75d41fbab6583d1312c43e434c531e6b47564a0ab9ed3aacb7382a1705032798\": container with ID starting with 75d41fbab6583d1312c43e434c531e6b47564a0ab9ed3aacb7382a1705032798 not found: ID does not exist"
Oct 30 21:16:42.730926 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.730899    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-2b425d/distinct-2-w5-2b425d-kserve-6694867c68-bzpvn"]
Oct 30 21:16:42.747153 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.747127    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-2b425d/premium-w5-2b425d-kserve-78756b6c5b-9x5xb"]
Oct 30 21:16:42.759956 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.759917    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-2b425d/simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk"]
Oct 30 21:16:42.807582 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.807545    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5"]
Oct 30 21:16:42.814715 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.814682    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-4w7f5"]
Oct 30 21:16:42.922097 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.922018    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz"]
Oct 30 21:16:42.928235 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:42.928205    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-b888d0-openshift-default-65d785fbfc-cj9mz"]
Oct 30 21:16:43.064239 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:43.064211    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a26a6cf1-a0b9-4470-93ea-557cae471765" path="/var/lib/kubelet/pods/a26a6cf1-a0b9-4470-93ea-557cae471765/volumes"
Oct 30 21:16:43.064684 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:43.064670    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3000e9c-41b7-4667-b9e1-c90ecf6c9b63" path="/var/lib/kubelet/pods/e3000e9c-41b7-4667-b9e1-c90ecf6c9b63/volumes"
Oct 30 21:16:50.110205 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:50.110125    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-vmz5r"]
Oct 30 21:16:50.111153 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:50.111127    5713 cpu_manager.go:414] "containerMap: removing container" podUID="e3000e9c-41b7-4667-b9e1-c90ecf6c9b63" containerName="istio-proxy"
Oct 30 21:16:50.111153 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:50.111152    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3000e9c-41b7-4667-b9e1-c90ecf6c9b63" containerName="istio-proxy"
Oct 30 21:16:50.111493 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:50.111173    5713 cpu_manager.go:414] "containerMap: removing container" podUID="94effd3f-fc12-433c-aad6-82f48b4ec4fa" containerName="revoke-keys"
Oct 30 21:16:50.111493 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:50.111182    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94effd3f-fc12-433c-aad6-82f48b4ec4fa" containerName="revoke-keys"
Oct 30 21:16:50.111493 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:50.111258    5713 cpu_manager.go:414] "containerMap: removing container" podUID="e3000e9c-41b7-4667-b9e1-c90ecf6c9b63" containerName="istio-proxy"
Oct 30 21:16:50.111493 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:50.111267    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3000e9c-41b7-4667-b9e1-c90ecf6c9b63" containerName="istio-proxy"
Oct 30 21:16:50.111493 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:50.111286    5713 cpu_manager.go:414] "containerMap: removing container" podUID="a26a6cf1-a0b9-4470-93ea-557cae471765" containerName="istio-proxy"
Oct 30 21:16:50.111493 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:50.111294    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a26a6cf1-a0b9-4470-93ea-557cae471765" containerName="istio-proxy"
Oct 30 21:16:50.111493 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:50.111307    5713 cpu_manager.go:414] "containerMap: removing container" podUID="f2ecbf1c-d317-4e83-98ac-4aea45a351c3" containerName="revoke-keys"
Oct 30 21:16:50.111493 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:50.111315    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2ecbf1c-d317-4e83-98ac-4aea45a351c3" containerName="revoke-keys"
Oct 30 21:16:50.111493 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:50.111481    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="94effd3f-fc12-433c-aad6-82f48b4ec4fa" containerName="revoke-keys"
Oct 30 21:16:50.112029 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:50.111509    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3000e9c-41b7-4667-b9e1-c90ecf6c9b63" containerName="istio-proxy"
Oct 30 21:16:50.112029 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:50.111519    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="a26a6cf1-a0b9-4470-93ea-557cae471765" containerName="istio-proxy"
Oct 30 21:16:50.112029 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:50.111529    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2ecbf1c-d317-4e83-98ac-4aea45a351c3" containerName="revoke-keys"
Oct 30 21:16:50.121074 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:50.119123    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-vmz5r"
Oct 30 21:16:50.175968 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:50.175934    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-vmz5r"]
Oct 30 21:16:50.264538 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:50.264495    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94c10451-978e-44bb-8fa5-864dcd6f6d93-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-vmz5r\" (UID: \"94c10451-978e-44bb-8fa5-864dcd6f6d93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-vmz5r"
Oct 30 21:16:50.264765 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:50.264639    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94c10451-978e-44bb-8fa5-864dcd6f6d93-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-vmz5r\" (UID: \"94c10451-978e-44bb-8fa5-864dcd6f6d93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-vmz5r"
Oct 30 21:16:50.264765 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:50.264666    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94c10451-978e-44bb-8fa5-864dcd6f6d93-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-vmz5r\" (UID: \"94c10451-978e-44bb-8fa5-864dcd6f6d93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-vmz5r"
Oct 30 21:16:50.264968 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:50.264869    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7v5w\" (UniqueName: \"kubernetes.io/projected/94c10451-978e-44bb-8fa5-864dcd6f6d93-kube-api-access-n7v5w\") pod \"maas-default-gateway-openshift-default-769df7846c-vmz5r\" (UID: \"94c10451-978e-44bb-8fa5-864dcd6f6d93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-vmz5r"
Oct 30 21:16:50.264968 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:50.264926    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94c10451-978e-44bb-8fa5-864dcd6f6d93-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-vmz5r\" (UID: \"94c10451-978e-44bb-8fa5-864dcd6f6d93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-vmz5r"
Oct 30 21:16:50.265081 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:50.265014    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c10451-978e-44bb-8fa5-864dcd6f6d93-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-vmz5r\" (UID: \"94c10451-978e-44bb-8fa5-864dcd6f6d93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-vmz5r"
Oct 30 21:16:50.265136 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:50.265093    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c10451-978e-44bb-8fa5-864dcd6f6d93-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-vmz5r\" (UID: \"94c10451-978e-44bb-8fa5-864dcd6f6d93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-vmz5r"
Oct 30 21:16:50.265212 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:50.265195    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94c10451-978e-44bb-8fa5-864dcd6f6d93-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-vmz5r\" (UID: \"94c10451-978e-44bb-8fa5-864dcd6f6d93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-vmz5r"
Oct 30 21:16:50.265249 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:50.265227    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94c10451-978e-44bb-8fa5-864dcd6f6d93-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-vmz5r\" (UID: \"94c10451-978e-44bb-8fa5-864dcd6f6d93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-vmz5r"
Oct 30 21:16:50.366349 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:50.366246    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94c10451-978e-44bb-8fa5-864dcd6f6d93-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-vmz5r\" (UID: \"94c10451-978e-44bb-8fa5-864dcd6f6d93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-vmz5r"
Oct 30 21:16:50.366349 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:50.366290    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94c10451-978e-44bb-8fa5-864dcd6f6d93-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-vmz5r\" (UID: \"94c10451-978e-44bb-8fa5-864dcd6f6d93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-vmz5r"
Oct 30 21:16:50.366349 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:50.366349    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7v5w\" (UniqueName: \"kubernetes.io/projected/94c10451-978e-44bb-8fa5-864dcd6f6d93-kube-api-access-n7v5w\") pod \"maas-default-gateway-openshift-default-769df7846c-vmz5r\" (UID: \"94c10451-978e-44bb-8fa5-864dcd6f6d93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-vmz5r"
Oct 30 21:16:50.366675 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:50.366374    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94c10451-978e-44bb-8fa5-864dcd6f6d93-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-vmz5r\" (UID: \"94c10451-978e-44bb-8fa5-864dcd6f6d93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-vmz5r"
Oct 30 21:16:50.366675 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:50.366403    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c10451-978e-44bb-8fa5-864dcd6f6d93-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-vmz5r\" (UID: \"94c10451-978e-44bb-8fa5-864dcd6f6d93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-vmz5r"
Oct 30 21:16:50.366675 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:50.366441    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c10451-978e-44bb-8fa5-864dcd6f6d93-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-vmz5r\" (UID: \"94c10451-978e-44bb-8fa5-864dcd6f6d93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-vmz5r"
Oct 30 21:16:50.366675 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:50.366515    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94c10451-978e-44bb-8fa5-864dcd6f6d93-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-vmz5r\" (UID: \"94c10451-978e-44bb-8fa5-864dcd6f6d93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-vmz5r"
Oct 30 21:16:50.366675 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:50.366547    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94c10451-978e-44bb-8fa5-864dcd6f6d93-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-vmz5r\" (UID: \"94c10451-978e-44bb-8fa5-864dcd6f6d93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-vmz5r"
Oct 30 21:16:50.366675 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:50.366625    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94c10451-978e-44bb-8fa5-864dcd6f6d93-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-vmz5r\" (UID: \"94c10451-978e-44bb-8fa5-864dcd6f6d93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-vmz5r"
Oct 30 21:16:50.366978 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:50.366686    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94c10451-978e-44bb-8fa5-864dcd6f6d93-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-vmz5r\" (UID: \"94c10451-978e-44bb-8fa5-864dcd6f6d93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-vmz5r"
Oct 30 21:16:50.366978 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:50.366751    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94c10451-978e-44bb-8fa5-864dcd6f6d93-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-vmz5r\" (UID: \"94c10451-978e-44bb-8fa5-864dcd6f6d93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-vmz5r"
Oct 30 21:16:50.367067 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:50.367025    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c10451-978e-44bb-8fa5-864dcd6f6d93-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-vmz5r\" (UID: \"94c10451-978e-44bb-8fa5-864dcd6f6d93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-vmz5r"
Oct 30 21:16:50.367067 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:50.367057    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c10451-978e-44bb-8fa5-864dcd6f6d93-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-vmz5r\" (UID: \"94c10451-978e-44bb-8fa5-864dcd6f6d93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-vmz5r"
Oct 30 21:16:50.367345 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:50.367326    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94c10451-978e-44bb-8fa5-864dcd6f6d93-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-vmz5r\" (UID: \"94c10451-978e-44bb-8fa5-864dcd6f6d93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-vmz5r"
Oct 30 21:16:50.369045 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:50.369022    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94c10451-978e-44bb-8fa5-864dcd6f6d93-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-vmz5r\" (UID: \"94c10451-978e-44bb-8fa5-864dcd6f6d93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-vmz5r"
Oct 30 21:16:50.369597 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:50.369571    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94c10451-978e-44bb-8fa5-864dcd6f6d93-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-vmz5r\" (UID: \"94c10451-978e-44bb-8fa5-864dcd6f6d93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-vmz5r"
Oct 30 21:16:50.430723 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:50.430657    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7v5w\" (UniqueName: \"kubernetes.io/projected/94c10451-978e-44bb-8fa5-864dcd6f6d93-kube-api-access-n7v5w\") pod \"maas-default-gateway-openshift-default-769df7846c-vmz5r\" (UID: \"94c10451-978e-44bb-8fa5-864dcd6f6d93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-vmz5r"
Oct 30 21:16:50.432235 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:50.432207    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94c10451-978e-44bb-8fa5-864dcd6f6d93-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-vmz5r\" (UID: \"94c10451-978e-44bb-8fa5-864dcd6f6d93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-vmz5r"
Oct 30 21:16:50.445263 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:50.445228    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-vmz5r"
Oct 30 21:16:50.885612 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:50.885589    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-vmz5r"]
Oct 30 21:16:50.886592 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:16:50.886569    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94c10451_978e_44bb_8fa5_864dcd6f6d93.slice/crio-689b180edf50240abb26217d148050cc2053a7aa34d6b5c9e116bc5194013ff8 WatchSource:0}: Error finding container 689b180edf50240abb26217d148050cc2053a7aa34d6b5c9e116bc5194013ff8: Status 404 returned error can't find the container with id 689b180edf50240abb26217d148050cc2053a7aa34d6b5c9e116bc5194013ff8
Oct 30 21:16:50.889267 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:50.889236    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:16:50.889343 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:50.889299    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:16:50.889343 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:50.889329    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:16:51.730322 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:51.730278    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-vmz5r" event={"ID":"94c10451-978e-44bb-8fa5-864dcd6f6d93","Type":"ContainerStarted","Data":"44a3d1cade193a7daf80aed75493a7b2326af8ef29549b67503878efb8dd6cc3"}
Oct 30 21:16:51.730732 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:51.730364    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-vmz5r" event={"ID":"94c10451-978e-44bb-8fa5-864dcd6f6d93","Type":"ContainerStarted","Data":"689b180edf50240abb26217d148050cc2053a7aa34d6b5c9e116bc5194013ff8"}
Oct 30 21:16:51.959660 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:51.959592    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-vmz5r" podStartSLOduration=1.959573417 podStartE2EDuration="1.959573417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:51.897495751 +0000 UTC m=+2294.962272144" watchObservedRunningTime="2026-10-30 21:16:51.959573417 +0000 UTC m=+2295.024349815"
Oct 30 21:16:52.454816 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:52.448095    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-vmz5r"
Oct 30 21:16:52.454816 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:52.454273    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-vmz5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body=
Oct 30 21:16:52.454816 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:52.454355    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-vmz5r" podUID="94c10451-978e-44bb-8fa5-864dcd6f6d93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused"
Oct 30 21:16:53.458815 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:53.452227    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-vmz5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body=
Oct 30 21:16:53.458815 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:53.452290    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-vmz5r" podUID="94c10451-978e-44bb-8fa5-864dcd6f6d93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused"
Oct 30 21:16:54.449666 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:54.449197    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-vmz5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body=
Oct 30 21:16:54.449666 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:54.449260    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-vmz5r" podUID="94c10451-978e-44bb-8fa5-864dcd6f6d93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused"
Oct 30 21:16:54.480169 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:16:54.480136    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:16:55.450866 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:55.450826    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-vmz5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body=
Oct 30 21:16:55.451218 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:55.451187    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-vmz5r" podUID="94c10451-978e-44bb-8fa5-864dcd6f6d93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused"
Oct 30 21:16:56.446260 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:56.446218    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-vmz5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body=
Oct 30 21:16:56.447015 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:56.446978    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-vmz5r" podUID="94c10451-978e-44bb-8fa5-864dcd6f6d93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused"
Oct 30 21:16:57.445703 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:57.445668    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-vmz5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body=
Oct 30 21:16:57.445903 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:57.445719    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-vmz5r" podUID="94c10451-978e-44bb-8fa5-864dcd6f6d93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused"
Oct 30 21:16:57.753093 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:57.752995    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-2b425d/distinct-2-w5-2b425d-kserve-6694867c68-bzpvn" podUID="b9d391e6-de6f-476d-a926-d1b333474109" containerName="main" containerID="cri-o://9c10bf76a8d288d886bd48cea2c7614218fd4d9ea813f155b3155eab278a110e" gracePeriod=45
Oct 30 21:16:57.774990 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:57.774950    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-2b425d/premium-w5-2b425d-kserve-78756b6c5b-9x5xb" podUID="abf7b512-50e5-4400-9ca3-9ffccaead3d7" containerName="main" containerID="cri-o://a111f87edb57dccdaa4d3907e474c9625246c8f320d9a38a4446e6807dd1f967" gracePeriod=45
Oct 30 21:16:57.782872 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:57.782836    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-2b425d/simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk" podUID="e8108824-03ff-4c1d-9f5b-07be3a3a92d6" containerName="main" containerID="cri-o://559e5d07ad88e9d0f729142d98d4be7401236036557c9dfec80618fffd659db5" gracePeriod=45
Oct 30 21:16:58.019828 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.018405    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b425d/simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk" podUID="e8108824-03ff-4c1d-9f5b-07be3a3a92d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": dial tcp 10.134.0.50:8000: connect: connection refused"
Oct 30 21:16:58.117188 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.117153    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2b425d/distinct-2-w5-2b425d-kserve-6694867c68-bzpvn"
Oct 30 21:16:58.120966 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.120939    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2b425d/premium-w5-2b425d-kserve-78756b6c5b-9x5xb"
Oct 30 21:16:58.125311 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.125287    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2b425d/simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk"
Oct 30 21:16:58.257075 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.257043    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9d391e6-de6f-476d-a926-d1b333474109-kube-api-access-bpcth\" (UniqueName: \"kubernetes.io/projected/b9d391e6-de6f-476d-a926-d1b333474109-kube-api-access-bpcth\") pod \"b9d391e6-de6f-476d-a926-d1b333474109\" (UID: \"b9d391e6-de6f-476d-a926-d1b333474109\") "
Oct 30 21:16:58.257368 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.257092    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9d391e6-de6f-476d-a926-d1b333474109-tls-certs\" (UniqueName: \"kubernetes.io/secret/b9d391e6-de6f-476d-a926-d1b333474109-tls-certs\") pod \"b9d391e6-de6f-476d-a926-d1b333474109\" (UID: \"b9d391e6-de6f-476d-a926-d1b333474109\") "
Oct 30 21:16:58.259453 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.259426    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abf7b512-50e5-4400-9ca3-9ffccaead3d7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abf7b512-50e5-4400-9ca3-9ffccaead3d7-tmp-dir\") pod \"abf7b512-50e5-4400-9ca3-9ffccaead3d7\" (UID: \"abf7b512-50e5-4400-9ca3-9ffccaead3d7\") "
Oct 30 21:16:58.260202 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.260182    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8108824-03ff-4c1d-9f5b-07be3a3a92d6-tls-certs\" (UniqueName: \"kubernetes.io/secret/e8108824-03ff-4c1d-9f5b-07be3a3a92d6-tls-certs\") pod \"e8108824-03ff-4c1d-9f5b-07be3a3a92d6\" (UID: \"e8108824-03ff-4c1d-9f5b-07be3a3a92d6\") "
Oct 30 21:16:58.260321 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.260218    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8108824-03ff-4c1d-9f5b-07be3a3a92d6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8108824-03ff-4c1d-9f5b-07be3a3a92d6-tmp-dir\") pod \"e8108824-03ff-4c1d-9f5b-07be3a3a92d6\" (UID: \"e8108824-03ff-4c1d-9f5b-07be3a3a92d6\") "
Oct 30 21:16:58.260321 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.260237    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abf7b512-50e5-4400-9ca3-9ffccaead3d7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/abf7b512-50e5-4400-9ca3-9ffccaead3d7-dshm\") pod \"abf7b512-50e5-4400-9ca3-9ffccaead3d7\" (UID: \"abf7b512-50e5-4400-9ca3-9ffccaead3d7\") "
Oct 30 21:16:58.260321 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.260258    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9d391e6-de6f-476d-a926-d1b333474109-home\" (UniqueName: \"kubernetes.io/empty-dir/b9d391e6-de6f-476d-a926-d1b333474109-home\") pod \"b9d391e6-de6f-476d-a926-d1b333474109\" (UID: \"b9d391e6-de6f-476d-a926-d1b333474109\") "
Oct 30 21:16:58.260321 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.260316    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9d391e6-de6f-476d-a926-d1b333474109-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9d391e6-de6f-476d-a926-d1b333474109-tmp-dir\") pod \"b9d391e6-de6f-476d-a926-d1b333474109\" (UID: \"b9d391e6-de6f-476d-a926-d1b333474109\") "
Oct 30 21:16:58.260563 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.260354    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9d391e6-de6f-476d-a926-d1b333474109-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9d391e6-de6f-476d-a926-d1b333474109-model-cache\") pod \"b9d391e6-de6f-476d-a926-d1b333474109\" (UID: \"b9d391e6-de6f-476d-a926-d1b333474109\") "
Oct 30 21:16:58.260563 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.260389    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abf7b512-50e5-4400-9ca3-9ffccaead3d7-tls-certs\" (UniqueName: \"kubernetes.io/secret/abf7b512-50e5-4400-9ca3-9ffccaead3d7-tls-certs\") pod \"abf7b512-50e5-4400-9ca3-9ffccaead3d7\" (UID: \"abf7b512-50e5-4400-9ca3-9ffccaead3d7\") "
Oct 30 21:16:58.260563 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.260415    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abf7b512-50e5-4400-9ca3-9ffccaead3d7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abf7b512-50e5-4400-9ca3-9ffccaead3d7-model-cache\") pod \"abf7b512-50e5-4400-9ca3-9ffccaead3d7\" (UID: \"abf7b512-50e5-4400-9ca3-9ffccaead3d7\") "
Oct 30 21:16:58.260563 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.260460    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8108824-03ff-4c1d-9f5b-07be3a3a92d6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8108824-03ff-4c1d-9f5b-07be3a3a92d6-model-cache\") pod \"e8108824-03ff-4c1d-9f5b-07be3a3a92d6\" (UID: \"e8108824-03ff-4c1d-9f5b-07be3a3a92d6\") "
Oct 30 21:16:58.260563 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.260553    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abf7b512-50e5-4400-9ca3-9ffccaead3d7-kube-api-access-8nh49\" (UniqueName: \"kubernetes.io/projected/abf7b512-50e5-4400-9ca3-9ffccaead3d7-kube-api-access-8nh49\") pod \"abf7b512-50e5-4400-9ca3-9ffccaead3d7\" (UID: \"abf7b512-50e5-4400-9ca3-9ffccaead3d7\") "
Oct 30 21:16:58.261076 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.260598    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9d391e6-de6f-476d-a926-d1b333474109-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9d391e6-de6f-476d-a926-d1b333474109-dshm\") pod \"b9d391e6-de6f-476d-a926-d1b333474109\" (UID: \"b9d391e6-de6f-476d-a926-d1b333474109\") "
Oct 30 21:16:58.261076 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.260626    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8108824-03ff-4c1d-9f5b-07be3a3a92d6-kube-api-access-kkzkj\" (UniqueName: \"kubernetes.io/projected/e8108824-03ff-4c1d-9f5b-07be3a3a92d6-kube-api-access-kkzkj\") pod \"e8108824-03ff-4c1d-9f5b-07be3a3a92d6\" (UID: \"e8108824-03ff-4c1d-9f5b-07be3a3a92d6\") "
Oct 30 21:16:58.261076 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.260644    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8108824-03ff-4c1d-9f5b-07be3a3a92d6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8108824-03ff-4c1d-9f5b-07be3a3a92d6-dshm\") pod \"e8108824-03ff-4c1d-9f5b-07be3a3a92d6\" (UID: \"e8108824-03ff-4c1d-9f5b-07be3a3a92d6\") "
Oct 30 21:16:58.261076 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.260669    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abf7b512-50e5-4400-9ca3-9ffccaead3d7-home\" (UniqueName: \"kubernetes.io/empty-dir/abf7b512-50e5-4400-9ca3-9ffccaead3d7-home\") pod \"abf7b512-50e5-4400-9ca3-9ffccaead3d7\" (UID: \"abf7b512-50e5-4400-9ca3-9ffccaead3d7\") "
Oct 30 21:16:58.261076 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.260750    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8108824-03ff-4c1d-9f5b-07be3a3a92d6-home\" (UniqueName: \"kubernetes.io/empty-dir/e8108824-03ff-4c1d-9f5b-07be3a3a92d6-home\") pod \"e8108824-03ff-4c1d-9f5b-07be3a3a92d6\" (UID: \"e8108824-03ff-4c1d-9f5b-07be3a3a92d6\") "
Oct 30 21:16:58.261414 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.259683    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abf7b512-50e5-4400-9ca3-9ffccaead3d7-tmp-dir" pod "abf7b512-50e5-4400-9ca3-9ffccaead3d7" (UID: "abf7b512-50e5-4400-9ca3-9ffccaead3d7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:16:58.261547 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.259938    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9d391e6-de6f-476d-a926-d1b333474109-tls-certs" pod "b9d391e6-de6f-476d-a926-d1b333474109" (UID: "b9d391e6-de6f-476d-a926-d1b333474109"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:16:58.261547 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.260087    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9d391e6-de6f-476d-a926-d1b333474109-kube-api-access-bpcth" pod "b9d391e6-de6f-476d-a926-d1b333474109" (UID: "b9d391e6-de6f-476d-a926-d1b333474109"). InnerVolumeSpecName "kube-api-access-bpcth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:16:58.262295 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.261833    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8108824-03ff-4c1d-9f5b-07be3a3a92d6-model-cache" pod "e8108824-03ff-4c1d-9f5b-07be3a3a92d6" (UID: "e8108824-03ff-4c1d-9f5b-07be3a3a92d6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:16:58.262295 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.262101    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9d391e6-de6f-476d-a926-d1b333474109-home" pod "b9d391e6-de6f-476d-a926-d1b333474109" (UID: "b9d391e6-de6f-476d-a926-d1b333474109"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:16:58.262295 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.262285    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8108824-03ff-4c1d-9f5b-07be3a3a92d6-tmp-dir" pod "e8108824-03ff-4c1d-9f5b-07be3a3a92d6" (UID: "e8108824-03ff-4c1d-9f5b-07be3a3a92d6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:16:58.262554 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.262525    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abf7b512-50e5-4400-9ca3-9ffccaead3d7-home" pod "abf7b512-50e5-4400-9ca3-9ffccaead3d7" (UID: "abf7b512-50e5-4400-9ca3-9ffccaead3d7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:16:58.262759 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.262728    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9d391e6-de6f-476d-a926-d1b333474109-model-cache" pod "b9d391e6-de6f-476d-a926-d1b333474109" (UID: "b9d391e6-de6f-476d-a926-d1b333474109"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:16:58.262989 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.262943    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9d391e6-de6f-476d-a926-d1b333474109-tmp-dir" pod "b9d391e6-de6f-476d-a926-d1b333474109" (UID: "b9d391e6-de6f-476d-a926-d1b333474109"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:16:58.262989 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.262974    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abf7b512-50e5-4400-9ca3-9ffccaead3d7-model-cache" pod "abf7b512-50e5-4400-9ca3-9ffccaead3d7" (UID: "abf7b512-50e5-4400-9ca3-9ffccaead3d7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:16:58.263192 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.263135    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8108824-03ff-4c1d-9f5b-07be3a3a92d6-tls-certs" pod "e8108824-03ff-4c1d-9f5b-07be3a3a92d6" (UID: "e8108824-03ff-4c1d-9f5b-07be3a3a92d6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:16:58.263264 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.263244    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8108824-03ff-4c1d-9f5b-07be3a3a92d6-home" pod "e8108824-03ff-4c1d-9f5b-07be3a3a92d6" (UID: "e8108824-03ff-4c1d-9f5b-07be3a3a92d6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:16:58.266556 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.266529    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8108824-03ff-4c1d-9f5b-07be3a3a92d6-kube-api-access-kkzkj" pod "e8108824-03ff-4c1d-9f5b-07be3a3a92d6" (UID: "e8108824-03ff-4c1d-9f5b-07be3a3a92d6"). InnerVolumeSpecName "kube-api-access-kkzkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:16:58.267570 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.267532    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abf7b512-50e5-4400-9ca3-9ffccaead3d7-tls-certs" pod "abf7b512-50e5-4400-9ca3-9ffccaead3d7" (UID: "abf7b512-50e5-4400-9ca3-9ffccaead3d7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:16:58.268133 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.268024    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9d391e6-de6f-476d-a926-d1b333474109-dshm" pod "b9d391e6-de6f-476d-a926-d1b333474109" (UID: "b9d391e6-de6f-476d-a926-d1b333474109"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:16:58.268133 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.268106    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8108824-03ff-4c1d-9f5b-07be3a3a92d6-dshm" pod "e8108824-03ff-4c1d-9f5b-07be3a3a92d6" (UID: "e8108824-03ff-4c1d-9f5b-07be3a3a92d6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:16:58.268481 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.268454    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abf7b512-50e5-4400-9ca3-9ffccaead3d7-kube-api-access-8nh49" pod "abf7b512-50e5-4400-9ca3-9ffccaead3d7" (UID: "abf7b512-50e5-4400-9ca3-9ffccaead3d7"). InnerVolumeSpecName "kube-api-access-8nh49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:16:58.270330 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.270268    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abf7b512-50e5-4400-9ca3-9ffccaead3d7-dshm" pod "abf7b512-50e5-4400-9ca3-9ffccaead3d7" (UID: "abf7b512-50e5-4400-9ca3-9ffccaead3d7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:16:58.363485 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.363446    5713 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8108824-03ff-4c1d-9f5b-07be3a3a92d6-home\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:16:58.363485 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.363490    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpcth\" (UniqueName: \"kubernetes.io/projected/b9d391e6-de6f-476d-a926-d1b333474109-kube-api-access-bpcth\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:16:58.363759 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.363508    5713 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9d391e6-de6f-476d-a926-d1b333474109-tls-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:16:58.363759 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.363526    5713 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abf7b512-50e5-4400-9ca3-9ffccaead3d7-tmp-dir\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:16:58.363759 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.363541    5713 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8108824-03ff-4c1d-9f5b-07be3a3a92d6-tls-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:16:58.363759 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.363555    5713 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8108824-03ff-4c1d-9f5b-07be3a3a92d6-tmp-dir\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:16:58.363759 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.363569    5713 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abf7b512-50e5-4400-9ca3-9ffccaead3d7-dshm\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:16:58.363759 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.363582    5713 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9d391e6-de6f-476d-a926-d1b333474109-home\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:16:58.363759 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.363595    5713 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9d391e6-de6f-476d-a926-d1b333474109-tmp-dir\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:16:58.363759 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.363610    5713 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9d391e6-de6f-476d-a926-d1b333474109-model-cache\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:16:58.363759 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.363624    5713 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abf7b512-50e5-4400-9ca3-9ffccaead3d7-tls-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:16:58.363759 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.363640    5713 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abf7b512-50e5-4400-9ca3-9ffccaead3d7-model-cache\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:16:58.363759 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.363655    5713 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8108824-03ff-4c1d-9f5b-07be3a3a92d6-model-cache\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:16:58.363759 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.363669    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nh49\" (UniqueName: \"kubernetes.io/projected/abf7b512-50e5-4400-9ca3-9ffccaead3d7-kube-api-access-8nh49\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:16:58.363759 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.363682    5713 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9d391e6-de6f-476d-a926-d1b333474109-dshm\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:16:58.363759 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.363696    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkzkj\" (UniqueName: \"kubernetes.io/projected/e8108824-03ff-4c1d-9f5b-07be3a3a92d6-kube-api-access-kkzkj\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:16:58.363759 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.363708    5713 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8108824-03ff-4c1d-9f5b-07be3a3a92d6-dshm\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:16:58.363759 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.363723    5713 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abf7b512-50e5-4400-9ca3-9ffccaead3d7-home\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:16:58.446660 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.446621    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-vmz5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body=
Oct 30 21:16:58.446854 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.446683    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-vmz5r" podUID="94c10451-978e-44bb-8fa5-864dcd6f6d93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused"
Oct 30 21:16:58.869639 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.869591    5713 generic.go:356] "Generic (PLEG): container finished" podID="e8108824-03ff-4c1d-9f5b-07be3a3a92d6" containerID="559e5d07ad88e9d0f729142d98d4be7401236036557c9dfec80618fffd659db5" exitCode=0
Oct 30 21:16:58.870188 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.869732    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2b425d/simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk" event={"ID":"e8108824-03ff-4c1d-9f5b-07be3a3a92d6","Type":"ContainerDied","Data":"559e5d07ad88e9d0f729142d98d4be7401236036557c9dfec80618fffd659db5"}
Oct 30 21:16:58.870188 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.869772    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2b425d/simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk" event={"ID":"e8108824-03ff-4c1d-9f5b-07be3a3a92d6","Type":"ContainerDied","Data":"0dbe75693a989a7d5180176d8a3d9742cd0328525eb45943ed1c8366782cc0b5"}
Oct 30 21:16:58.870188 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.869841    5713 scope.go:122] "RemoveContainer" containerID="559e5d07ad88e9d0f729142d98d4be7401236036557c9dfec80618fffd659db5"
Oct 30 21:16:58.870188 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.870031    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2b425d/simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk"
Oct 30 21:16:58.878484 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.878450    5713 generic.go:356] "Generic (PLEG): container finished" podID="b9d391e6-de6f-476d-a926-d1b333474109" containerID="9c10bf76a8d288d886bd48cea2c7614218fd4d9ea813f155b3155eab278a110e" exitCode=0
Oct 30 21:16:58.878635 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.878568    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2b425d/distinct-2-w5-2b425d-kserve-6694867c68-bzpvn"
Oct 30 21:16:58.878713 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.878677    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2b425d/distinct-2-w5-2b425d-kserve-6694867c68-bzpvn" event={"ID":"b9d391e6-de6f-476d-a926-d1b333474109","Type":"ContainerDied","Data":"9c10bf76a8d288d886bd48cea2c7614218fd4d9ea813f155b3155eab278a110e"}
Oct 30 21:16:58.878781 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.878714    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2b425d/distinct-2-w5-2b425d-kserve-6694867c68-bzpvn" event={"ID":"b9d391e6-de6f-476d-a926-d1b333474109","Type":"ContainerDied","Data":"835bc0c9e5f8445d3fa402e10fd8cdf342781b161f039a6caa147a5dfc542781"}
Oct 30 21:16:58.881995 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.881974    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2b425d/premium-w5-2b425d-kserve-78756b6c5b-9x5xb"
Oct 30 21:16:58.882093 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.882024    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2b425d/premium-w5-2b425d-kserve-78756b6c5b-9x5xb" event={"ID":"abf7b512-50e5-4400-9ca3-9ffccaead3d7","Type":"ContainerDied","Data":"a111f87edb57dccdaa4d3907e474c9625246c8f320d9a38a4446e6807dd1f967"}
Oct 30 21:16:58.882093 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.881962    5713 generic.go:356] "Generic (PLEG): container finished" podID="abf7b512-50e5-4400-9ca3-9ffccaead3d7" containerID="a111f87edb57dccdaa4d3907e474c9625246c8f320d9a38a4446e6807dd1f967" exitCode=0
Oct 30 21:16:58.882226 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.882150    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2b425d/premium-w5-2b425d-kserve-78756b6c5b-9x5xb" event={"ID":"abf7b512-50e5-4400-9ca3-9ffccaead3d7","Type":"ContainerDied","Data":"1571665678e05b987149fa7dd3b613cb6da99b457ea6c348d1fc6426f9ecc778"}
Oct 30 21:16:58.893111 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.893091    5713 scope.go:122] "RemoveContainer" containerID="559e5d07ad88e9d0f729142d98d4be7401236036557c9dfec80618fffd659db5"
Oct 30 21:16:58.893477 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:16:58.893448    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"559e5d07ad88e9d0f729142d98d4be7401236036557c9dfec80618fffd659db5\": container with ID starting with 559e5d07ad88e9d0f729142d98d4be7401236036557c9dfec80618fffd659db5 not found: ID does not exist" containerID="559e5d07ad88e9d0f729142d98d4be7401236036557c9dfec80618fffd659db5"
Oct 30 21:16:58.893555 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.893488    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"559e5d07ad88e9d0f729142d98d4be7401236036557c9dfec80618fffd659db5"} err="failed to get container status \"559e5d07ad88e9d0f729142d98d4be7401236036557c9dfec80618fffd659db5\": rpc error: code = NotFound desc = could not find container \"559e5d07ad88e9d0f729142d98d4be7401236036557c9dfec80618fffd659db5\": container with ID starting with 559e5d07ad88e9d0f729142d98d4be7401236036557c9dfec80618fffd659db5 not found: ID does not exist"
Oct 30 21:16:58.893555 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.893511    5713 scope.go:122] "RemoveContainer" containerID="9c10bf76a8d288d886bd48cea2c7614218fd4d9ea813f155b3155eab278a110e"
Oct 30 21:16:58.904000 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.903983    5713 scope.go:122] "RemoveContainer" containerID="9c10bf76a8d288d886bd48cea2c7614218fd4d9ea813f155b3155eab278a110e"
Oct 30 21:16:58.904300 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:16:58.904271    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c10bf76a8d288d886bd48cea2c7614218fd4d9ea813f155b3155eab278a110e\": container with ID starting with 9c10bf76a8d288d886bd48cea2c7614218fd4d9ea813f155b3155eab278a110e not found: ID does not exist" containerID="9c10bf76a8d288d886bd48cea2c7614218fd4d9ea813f155b3155eab278a110e"
Oct 30 21:16:58.904300 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.904296    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c10bf76a8d288d886bd48cea2c7614218fd4d9ea813f155b3155eab278a110e"} err="failed to get container status \"9c10bf76a8d288d886bd48cea2c7614218fd4d9ea813f155b3155eab278a110e\": rpc error: code = NotFound desc = could not find container \"9c10bf76a8d288d886bd48cea2c7614218fd4d9ea813f155b3155eab278a110e\": container with ID starting with 9c10bf76a8d288d886bd48cea2c7614218fd4d9ea813f155b3155eab278a110e not found: ID does not exist"
Oct 30 21:16:58.904468 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.904313    5713 scope.go:122] "RemoveContainer" containerID="a111f87edb57dccdaa4d3907e474c9625246c8f320d9a38a4446e6807dd1f967"
Oct 30 21:16:58.916554 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.916535    5713 scope.go:122] "RemoveContainer" containerID="a111f87edb57dccdaa4d3907e474c9625246c8f320d9a38a4446e6807dd1f967"
Oct 30 21:16:58.916858 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:16:58.916839    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a111f87edb57dccdaa4d3907e474c9625246c8f320d9a38a4446e6807dd1f967\": container with ID starting with a111f87edb57dccdaa4d3907e474c9625246c8f320d9a38a4446e6807dd1f967 not found: ID does not exist" containerID="a111f87edb57dccdaa4d3907e474c9625246c8f320d9a38a4446e6807dd1f967"
Oct 30 21:16:58.916962 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.916866    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a111f87edb57dccdaa4d3907e474c9625246c8f320d9a38a4446e6807dd1f967"} err="failed to get container status \"a111f87edb57dccdaa4d3907e474c9625246c8f320d9a38a4446e6807dd1f967\": rpc error: code = NotFound desc = could not find container \"a111f87edb57dccdaa4d3907e474c9625246c8f320d9a38a4446e6807dd1f967\": container with ID starting with a111f87edb57dccdaa4d3907e474c9625246c8f320d9a38a4446e6807dd1f967 not found: ID does not exist"
Oct 30 21:16:58.935711 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.935681    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-639b07-6dd84859d7-k49qq"]
Oct 30 21:16:58.936660 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.936639    5713 cpu_manager.go:414] "containerMap: removing container" podUID="e8108824-03ff-4c1d-9f5b-07be3a3a92d6" containerName="main"
Oct 30 21:16:58.936731 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.936666    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8108824-03ff-4c1d-9f5b-07be3a3a92d6" containerName="main"
Oct 30 21:16:58.936805 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.936729    5713 cpu_manager.go:414] "containerMap: removing container" podUID="abf7b512-50e5-4400-9ca3-9ffccaead3d7" containerName="main"
Oct 30 21:16:58.936805 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.936739    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abf7b512-50e5-4400-9ca3-9ffccaead3d7" containerName="main"
Oct 30 21:16:58.936805 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.936752    5713 cpu_manager.go:414] "containerMap: removing container" podUID="b9d391e6-de6f-476d-a926-d1b333474109" containerName="main"
Oct 30 21:16:58.936805 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.936760    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9d391e6-de6f-476d-a926-d1b333474109" containerName="main"
Oct 30 21:16:58.937017 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.936981    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="abf7b512-50e5-4400-9ca3-9ffccaead3d7" containerName="main"
Oct 30 21:16:58.937017 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.937000    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9d391e6-de6f-476d-a926-d1b333474109" containerName="main"
Oct 30 21:16:58.937116 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.937020    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8108824-03ff-4c1d-9f5b-07be3a3a92d6" containerName="main"
Oct 30 21:16:58.937116 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.937034    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3000e9c-41b7-4667-b9e1-c90ecf6c9b63" containerName="istio-proxy"
Oct 30 21:16:58.942764 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.942742    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-639b07-6dd84859d7-k49qq"
Oct 30 21:16:58.981246 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.981209    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f469bcf6-121f-4e38-a9a1-e0f4237b9242-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-639b07-6dd84859d7-k49qq\" (UID: \"f469bcf6-121f-4e38-a9a1-e0f4237b9242\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-639b07-6dd84859d7-k49qq"
Oct 30 21:16:58.981415 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.981390    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68jhz\" (UniqueName: \"kubernetes.io/projected/f469bcf6-121f-4e38-a9a1-e0f4237b9242-kube-api-access-68jhz\") pod \"payload-pre-processing-e2e-inf-b-639b07-6dd84859d7-k49qq\" (UID: \"f469bcf6-121f-4e38-a9a1-e0f4237b9242\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-639b07-6dd84859d7-k49qq"
Oct 30 21:16:59.003816 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.997143    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-639b07-6dd84859d7-k49qq"]
Oct 30 21:16:59.003816 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.997278    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-639b07\""
Oct 30 21:16:59.003816 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:58.997405    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-639b07-dockercfg-rfzg7\""
Oct 30 21:16:59.054872 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:59.047475    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b425d/premium-w5-2b425d-kserve-78756b6c5b-9x5xb" podUID="abf7b512-50e5-4400-9ca3-9ffccaead3d7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:59.110349 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:59.093457    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68jhz\" (UniqueName: \"kubernetes.io/projected/f469bcf6-121f-4e38-a9a1-e0f4237b9242-kube-api-access-68jhz\") pod \"payload-pre-processing-e2e-inf-b-639b07-6dd84859d7-k49qq\" (UID: \"f469bcf6-121f-4e38-a9a1-e0f4237b9242\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-639b07-6dd84859d7-k49qq"
Oct 30 21:16:59.110349 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:59.093660    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f469bcf6-121f-4e38-a9a1-e0f4237b9242-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-639b07-6dd84859d7-k49qq\" (UID: \"f469bcf6-121f-4e38-a9a1-e0f4237b9242\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-639b07-6dd84859d7-k49qq"
Oct 30 21:16:59.110349 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:59.094542    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f469bcf6-121f-4e38-a9a1-e0f4237b9242-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-639b07-6dd84859d7-k49qq\" (UID: \"f469bcf6-121f-4e38-a9a1-e0f4237b9242\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-639b07-6dd84859d7-k49qq"
Oct 30 21:16:59.120859 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:59.120825    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-2b425d/simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk"]
Oct 30 21:16:59.142591 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:59.142544    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-2b425d/simulator-w5-2b425d-kserve-7bdd96cddf-mtqxk"]
Oct 30 21:16:59.159687 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:59.155874    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-639b07-59547c9664-bqxz8"]
Oct 30 21:16:59.161483 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:59.161186    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-639b07-59547c9664-bqxz8"]
Oct 30 21:16:59.161483 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:59.161303    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-639b07-59547c9664-bqxz8"
Oct 30 21:16:59.176475 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:59.176244    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68jhz\" (UniqueName: \"kubernetes.io/projected/f469bcf6-121f-4e38-a9a1-e0f4237b9242-kube-api-access-68jhz\") pod \"payload-pre-processing-e2e-inf-b-639b07-6dd84859d7-k49qq\" (UID: \"f469bcf6-121f-4e38-a9a1-e0f4237b9242\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-639b07-6dd84859d7-k49qq"
Oct 30 21:16:59.179755 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:59.179714    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-2b425d/distinct-2-w5-2b425d-kserve-6694867c68-bzpvn"]
Oct 30 21:16:59.191290 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:59.191256    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-2b425d/distinct-2-w5-2b425d-kserve-6694867c68-bzpvn"]
Oct 30 21:16:59.208152 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:59.208115    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx2qc\" (UniqueName: \"kubernetes.io/projected/31d0639c-3365-4848-a878-3388c779886c-kube-api-access-bx2qc\") pod \"payload-processing-e2e-inf-b-639b07-59547c9664-bqxz8\" (UID: \"31d0639c-3365-4848-a878-3388c779886c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-639b07-59547c9664-bqxz8"
Oct 30 21:16:59.208309 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:59.208178    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31d0639c-3365-4848-a878-3388c779886c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-639b07-59547c9664-bqxz8\" (UID: \"31d0639c-3365-4848-a878-3388c779886c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-639b07-59547c9664-bqxz8"
Oct 30 21:16:59.258033 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:59.257995    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-639b07-6dd84859d7-k49qq"
Oct 30 21:16:59.290262 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:59.290232    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-2b425d/premium-w5-2b425d-kserve-78756b6c5b-9x5xb"]
Oct 30 21:16:59.295821 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:59.295772    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-2b425d/premium-w5-2b425d-kserve-78756b6c5b-9x5xb"]
Oct 30 21:16:59.310420 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:59.310338    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx2qc\" (UniqueName: \"kubernetes.io/projected/31d0639c-3365-4848-a878-3388c779886c-kube-api-access-bx2qc\") pod \"payload-processing-e2e-inf-b-639b07-59547c9664-bqxz8\" (UID: \"31d0639c-3365-4848-a878-3388c779886c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-639b07-59547c9664-bqxz8"
Oct 30 21:16:59.310420 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:59.310397    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31d0639c-3365-4848-a878-3388c779886c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-639b07-59547c9664-bqxz8\" (UID: \"31d0639c-3365-4848-a878-3388c779886c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-639b07-59547c9664-bqxz8"
Oct 30 21:16:59.311399 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:59.311374    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31d0639c-3365-4848-a878-3388c779886c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-639b07-59547c9664-bqxz8\" (UID: \"31d0639c-3365-4848-a878-3388c779886c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-639b07-59547c9664-bqxz8"
Oct 30 21:16:59.368549 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:59.368482    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx2qc\" (UniqueName: \"kubernetes.io/projected/31d0639c-3365-4848-a878-3388c779886c-kube-api-access-bx2qc\") pod \"payload-processing-e2e-inf-b-639b07-59547c9664-bqxz8\" (UID: \"31d0639c-3365-4848-a878-3388c779886c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-639b07-59547c9664-bqxz8"
Oct 30 21:16:59.511965 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:59.511875    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-639b07-59547c9664-bqxz8"
Oct 30 21:16:59.653739 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:59.652215    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-639b07-6dd84859d7-k49qq"]
Oct 30 21:16:59.766248 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:59.766217    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-vmz5r"
Oct 30 21:16:59.767444 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:59.767403    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-vmz5r"
Oct 30 21:16:59.767989 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:59.767972    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-vmz5r"
Oct 30 21:16:59.892107 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:59.892001    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-639b07-6dd84859d7-k49qq" event={"ID":"f469bcf6-121f-4e38-a9a1-e0f4237b9242","Type":"ContainerStarted","Data":"7197b609e3127f2f63972f1822cd3c26aa25a79affdd079bd7ed7db11128c1a8"}
Oct 30 21:16:59.892107 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:59.892061    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-639b07-6dd84859d7-k49qq" event={"ID":"f469bcf6-121f-4e38-a9a1-e0f4237b9242","Type":"ContainerStarted","Data":"caf4d205ace57903a1bea9b991c57032d72abb57c97423a275287aafe5adbc9b"}
Oct 30 21:16:59.892590 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:59.892126    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-639b07-6dd84859d7-k49qq"
Oct 30 21:16:59.905050 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:16:59.905021    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-639b07-59547c9664-bqxz8"]
Oct 30 21:16:59.906367 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:16:59.906343    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31d0639c_3365_4848_a878_3388c779886c.slice/crio-8c1b53c84601cc4682c81f749258771f3c6d31e2e56adfd2a9b445299fd94534 WatchSource:0}: Error finding container 8c1b53c84601cc4682c81f749258771f3c6d31e2e56adfd2a9b445299fd94534: Status 404 returned error can't find the container with id 8c1b53c84601cc4682c81f749258771f3c6d31e2e56adfd2a9b445299fd94534
Oct 30 21:17:00.126453 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:00.126405    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-639b07-6dd84859d7-k49qq" podStartSLOduration=2.126391082 podStartE2EDuration="2.126391082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:00.122442102 +0000 UTC m=+2303.187218513" watchObservedRunningTime="2026-10-30 21:17:00.126391082 +0000 UTC m=+2303.191167481"
Oct 30 21:17:00.905298 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:00.905247    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-639b07-59547c9664-bqxz8" event={"ID":"31d0639c-3365-4848-a878-3388c779886c","Type":"ContainerStarted","Data":"e7c196fa1389ed446d6130ebc54a129530a0401aa6cf39f17acec6e539028598"}
Oct 30 21:17:00.905765 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:00.905308    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-639b07-59547c9664-bqxz8" event={"ID":"31d0639c-3365-4848-a878-3388c779886c","Type":"ContainerStarted","Data":"8c1b53c84601cc4682c81f749258771f3c6d31e2e56adfd2a9b445299fd94534"}
Oct 30 21:17:00.905765 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:00.905451    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-639b07-59547c9664-bqxz8"
Oct 30 21:17:01.018429 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:01.018373    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-639b07-59547c9664-bqxz8" podStartSLOduration=2.018356228 podStartE2EDuration="2.018356228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:01.017398377 +0000 UTC m=+2304.082174776" watchObservedRunningTime="2026-10-30 21:17:01.018356228 +0000 UTC m=+2304.083132627"
Oct 30 21:17:01.064027 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:01.063993    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abf7b512-50e5-4400-9ca3-9ffccaead3d7" path="/var/lib/kubelet/pods/abf7b512-50e5-4400-9ca3-9ffccaead3d7/volumes"
Oct 30 21:17:01.064469 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:01.064454    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9d391e6-de6f-476d-a926-d1b333474109" path="/var/lib/kubelet/pods/b9d391e6-de6f-476d-a926-d1b333474109/volumes"
Oct 30 21:17:01.064890 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:01.064875    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8108824-03ff-4c1d-9f5b-07be3a3a92d6" path="/var/lib/kubelet/pods/e8108824-03ff-4c1d-9f5b-07be3a3a92d6/volumes"
Oct 30 21:17:05.339535 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:05.339492    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-caf211ae-rd6t4"]
Oct 30 21:17:05.345097 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:05.345071    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caf211ae-rd6t4"
Oct 30 21:17:05.400607 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:05.400574    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-caf211ae-rd6t4"]
Oct 30 21:17:05.445664 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:05.445632    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:17:05.445889 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:05.445846    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:17:05.445889 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:05.445844    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:17:05.447147 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:05.447130    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:17:05.499250 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:05.499219    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/241cf10b-84bc-455a-aa52-43cd7da69309-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-caf211ae-rd6t4\" (UID: \"241cf10b-84bc-455a-aa52-43cd7da69309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caf211ae-rd6t4"
Oct 30 21:17:05.499400 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:05.499261    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/241cf10b-84bc-455a-aa52-43cd7da69309-maas-api-xks-ca\") pod \"maas-api-revoke-sub-caf211ae-rd6t4\" (UID: \"241cf10b-84bc-455a-aa52-43cd7da69309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caf211ae-rd6t4"
Oct 30 21:17:05.499442 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:05.499416    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmdgr\" (UniqueName: \"kubernetes.io/projected/241cf10b-84bc-455a-aa52-43cd7da69309-kube-api-access-vmdgr\") pod \"maas-api-revoke-sub-caf211ae-rd6t4\" (UID: \"241cf10b-84bc-455a-aa52-43cd7da69309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caf211ae-rd6t4"
Oct 30 21:17:05.601079 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:05.601005    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmdgr\" (UniqueName: \"kubernetes.io/projected/241cf10b-84bc-455a-aa52-43cd7da69309-kube-api-access-vmdgr\") pod \"maas-api-revoke-sub-caf211ae-rd6t4\" (UID: \"241cf10b-84bc-455a-aa52-43cd7da69309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caf211ae-rd6t4"
Oct 30 21:17:05.601224 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:05.601152    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/241cf10b-84bc-455a-aa52-43cd7da69309-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-caf211ae-rd6t4\" (UID: \"241cf10b-84bc-455a-aa52-43cd7da69309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caf211ae-rd6t4"
Oct 30 21:17:05.601224 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:05.601178    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/241cf10b-84bc-455a-aa52-43cd7da69309-maas-api-xks-ca\") pod \"maas-api-revoke-sub-caf211ae-rd6t4\" (UID: \"241cf10b-84bc-455a-aa52-43cd7da69309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caf211ae-rd6t4"
Oct 30 21:17:05.602106 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:05.602032    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/241cf10b-84bc-455a-aa52-43cd7da69309-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-caf211ae-rd6t4\" (UID: \"241cf10b-84bc-455a-aa52-43cd7da69309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caf211ae-rd6t4"
Oct 30 21:17:05.603836 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:05.603814    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/241cf10b-84bc-455a-aa52-43cd7da69309-maas-api-xks-ca\") pod \"maas-api-revoke-sub-caf211ae-rd6t4\" (UID: \"241cf10b-84bc-455a-aa52-43cd7da69309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caf211ae-rd6t4"
Oct 30 21:17:05.658213 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:05.658190    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmdgr\" (UniqueName: \"kubernetes.io/projected/241cf10b-84bc-455a-aa52-43cd7da69309-kube-api-access-vmdgr\") pod \"maas-api-revoke-sub-caf211ae-rd6t4\" (UID: \"241cf10b-84bc-455a-aa52-43cd7da69309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caf211ae-rd6t4"
Oct 30 21:17:05.659009 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:05.658992    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caf211ae-rd6t4"
Oct 30 21:17:06.042954 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:06.042924    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-caf211ae-rd6t4"]
Oct 30 21:17:06.044104 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:17:06.044077    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod241cf10b_84bc_455a_aa52_43cd7da69309.slice/crio-7e206e2e602878494d99f35dfe0faf046877c657b7b6cb930ae5a3a54ed9ffdf WatchSource:0}: Error finding container 7e206e2e602878494d99f35dfe0faf046877c657b7b6cb930ae5a3a54ed9ffdf: Status 404 returned error can't find the container with id 7e206e2e602878494d99f35dfe0faf046877c657b7b6cb930ae5a3a54ed9ffdf
Oct 30 21:17:06.957950 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:06.957906    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caf211ae-rd6t4" event={"ID":"241cf10b-84bc-455a-aa52-43cd7da69309","Type":"ContainerStarted","Data":"2e1cf5cdc07bb868a125c9bedbd39f925396229d24ff6995c4bf38f70f88913d"}
Oct 30 21:17:06.958394 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:06.958010    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caf211ae-rd6t4" event={"ID":"241cf10b-84bc-455a-aa52-43cd7da69309","Type":"ContainerStarted","Data":"7e206e2e602878494d99f35dfe0faf046877c657b7b6cb930ae5a3a54ed9ffdf"}
Oct 30 21:17:07.074828 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:07.074745    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caf211ae-rd6t4" podStartSLOduration=2.074726125 podStartE2EDuration="2.074726125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:17:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:07.070528038 +0000 UTC m=+2310.135304438" watchObservedRunningTime="2026-10-30 21:17:07.074726125 +0000 UTC m=+2310.139502524"
Oct 30 21:17:07.967942 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:07.967901    5713 generic.go:356] "Generic (PLEG): container finished" podID="241cf10b-84bc-455a-aa52-43cd7da69309" containerID="2e1cf5cdc07bb868a125c9bedbd39f925396229d24ff6995c4bf38f70f88913d" exitCode=0
Oct 30 21:17:07.968477 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:07.967996    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caf211ae-rd6t4" event={"ID":"241cf10b-84bc-455a-aa52-43cd7da69309","Type":"ContainerDied","Data":"2e1cf5cdc07bb868a125c9bedbd39f925396229d24ff6995c4bf38f70f88913d"}
Oct 30 21:17:09.138677 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:09.138644    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caf211ae-rd6t4"
Oct 30 21:17:09.256755 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:09.256671    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/241cf10b-84bc-455a-aa52-43cd7da69309-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/241cf10b-84bc-455a-aa52-43cd7da69309-maas-api-openshift-ca\") pod \"241cf10b-84bc-455a-aa52-43cd7da69309\" (UID: \"241cf10b-84bc-455a-aa52-43cd7da69309\") "
Oct 30 21:17:09.256934 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:09.256811    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/241cf10b-84bc-455a-aa52-43cd7da69309-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/241cf10b-84bc-455a-aa52-43cd7da69309-maas-api-xks-ca\") pod \"241cf10b-84bc-455a-aa52-43cd7da69309\" (UID: \"241cf10b-84bc-455a-aa52-43cd7da69309\") "
Oct 30 21:17:09.256934 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:09.256869    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/241cf10b-84bc-455a-aa52-43cd7da69309-kube-api-access-vmdgr\" (UniqueName: \"kubernetes.io/projected/241cf10b-84bc-455a-aa52-43cd7da69309-kube-api-access-vmdgr\") pod \"241cf10b-84bc-455a-aa52-43cd7da69309\" (UID: \"241cf10b-84bc-455a-aa52-43cd7da69309\") "
Oct 30 21:17:09.257104 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:09.257075    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/241cf10b-84bc-455a-aa52-43cd7da69309-maas-api-openshift-ca" pod "241cf10b-84bc-455a-aa52-43cd7da69309" (UID: "241cf10b-84bc-455a-aa52-43cd7da69309"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:17:09.257380 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:09.257363    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/241cf10b-84bc-455a-aa52-43cd7da69309-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:17:09.259262 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:09.259229    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/241cf10b-84bc-455a-aa52-43cd7da69309-maas-api-xks-ca" pod "241cf10b-84bc-455a-aa52-43cd7da69309" (UID: "241cf10b-84bc-455a-aa52-43cd7da69309"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:17:09.259373 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:09.259275    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/241cf10b-84bc-455a-aa52-43cd7da69309-kube-api-access-vmdgr" pod "241cf10b-84bc-455a-aa52-43cd7da69309" (UID: "241cf10b-84bc-455a-aa52-43cd7da69309"). InnerVolumeSpecName "kube-api-access-vmdgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:17:09.358938 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:09.358900    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/241cf10b-84bc-455a-aa52-43cd7da69309-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:17:09.358938 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:09.358931    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmdgr\" (UniqueName: \"kubernetes.io/projected/241cf10b-84bc-455a-aa52-43cd7da69309-kube-api-access-vmdgr\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:17:09.988938 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:09.988904    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caf211ae-rd6t4"
Oct 30 21:17:09.989123 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:09.988932    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caf211ae-rd6t4" event={"ID":"241cf10b-84bc-455a-aa52-43cd7da69309","Type":"ContainerDied","Data":"7e206e2e602878494d99f35dfe0faf046877c657b7b6cb930ae5a3a54ed9ffdf"}
Oct 30 21:17:09.989123 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:09.989010    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e206e2e602878494d99f35dfe0faf046877c657b7b6cb930ae5a3a54ed9ffdf"
Oct 30 21:17:10.580474 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:10.580441    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b379ae2e-7gr7m"]
Oct 30 21:17:10.581108 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:10.581091    5713 cpu_manager.go:414] "containerMap: removing container" podUID="241cf10b-84bc-455a-aa52-43cd7da69309" containerName="revoke-keys"
Oct 30 21:17:10.581156 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:10.581109    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="241cf10b-84bc-455a-aa52-43cd7da69309" containerName="revoke-keys"
Oct 30 21:17:10.581264 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:10.581255    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="241cf10b-84bc-455a-aa52-43cd7da69309" containerName="revoke-keys"
Oct 30 21:17:10.584879 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:10.584863    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b379ae2e-7gr7m"
Oct 30 21:17:10.637826 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:10.636728    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e78449d-9whnj"]
Oct 30 21:17:10.641601 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:10.641571    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b379ae2e-7gr7m"]
Oct 30 21:17:10.641779 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:10.641679    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e78449d-9whnj"
Oct 30 21:17:10.682165 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:10.682124    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:17:10.684550 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:10.684531    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:17:10.684666 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:10.684648    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:17:10.684707 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:10.684647    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:17:10.696254 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:10.696218    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e78449d-9whnj"]
Oct 30 21:17:10.776604 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:10.776559    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggtwl\" (UniqueName: \"kubernetes.io/projected/e9494ab2-0676-4097-b0e3-e28c7a04cae2-kube-api-access-ggtwl\") pod \"maas-api-revoke-sub-6e78449d-9whnj\" (UID: \"e9494ab2-0676-4097-b0e3-e28c7a04cae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e78449d-9whnj"
Oct 30 21:17:10.776827 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:10.776714    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9494ab2-0676-4097-b0e3-e28c7a04cae2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e78449d-9whnj\" (UID: \"e9494ab2-0676-4097-b0e3-e28c7a04cae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e78449d-9whnj"
Oct 30 21:17:10.776827 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:10.776779    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68de3f53-caec-4639-ab62-0c225a536f8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b379ae2e-7gr7m\" (UID: \"68de3f53-caec-4639-ab62-0c225a536f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b379ae2e-7gr7m"
Oct 30 21:17:10.776917 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:10.776863    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68de3f53-caec-4639-ab62-0c225a536f8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b379ae2e-7gr7m\" (UID: \"68de3f53-caec-4639-ab62-0c225a536f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b379ae2e-7gr7m"
Oct 30 21:17:10.776917 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:10.776896    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lh5z\" (UniqueName: \"kubernetes.io/projected/68de3f53-caec-4639-ab62-0c225a536f8e-kube-api-access-2lh5z\") pod \"maas-api-revoke-sub-b379ae2e-7gr7m\" (UID: \"68de3f53-caec-4639-ab62-0c225a536f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b379ae2e-7gr7m"
Oct 30 21:17:10.777126 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:10.777100    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9494ab2-0676-4097-b0e3-e28c7a04cae2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e78449d-9whnj\" (UID: \"e9494ab2-0676-4097-b0e3-e28c7a04cae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e78449d-9whnj"
Oct 30 21:17:10.878164 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:10.878070    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68de3f53-caec-4639-ab62-0c225a536f8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b379ae2e-7gr7m\" (UID: \"68de3f53-caec-4639-ab62-0c225a536f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b379ae2e-7gr7m"
Oct 30 21:17:10.878164 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:10.878134    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68de3f53-caec-4639-ab62-0c225a536f8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b379ae2e-7gr7m\" (UID: \"68de3f53-caec-4639-ab62-0c225a536f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b379ae2e-7gr7m"
Oct 30 21:17:10.878164 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:10.878165    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lh5z\" (UniqueName: \"kubernetes.io/projected/68de3f53-caec-4639-ab62-0c225a536f8e-kube-api-access-2lh5z\") pod \"maas-api-revoke-sub-b379ae2e-7gr7m\" (UID: \"68de3f53-caec-4639-ab62-0c225a536f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b379ae2e-7gr7m"
Oct 30 21:17:10.878387 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:10.878246    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9494ab2-0676-4097-b0e3-e28c7a04cae2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e78449d-9whnj\" (UID: \"e9494ab2-0676-4097-b0e3-e28c7a04cae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e78449d-9whnj"
Oct 30 21:17:10.878387 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:10.878306    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggtwl\" (UniqueName: \"kubernetes.io/projected/e9494ab2-0676-4097-b0e3-e28c7a04cae2-kube-api-access-ggtwl\") pod \"maas-api-revoke-sub-6e78449d-9whnj\" (UID: \"e9494ab2-0676-4097-b0e3-e28c7a04cae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e78449d-9whnj"
Oct 30 21:17:10.878474 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:10.878390    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9494ab2-0676-4097-b0e3-e28c7a04cae2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e78449d-9whnj\" (UID: \"e9494ab2-0676-4097-b0e3-e28c7a04cae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e78449d-9whnj"
Oct 30 21:17:10.878946 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:10.878879    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68de3f53-caec-4639-ab62-0c225a536f8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b379ae2e-7gr7m\" (UID: \"68de3f53-caec-4639-ab62-0c225a536f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b379ae2e-7gr7m"
Oct 30 21:17:10.879195 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:10.879072    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9494ab2-0676-4097-b0e3-e28c7a04cae2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e78449d-9whnj\" (UID: \"e9494ab2-0676-4097-b0e3-e28c7a04cae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e78449d-9whnj"
Oct 30 21:17:10.880879 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:10.880861    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68de3f53-caec-4639-ab62-0c225a536f8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b379ae2e-7gr7m\" (UID: \"68de3f53-caec-4639-ab62-0c225a536f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b379ae2e-7gr7m"
Oct 30 21:17:10.881023 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:10.880890    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9494ab2-0676-4097-b0e3-e28c7a04cae2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e78449d-9whnj\" (UID: \"e9494ab2-0676-4097-b0e3-e28c7a04cae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e78449d-9whnj"
Oct 30 21:17:10.908904 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:10.908878    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-639b07-6dd84859d7-k49qq"
Oct 30 21:17:10.938124 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:10.938091    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggtwl\" (UniqueName: \"kubernetes.io/projected/e9494ab2-0676-4097-b0e3-e28c7a04cae2-kube-api-access-ggtwl\") pod \"maas-api-revoke-sub-6e78449d-9whnj\" (UID: \"e9494ab2-0676-4097-b0e3-e28c7a04cae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e78449d-9whnj"
Oct 30 21:17:10.939478 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:10.939449    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lh5z\" (UniqueName: \"kubernetes.io/projected/68de3f53-caec-4639-ab62-0c225a536f8e-kube-api-access-2lh5z\") pod \"maas-api-revoke-sub-b379ae2e-7gr7m\" (UID: \"68de3f53-caec-4639-ab62-0c225a536f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b379ae2e-7gr7m"
Oct 30 21:17:10.960399 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:10.960366    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e78449d-9whnj"
Oct 30 21:17:11.199043 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:11.199007    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b379ae2e-7gr7m"
Oct 30 21:17:11.345835 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:11.345782    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e78449d-9whnj"]
Oct 30 21:17:11.346297 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:17:11.346272    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9494ab2_0676_4097_b0e3_e28c7a04cae2.slice/crio-8c272a16145c699813a2e294d3dd995ce5c08fbc1a4ca69ebacb376e23706a3c WatchSource:0}: Error finding container 8c272a16145c699813a2e294d3dd995ce5c08fbc1a4ca69ebacb376e23706a3c: Status 404 returned error can't find the container with id 8c272a16145c699813a2e294d3dd995ce5c08fbc1a4ca69ebacb376e23706a3c
Oct 30 21:17:11.578229 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:11.578202    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b379ae2e-7gr7m"]
Oct 30 21:17:11.578971 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:17:11.578950    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68de3f53_caec_4639_ab62_0c225a536f8e.slice/crio-f0a2a4ddc2ec9ea6a91c11aa35e71358d752a8ab04aff58c1ac92ef174cdcbdb WatchSource:0}: Error finding container f0a2a4ddc2ec9ea6a91c11aa35e71358d752a8ab04aff58c1ac92ef174cdcbdb: Status 404 returned error can't find the container with id f0a2a4ddc2ec9ea6a91c11aa35e71358d752a8ab04aff58c1ac92ef174cdcbdb
Oct 30 21:17:11.917226 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:11.917196    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-639b07-59547c9664-bqxz8"
Oct 30 21:17:12.010471 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:12.010431    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b379ae2e-7gr7m" event={"ID":"68de3f53-caec-4639-ab62-0c225a536f8e","Type":"ContainerStarted","Data":"d9b2b969fb1bce450c7bcb55efe7f7f26c3a8691ca5cc084880d196e62eec877"}
Oct 30 21:17:12.010655 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:12.010495    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b379ae2e-7gr7m" event={"ID":"68de3f53-caec-4639-ab62-0c225a536f8e","Type":"ContainerStarted","Data":"f0a2a4ddc2ec9ea6a91c11aa35e71358d752a8ab04aff58c1ac92ef174cdcbdb"}
Oct 30 21:17:12.012241 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:12.012216    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e78449d-9whnj" event={"ID":"e9494ab2-0676-4097-b0e3-e28c7a04cae2","Type":"ContainerStarted","Data":"006654326e72547b4c6660562faa7365d04c9b61c23f23549f15dbb89b2d1211"}
Oct 30 21:17:12.012372 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:12.012262    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e78449d-9whnj" event={"ID":"e9494ab2-0676-4097-b0e3-e28c7a04cae2","Type":"ContainerStarted","Data":"8c272a16145c699813a2e294d3dd995ce5c08fbc1a4ca69ebacb376e23706a3c"}
Oct 30 21:17:12.148535 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:12.148430    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e78449d-9whnj" podStartSLOduration=2.148414706 podStartE2EDuration="2.148414706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:17:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:12.146143013 +0000 UTC m=+2315.210919412" watchObservedRunningTime="2026-10-30 21:17:12.148414706 +0000 UTC m=+2315.213191104"
Oct 30 21:17:12.259956 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:12.259905    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b379ae2e-7gr7m" podStartSLOduration=2.259890029 podStartE2EDuration="2.259890029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:17:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:12.258224743 +0000 UTC m=+2315.323001142" watchObservedRunningTime="2026-10-30 21:17:12.259890029 +0000 UTC m=+2315.324666457"
Oct 30 21:17:13.021972 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:13.021933    5713 generic.go:356] "Generic (PLEG): container finished" podID="68de3f53-caec-4639-ab62-0c225a536f8e" containerID="d9b2b969fb1bce450c7bcb55efe7f7f26c3a8691ca5cc084880d196e62eec877" exitCode=0
Oct 30 21:17:13.022380 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:13.022020    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b379ae2e-7gr7m" event={"ID":"68de3f53-caec-4639-ab62-0c225a536f8e","Type":"ContainerDied","Data":"d9b2b969fb1bce450c7bcb55efe7f7f26c3a8691ca5cc084880d196e62eec877"}
Oct 30 21:17:13.023903 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:13.023865    5713 generic.go:356] "Generic (PLEG): container finished" podID="e9494ab2-0676-4097-b0e3-e28c7a04cae2" containerID="006654326e72547b4c6660562faa7365d04c9b61c23f23549f15dbb89b2d1211" exitCode=0
Oct 30 21:17:13.024044 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:13.023947    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e78449d-9whnj" event={"ID":"e9494ab2-0676-4097-b0e3-e28c7a04cae2","Type":"ContainerDied","Data":"006654326e72547b4c6660562faa7365d04c9b61c23f23549f15dbb89b2d1211"}
Oct 30 21:17:14.218227 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:14.218203    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b379ae2e-7gr7m"
Oct 30 21:17:14.221658 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:14.221638    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e78449d-9whnj"
Oct 30 21:17:14.331655 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:14.331562    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9494ab2-0676-4097-b0e3-e28c7a04cae2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9494ab2-0676-4097-b0e3-e28c7a04cae2-maas-api-openshift-ca\") pod \"e9494ab2-0676-4097-b0e3-e28c7a04cae2\" (UID: \"e9494ab2-0676-4097-b0e3-e28c7a04cae2\") "
Oct 30 21:17:14.331655 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:14.331649    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9494ab2-0676-4097-b0e3-e28c7a04cae2-kube-api-access-ggtwl\" (UniqueName: \"kubernetes.io/projected/e9494ab2-0676-4097-b0e3-e28c7a04cae2-kube-api-access-ggtwl\") pod \"e9494ab2-0676-4097-b0e3-e28c7a04cae2\" (UID: \"e9494ab2-0676-4097-b0e3-e28c7a04cae2\") "
Oct 30 21:17:14.331927 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:14.331733    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68de3f53-caec-4639-ab62-0c225a536f8e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68de3f53-caec-4639-ab62-0c225a536f8e-maas-api-xks-ca\") pod \"68de3f53-caec-4639-ab62-0c225a536f8e\" (UID: \"68de3f53-caec-4639-ab62-0c225a536f8e\") "
Oct 30 21:17:14.331927 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:14.331779    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9494ab2-0676-4097-b0e3-e28c7a04cae2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9494ab2-0676-4097-b0e3-e28c7a04cae2-maas-api-xks-ca\") pod \"e9494ab2-0676-4097-b0e3-e28c7a04cae2\" (UID: \"e9494ab2-0676-4097-b0e3-e28c7a04cae2\") "
Oct 30 21:17:14.332040 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:14.331956    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68de3f53-caec-4639-ab62-0c225a536f8e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68de3f53-caec-4639-ab62-0c225a536f8e-maas-api-openshift-ca\") pod \"68de3f53-caec-4639-ab62-0c225a536f8e\" (UID: \"68de3f53-caec-4639-ab62-0c225a536f8e\") "
Oct 30 21:17:14.332040 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:14.332004    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68de3f53-caec-4639-ab62-0c225a536f8e-kube-api-access-2lh5z\" (UniqueName: \"kubernetes.io/projected/68de3f53-caec-4639-ab62-0c225a536f8e-kube-api-access-2lh5z\") pod \"68de3f53-caec-4639-ab62-0c225a536f8e\" (UID: \"68de3f53-caec-4639-ab62-0c225a536f8e\") "
Oct 30 21:17:14.332395 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:14.332365    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68de3f53-caec-4639-ab62-0c225a536f8e-maas-api-openshift-ca" pod "68de3f53-caec-4639-ab62-0c225a536f8e" (UID: "68de3f53-caec-4639-ab62-0c225a536f8e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:17:14.332715 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:14.332687    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68de3f53-caec-4639-ab62-0c225a536f8e-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:17:14.333242 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:14.333205    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9494ab2-0676-4097-b0e3-e28c7a04cae2-maas-api-openshift-ca" pod "e9494ab2-0676-4097-b0e3-e28c7a04cae2" (UID: "e9494ab2-0676-4097-b0e3-e28c7a04cae2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:17:14.334749 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:14.334710    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68de3f53-caec-4639-ab62-0c225a536f8e-maas-api-xks-ca" pod "68de3f53-caec-4639-ab62-0c225a536f8e" (UID: "68de3f53-caec-4639-ab62-0c225a536f8e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:17:14.336022 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:14.335832    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68de3f53-caec-4639-ab62-0c225a536f8e-kube-api-access-2lh5z" pod "68de3f53-caec-4639-ab62-0c225a536f8e" (UID: "68de3f53-caec-4639-ab62-0c225a536f8e"). InnerVolumeSpecName "kube-api-access-2lh5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:17:14.336022 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:14.335912    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9494ab2-0676-4097-b0e3-e28c7a04cae2-kube-api-access-ggtwl" pod "e9494ab2-0676-4097-b0e3-e28c7a04cae2" (UID: "e9494ab2-0676-4097-b0e3-e28c7a04cae2"). InnerVolumeSpecName "kube-api-access-ggtwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:17:14.336379 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:14.336128    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9494ab2-0676-4097-b0e3-e28c7a04cae2-maas-api-xks-ca" pod "e9494ab2-0676-4097-b0e3-e28c7a04cae2" (UID: "e9494ab2-0676-4097-b0e3-e28c7a04cae2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:17:14.434547 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:14.434520    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9494ab2-0676-4097-b0e3-e28c7a04cae2-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:17:14.434547 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:14.434548    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lh5z\" (UniqueName: \"kubernetes.io/projected/68de3f53-caec-4639-ab62-0c225a536f8e-kube-api-access-2lh5z\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:17:14.434726 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:14.434559    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9494ab2-0676-4097-b0e3-e28c7a04cae2-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:17:14.434726 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:14.434568    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggtwl\" (UniqueName: \"kubernetes.io/projected/e9494ab2-0676-4097-b0e3-e28c7a04cae2-kube-api-access-ggtwl\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:17:14.434726 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:14.434580    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68de3f53-caec-4639-ab62-0c225a536f8e-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:17:15.044165 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:15.044131    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b379ae2e-7gr7m"
Oct 30 21:17:15.044361 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:15.044142    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b379ae2e-7gr7m" event={"ID":"68de3f53-caec-4639-ab62-0c225a536f8e","Type":"ContainerDied","Data":"f0a2a4ddc2ec9ea6a91c11aa35e71358d752a8ab04aff58c1ac92ef174cdcbdb"}
Oct 30 21:17:15.044361 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:15.044266    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0a2a4ddc2ec9ea6a91c11aa35e71358d752a8ab04aff58c1ac92ef174cdcbdb"
Oct 30 21:17:15.045973 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:15.045948    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e78449d-9whnj"
Oct 30 21:17:15.046103 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:15.045971    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e78449d-9whnj" event={"ID":"e9494ab2-0676-4097-b0e3-e28c7a04cae2","Type":"ContainerDied","Data":"8c272a16145c699813a2e294d3dd995ce5c08fbc1a4ca69ebacb376e23706a3c"}
Oct 30 21:17:15.046103 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:15.046010    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c272a16145c699813a2e294d3dd995ce5c08fbc1a4ca69ebacb376e23706a3c"
Oct 30 21:17:16.734570 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:16.734537    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f9f277-mfvlh"]
Oct 30 21:17:16.735295 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:16.735269    5713 cpu_manager.go:414] "containerMap: removing container" podUID="e9494ab2-0676-4097-b0e3-e28c7a04cae2" containerName="revoke-keys"
Oct 30 21:17:16.735295 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:16.735293    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9494ab2-0676-4097-b0e3-e28c7a04cae2" containerName="revoke-keys"
Oct 30 21:17:16.735450 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:16.735331    5713 cpu_manager.go:414] "containerMap: removing container" podUID="68de3f53-caec-4639-ab62-0c225a536f8e" containerName="revoke-keys"
Oct 30 21:17:16.735450 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:16.735337    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68de3f53-caec-4639-ab62-0c225a536f8e" containerName="revoke-keys"
Oct 30 21:17:16.735518 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:16.735450    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="68de3f53-caec-4639-ab62-0c225a536f8e" containerName="revoke-keys"
Oct 30 21:17:16.735518 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:16.735467    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9494ab2-0676-4097-b0e3-e28c7a04cae2" containerName="revoke-keys"
Oct 30 21:17:16.739159 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:16.739142    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f9f277-mfvlh"
Oct 30 21:17:16.796368 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:16.796343    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f9f277-mfvlh"]
Oct 30 21:17:16.841093 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:16.841051    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:17:16.841093 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:16.841070    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:17:16.841093 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:16.841069    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:17:16.841093 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:16.841061    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:17:16.866202 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:16.866173    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e635c82-bd6d-48b9-b19a-4d6f1cb4b2bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-f9f277-mfvlh\" (UID: \"9e635c82-bd6d-48b9-b19a-4d6f1cb4b2bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f9f277-mfvlh"
Oct 30 21:17:16.866401 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:16.866385    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwpvj\" (UniqueName: \"kubernetes.io/projected/9e635c82-bd6d-48b9-b19a-4d6f1cb4b2bc-kube-api-access-cwpvj\") pod \"maas-api-revoke-keys-e2e-worker-w1-f9f277-mfvlh\" (UID: \"9e635c82-bd6d-48b9-b19a-4d6f1cb4b2bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f9f277-mfvlh"
Oct 30 21:17:16.866477 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:16.866426    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e635c82-bd6d-48b9-b19a-4d6f1cb4b2bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-f9f277-mfvlh\" (UID: \"9e635c82-bd6d-48b9-b19a-4d6f1cb4b2bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f9f277-mfvlh"
Oct 30 21:17:16.968190 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:16.968161    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwpvj\" (UniqueName: \"kubernetes.io/projected/9e635c82-bd6d-48b9-b19a-4d6f1cb4b2bc-kube-api-access-cwpvj\") pod \"maas-api-revoke-keys-e2e-worker-w1-f9f277-mfvlh\" (UID: \"9e635c82-bd6d-48b9-b19a-4d6f1cb4b2bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f9f277-mfvlh"
Oct 30 21:17:16.968374 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:16.968207    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e635c82-bd6d-48b9-b19a-4d6f1cb4b2bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-f9f277-mfvlh\" (UID: \"9e635c82-bd6d-48b9-b19a-4d6f1cb4b2bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f9f277-mfvlh"
Oct 30 21:17:16.968374 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:16.968263    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e635c82-bd6d-48b9-b19a-4d6f1cb4b2bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-f9f277-mfvlh\" (UID: \"9e635c82-bd6d-48b9-b19a-4d6f1cb4b2bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f9f277-mfvlh"
Oct 30 21:17:16.969025 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:16.969002    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e635c82-bd6d-48b9-b19a-4d6f1cb4b2bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-f9f277-mfvlh\" (UID: \"9e635c82-bd6d-48b9-b19a-4d6f1cb4b2bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f9f277-mfvlh"
Oct 30 21:17:16.971099 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:16.971082    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e635c82-bd6d-48b9-b19a-4d6f1cb4b2bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-f9f277-mfvlh\" (UID: \"9e635c82-bd6d-48b9-b19a-4d6f1cb4b2bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f9f277-mfvlh"
Oct 30 21:17:17.025208 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:17.025140    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwpvj\" (UniqueName: \"kubernetes.io/projected/9e635c82-bd6d-48b9-b19a-4d6f1cb4b2bc-kube-api-access-cwpvj\") pod \"maas-api-revoke-keys-e2e-worker-w1-f9f277-mfvlh\" (UID: \"9e635c82-bd6d-48b9-b19a-4d6f1cb4b2bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f9f277-mfvlh"
Oct 30 21:17:17.053183 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:17.053155    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f9f277-mfvlh"
Oct 30 21:17:17.437740 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:17.437708    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f9f277-mfvlh"]
Oct 30 21:17:17.437985 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:17:17.437963    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e635c82_bd6d_48b9_b19a_4d6f1cb4b2bc.slice/crio-c782dc74c9332eaee2cb7561fe41440a8754bb8dda6e573389be65718115b081 WatchSource:0}: Error finding container c782dc74c9332eaee2cb7561fe41440a8754bb8dda6e573389be65718115b081: Status 404 returned error can't find the container with id c782dc74c9332eaee2cb7561fe41440a8754bb8dda6e573389be65718115b081
Oct 30 21:17:18.080528 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:18.080429    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f9f277-mfvlh" event={"ID":"9e635c82-bd6d-48b9-b19a-4d6f1cb4b2bc","Type":"ContainerStarted","Data":"1b96c773bdfa48925cff24fbd4767064c77bfcc230379e2d4b92f70e5d81fa76"}
Oct 30 21:17:18.080528 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:18.080486    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f9f277-mfvlh" event={"ID":"9e635c82-bd6d-48b9-b19a-4d6f1cb4b2bc","Type":"ContainerStarted","Data":"c782dc74c9332eaee2cb7561fe41440a8754bb8dda6e573389be65718115b081"}
Oct 30 21:17:18.201158 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:18.201090    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f9f277-mfvlh" podStartSLOduration=2.201073503 podStartE2EDuration="2.201073503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:17:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:18.197007139 +0000 UTC m=+2321.261783539" watchObservedRunningTime="2026-10-30 21:17:18.201073503 +0000 UTC m=+2321.265849907"
Oct 30 21:17:19.089686 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:19.089650    5713 generic.go:356] "Generic (PLEG): container finished" podID="9e635c82-bd6d-48b9-b19a-4d6f1cb4b2bc" containerID="1b96c773bdfa48925cff24fbd4767064c77bfcc230379e2d4b92f70e5d81fa76" exitCode=0
Oct 30 21:17:19.090293 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:19.090084    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f9f277-mfvlh" event={"ID":"9e635c82-bd6d-48b9-b19a-4d6f1cb4b2bc","Type":"ContainerDied","Data":"1b96c773bdfa48925cff24fbd4767064c77bfcc230379e2d4b92f70e5d81fa76"}
Oct 30 21:17:20.259593 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:20.259568    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f9f277-mfvlh"
Oct 30 21:17:20.319585 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:20.319556    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e635c82-bd6d-48b9-b19a-4d6f1cb4b2bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e635c82-bd6d-48b9-b19a-4d6f1cb4b2bc-maas-api-openshift-ca\") pod \"9e635c82-bd6d-48b9-b19a-4d6f1cb4b2bc\" (UID: \"9e635c82-bd6d-48b9-b19a-4d6f1cb4b2bc\") "
Oct 30 21:17:20.319773 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:20.319617    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e635c82-bd6d-48b9-b19a-4d6f1cb4b2bc-kube-api-access-cwpvj\" (UniqueName: \"kubernetes.io/projected/9e635c82-bd6d-48b9-b19a-4d6f1cb4b2bc-kube-api-access-cwpvj\") pod \"9e635c82-bd6d-48b9-b19a-4d6f1cb4b2bc\" (UID: \"9e635c82-bd6d-48b9-b19a-4d6f1cb4b2bc\") "
Oct 30 21:17:20.319773 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:20.319727    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e635c82-bd6d-48b9-b19a-4d6f1cb4b2bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e635c82-bd6d-48b9-b19a-4d6f1cb4b2bc-maas-api-xks-ca\") pod \"9e635c82-bd6d-48b9-b19a-4d6f1cb4b2bc\" (UID: \"9e635c82-bd6d-48b9-b19a-4d6f1cb4b2bc\") "
Oct 30 21:17:20.319963 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:20.319942    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e635c82-bd6d-48b9-b19a-4d6f1cb4b2bc-maas-api-openshift-ca" pod "9e635c82-bd6d-48b9-b19a-4d6f1cb4b2bc" (UID: "9e635c82-bd6d-48b9-b19a-4d6f1cb4b2bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:17:20.320265 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:20.320241    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e635c82-bd6d-48b9-b19a-4d6f1cb4b2bc-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:17:20.322093 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:20.322068    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e635c82-bd6d-48b9-b19a-4d6f1cb4b2bc-kube-api-access-cwpvj" pod "9e635c82-bd6d-48b9-b19a-4d6f1cb4b2bc" (UID: "9e635c82-bd6d-48b9-b19a-4d6f1cb4b2bc"). InnerVolumeSpecName "kube-api-access-cwpvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:17:20.322093 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:20.322085    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e635c82-bd6d-48b9-b19a-4d6f1cb4b2bc-maas-api-xks-ca" pod "9e635c82-bd6d-48b9-b19a-4d6f1cb4b2bc" (UID: "9e635c82-bd6d-48b9-b19a-4d6f1cb4b2bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:17:20.422166 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:20.422082    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwpvj\" (UniqueName: \"kubernetes.io/projected/9e635c82-bd6d-48b9-b19a-4d6f1cb4b2bc-kube-api-access-cwpvj\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:17:20.422166 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:20.422114    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e635c82-bd6d-48b9-b19a-4d6f1cb4b2bc-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:17:21.064863 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:21.064830    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-f9f277/premium-w1-f9f277-kserve-5dbfd4d496-72w4n"]
Oct 30 21:17:21.106445 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:21.106413    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f9f277-mfvlh"
Oct 30 21:17:21.106602 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:21.106441    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f9f277-mfvlh" event={"ID":"9e635c82-bd6d-48b9-b19a-4d6f1cb4b2bc","Type":"ContainerDied","Data":"c782dc74c9332eaee2cb7561fe41440a8754bb8dda6e573389be65718115b081"}
Oct 30 21:17:21.106602 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:21.106492    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c782dc74c9332eaee2cb7561fe41440a8754bb8dda6e573389be65718115b081"
Oct 30 21:17:24.462717 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:17:24.462672    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:17:35.250353 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:35.250318    5713 generic.go:356] "Generic (PLEG): container finished" podID="4d34434b-cc8a-4ee2-a774-cacdea5465de" containerID="e00836866e297848f37cc5fd9f64d6e203b7203567da64847fd01764c6023368" exitCode=0
Oct 30 21:17:35.250837 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:35.250392    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh" event={"ID":"4d34434b-cc8a-4ee2-a774-cacdea5465de","Type":"ContainerDied","Data":"e00836866e297848f37cc5fd9f64d6e203b7203567da64847fd01764c6023368"}
Oct 30 21:17:35.250837 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:35.250763    5713 scope.go:122] "RemoveContainer" containerID="e00836866e297848f37cc5fd9f64d6e203b7203567da64847fd01764c6023368"
Oct 30 21:17:35.252411 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:35.252386    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:17:35.252483 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:35.252464    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:17:35.252526 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:35.252497    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:17:36.090068 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:36.090028    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-f9f277/premium-w1-f9f277-kserve-5dbfd4d496-72w4n" podUID="1cc1f811-c149-4457-80f1-1a55621d73b4" containerName="main" containerID="cri-o://1f11d3a069df800c1ba3e46c988c04320df7c73816f7a56aadc02411254a1afc" gracePeriod=45
Oct 30 21:17:36.283810 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:36.283658    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh" event={"ID":"4d34434b-cc8a-4ee2-a774-cacdea5465de","Type":"ContainerStarted","Data":"de9c872d9aa839eacc1977ca29a1b5845f26958755b06df95d519916c82821cd"}
Oct 30 21:17:36.287070 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:36.287040    5713 generic.go:356] "Generic (PLEG): container finished" podID="1cc1f811-c149-4457-80f1-1a55621d73b4" containerID="1f11d3a069df800c1ba3e46c988c04320df7c73816f7a56aadc02411254a1afc" exitCode=0
Oct 30 21:17:36.287233 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:36.287079    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f9f277/premium-w1-f9f277-kserve-5dbfd4d496-72w4n" event={"ID":"1cc1f811-c149-4457-80f1-1a55621d73b4","Type":"ContainerDied","Data":"1f11d3a069df800c1ba3e46c988c04320df7c73816f7a56aadc02411254a1afc"}
Oct 30 21:17:36.432444 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:36.432419    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f9f277/premium-w1-f9f277-kserve-5dbfd4d496-72w4n"
Oct 30 21:17:36.493817 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:36.493338    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cc1f811-c149-4457-80f1-1a55621d73b4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cc1f811-c149-4457-80f1-1a55621d73b4-dshm\") pod \"1cc1f811-c149-4457-80f1-1a55621d73b4\" (UID: \"1cc1f811-c149-4457-80f1-1a55621d73b4\") "
Oct 30 21:17:36.493817 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:36.493395    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cc1f811-c149-4457-80f1-1a55621d73b4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cc1f811-c149-4457-80f1-1a55621d73b4-tmp-dir\") pod \"1cc1f811-c149-4457-80f1-1a55621d73b4\" (UID: \"1cc1f811-c149-4457-80f1-1a55621d73b4\") "
Oct 30 21:17:36.493817 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:36.493461    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cc1f811-c149-4457-80f1-1a55621d73b4-home\" (UniqueName: \"kubernetes.io/empty-dir/1cc1f811-c149-4457-80f1-1a55621d73b4-home\") pod \"1cc1f811-c149-4457-80f1-1a55621d73b4\" (UID: \"1cc1f811-c149-4457-80f1-1a55621d73b4\") "
Oct 30 21:17:36.493817 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:36.493548    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cc1f811-c149-4457-80f1-1a55621d73b4-kube-api-access-txhj7\" (UniqueName: \"kubernetes.io/projected/1cc1f811-c149-4457-80f1-1a55621d73b4-kube-api-access-txhj7\") pod \"1cc1f811-c149-4457-80f1-1a55621d73b4\" (UID: \"1cc1f811-c149-4457-80f1-1a55621d73b4\") "
Oct 30 21:17:36.493817 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:36.493645    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cc1f811-c149-4457-80f1-1a55621d73b4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cc1f811-c149-4457-80f1-1a55621d73b4-model-cache\") pod \"1cc1f811-c149-4457-80f1-1a55621d73b4\" (UID: \"1cc1f811-c149-4457-80f1-1a55621d73b4\") "
Oct 30 21:17:36.493817 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:36.493743    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1cc1f811-c149-4457-80f1-1a55621d73b4-tls-certs\" (UniqueName: \"kubernetes.io/secret/1cc1f811-c149-4457-80f1-1a55621d73b4-tls-certs\") pod \"1cc1f811-c149-4457-80f1-1a55621d73b4\" (UID: \"1cc1f811-c149-4457-80f1-1a55621d73b4\") "
Oct 30 21:17:36.498818 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:36.494829    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cc1f811-c149-4457-80f1-1a55621d73b4-home" pod "1cc1f811-c149-4457-80f1-1a55621d73b4" (UID: "1cc1f811-c149-4457-80f1-1a55621d73b4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:17:36.498818 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:36.495083    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cc1f811-c149-4457-80f1-1a55621d73b4-tmp-dir" pod "1cc1f811-c149-4457-80f1-1a55621d73b4" (UID: "1cc1f811-c149-4457-80f1-1a55621d73b4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:17:36.498818 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:36.495195    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cc1f811-c149-4457-80f1-1a55621d73b4-model-cache" pod "1cc1f811-c149-4457-80f1-1a55621d73b4" (UID: "1cc1f811-c149-4457-80f1-1a55621d73b4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:17:36.502603 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:36.502508    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cc1f811-c149-4457-80f1-1a55621d73b4-tls-certs" pod "1cc1f811-c149-4457-80f1-1a55621d73b4" (UID: "1cc1f811-c149-4457-80f1-1a55621d73b4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:17:36.504495 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:36.504431    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cc1f811-c149-4457-80f1-1a55621d73b4-dshm" pod "1cc1f811-c149-4457-80f1-1a55621d73b4" (UID: "1cc1f811-c149-4457-80f1-1a55621d73b4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:17:36.505349 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:36.505317    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cc1f811-c149-4457-80f1-1a55621d73b4-kube-api-access-txhj7" pod "1cc1f811-c149-4457-80f1-1a55621d73b4" (UID: "1cc1f811-c149-4457-80f1-1a55621d73b4"). InnerVolumeSpecName "kube-api-access-txhj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:17:36.595540 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:36.595362    5713 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cc1f811-c149-4457-80f1-1a55621d73b4-tmp-dir\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:17:36.595540 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:36.595394    5713 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cc1f811-c149-4457-80f1-1a55621d73b4-home\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:17:36.595540 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:36.595404    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txhj7\" (UniqueName: \"kubernetes.io/projected/1cc1f811-c149-4457-80f1-1a55621d73b4-kube-api-access-txhj7\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:17:36.595540 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:36.595416    5713 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cc1f811-c149-4457-80f1-1a55621d73b4-model-cache\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:17:36.595540 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:36.595440    5713 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cc1f811-c149-4457-80f1-1a55621d73b4-tls-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:17:36.595540 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:36.595455    5713 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cc1f811-c149-4457-80f1-1a55621d73b4-dshm\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:17:36.756090 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:36.753226    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh"
Oct 30 21:17:36.765953 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:36.765918    5713 patch_prober.go:29] interesting pod/e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Oct 30 21:17:36.766148 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:36.765980    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh" podUID="4d34434b-cc8a-4ee2-a774-cacdea5465de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Oct 30 21:17:37.296036 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:37.296003    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f9f277/premium-w1-f9f277-kserve-5dbfd4d496-72w4n"
Oct 30 21:17:37.296528 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:37.296047    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f9f277/premium-w1-f9f277-kserve-5dbfd4d496-72w4n" event={"ID":"1cc1f811-c149-4457-80f1-1a55621d73b4","Type":"ContainerDied","Data":"87c2ef76541bea66006ea0c4c0146177cf1fb3cb0584816acea452283471d9dd"}
Oct 30 21:17:37.296528 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:37.296114    5713 scope.go:122] "RemoveContainer" containerID="1f11d3a069df800c1ba3e46c988c04320df7c73816f7a56aadc02411254a1afc"
Oct 30 21:17:37.468828 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:37.468777    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-f9f277/premium-w1-f9f277-kserve-5dbfd4d496-72w4n"]
Oct 30 21:17:37.518057 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:37.518023    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-f9f277/premium-w1-f9f277-kserve-5dbfd4d496-72w4n"]
Oct 30 21:17:37.751775 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:37.751228    5713 patch_prober.go:29] interesting pod/e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Oct 30 21:17:37.751775 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:37.751293    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh" podUID="4d34434b-cc8a-4ee2-a774-cacdea5465de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Oct 30 21:17:37.771548 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:37.771517    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh"
Oct 30 21:17:38.224550 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:38.224518    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cb6404-hzrp4"]
Oct 30 21:17:38.227755 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:38.227732    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cb6404-hzrp4"]
Oct 30 21:17:38.753815 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:38.752284    5713 patch_prober.go:29] interesting pod/e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Oct 30 21:17:38.753815 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:38.752367    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh" podUID="4d34434b-cc8a-4ee2-a774-cacdea5465de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Oct 30 21:17:39.062205 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:39.062111    5713 scope.go:122] "RemoveContainer" containerID="38a6d001c0c0e7a02be45091eb6e21fc7f7b4f8e5e66e31c875c39424edb5f7e"
Oct 30 21:17:39.067961 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:39.067934    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cc1f811-c149-4457-80f1-1a55621d73b4" path="/var/lib/kubelet/pods/1cc1f811-c149-4457-80f1-1a55621d73b4/volumes"
Oct 30 21:17:39.068437 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:39.068416    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2ecbf1c-d317-4e83-98ac-4aea45a351c3" path="/var/lib/kubelet/pods/f2ecbf1c-d317-4e83-98ac-4aea45a351c3/volumes"
Oct 30 21:17:39.076968 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:39.076946    5713 scope.go:122] "RemoveContainer" containerID="a01279cbb593eac7fc7dfaef3670291c2c6940a4f73e45a8ad18a3f0b75eaafd"
Oct 30 21:17:39.756815 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:39.751903    5713 patch_prober.go:29] interesting pod/e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Oct 30 21:17:39.756815 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:39.751987    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh" podUID="4d34434b-cc8a-4ee2-a774-cacdea5465de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Oct 30 21:17:40.756256 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:40.756208    5713 patch_prober.go:29] interesting pod/e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Oct 30 21:17:40.756494 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:40.756280    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh" podUID="4d34434b-cc8a-4ee2-a774-cacdea5465de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Oct 30 21:17:41.753641 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:41.753606    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh"
Oct 30 21:17:41.754951 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:41.754931    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh"
Oct 30 21:17:44.903914 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:44.903868    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk"]
Oct 30 21:17:44.904408 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:44.904145    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk" podUID="237d222b-50be-4b97-b53d-49187434c09b" containerName="istio-proxy" containerID="cri-o://332957b8ee59b70dc1edf253b8a8fe242676318b11019866f8f09563dbb41904" gracePeriod=30
Oct 30 21:17:45.935553 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:45.935515    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-781058-dlclg"]
Oct 30 21:17:45.936235 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:45.936219    5713 cpu_manager.go:414] "containerMap: removing container" podUID="9e635c82-bd6d-48b9-b19a-4d6f1cb4b2bc" containerName="revoke-keys"
Oct 30 21:17:45.936302 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:45.936235    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e635c82-bd6d-48b9-b19a-4d6f1cb4b2bc" containerName="revoke-keys"
Oct 30 21:17:45.936302 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:45.936261    5713 cpu_manager.go:414] "containerMap: removing container" podUID="1cc1f811-c149-4457-80f1-1a55621d73b4" containerName="main"
Oct 30 21:17:45.936302 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:45.936266    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cc1f811-c149-4457-80f1-1a55621d73b4" containerName="main"
Oct 30 21:17:45.936457 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:45.936372    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e635c82-bd6d-48b9-b19a-4d6f1cb4b2bc" containerName="revoke-keys"
Oct 30 21:17:45.936457 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:45.936381    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cc1f811-c149-4457-80f1-1a55621d73b4" containerName="main"
Oct 30 21:17:45.940110 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:45.940082    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-781058-dlclg"
Oct 30 21:17:45.997943 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:45.997907    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-781058-dlclg"]
Oct 30 21:17:46.042366 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:46.042340    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:17:46.042670 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:46.042653    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:17:46.042780 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:46.042688    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:17:46.042780 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:46.042686    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c710850-22af-4868-863b-99b41649b9b7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-781058-dlclg\" (UID: \"2c710850-22af-4868-863b-99b41649b9b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-781058-dlclg"
Oct 30 21:17:46.042780 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:46.042712    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:17:46.042972 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:46.042916    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c710850-22af-4868-863b-99b41649b9b7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-781058-dlclg\" (UID: \"2c710850-22af-4868-863b-99b41649b9b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-781058-dlclg"
Oct 30 21:17:46.043032 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:46.042973    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qppzz\" (UniqueName: \"kubernetes.io/projected/2c710850-22af-4868-863b-99b41649b9b7-kube-api-access-qppzz\") pod \"maas-api-revoke-keys-e2e-worker-w0-781058-dlclg\" (UID: \"2c710850-22af-4868-863b-99b41649b9b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-781058-dlclg"
Oct 30 21:17:46.144286 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:46.144248    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c710850-22af-4868-863b-99b41649b9b7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-781058-dlclg\" (UID: \"2c710850-22af-4868-863b-99b41649b9b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-781058-dlclg"
Oct 30 21:17:46.144576 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:46.144352    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c710850-22af-4868-863b-99b41649b9b7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-781058-dlclg\" (UID: \"2c710850-22af-4868-863b-99b41649b9b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-781058-dlclg"
Oct 30 21:17:46.144576 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:46.144386    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qppzz\" (UniqueName: \"kubernetes.io/projected/2c710850-22af-4868-863b-99b41649b9b7-kube-api-access-qppzz\") pod \"maas-api-revoke-keys-e2e-worker-w0-781058-dlclg\" (UID: \"2c710850-22af-4868-863b-99b41649b9b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-781058-dlclg"
Oct 30 21:17:46.145182 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:46.145159    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c710850-22af-4868-863b-99b41649b9b7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-781058-dlclg\" (UID: \"2c710850-22af-4868-863b-99b41649b9b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-781058-dlclg"
Oct 30 21:17:46.146892 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:46.146866    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c710850-22af-4868-863b-99b41649b9b7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-781058-dlclg\" (UID: \"2c710850-22af-4868-863b-99b41649b9b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-781058-dlclg"
Oct 30 21:17:46.201937 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:46.201854    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qppzz\" (UniqueName: \"kubernetes.io/projected/2c710850-22af-4868-863b-99b41649b9b7-kube-api-access-qppzz\") pod \"maas-api-revoke-keys-e2e-worker-w0-781058-dlclg\" (UID: \"2c710850-22af-4868-863b-99b41649b9b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-781058-dlclg"
Oct 30 21:17:46.253102 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:46.253064    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-781058-dlclg"
Oct 30 21:17:46.632054 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:17:46.632018    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c710850_22af_4868_863b_99b41649b9b7.slice/crio-a9e6ed513b3855ec5998d423901a87fda28c7dece4fa4239336d501cbb1e0a74 WatchSource:0}: Error finding container a9e6ed513b3855ec5998d423901a87fda28c7dece4fa4239336d501cbb1e0a74: Status 404 returned error can't find the container with id a9e6ed513b3855ec5998d423901a87fda28c7dece4fa4239336d501cbb1e0a74
Oct 30 21:17:46.632735 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:46.632705    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-781058-dlclg"]
Oct 30 21:17:47.378905 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:47.378821    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-781058-dlclg" event={"ID":"2c710850-22af-4868-863b-99b41649b9b7","Type":"ContainerStarted","Data":"f14de084b4e59c5b10af5ded67c3cbba9c64401cc2e0d577edf5beec2dcccb49"}
Oct 30 21:17:47.378905 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:47.378879    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-781058-dlclg" event={"ID":"2c710850-22af-4868-863b-99b41649b9b7","Type":"ContainerStarted","Data":"a9e6ed513b3855ec5998d423901a87fda28c7dece4fa4239336d501cbb1e0a74"}
Oct 30 21:17:47.492102 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:47.492038    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-781058-dlclg" podStartSLOduration=2.4920221590000002 podStartE2EDuration="2.492022159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:17:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:47.491380552 +0000 UTC m=+2350.556156954" watchObservedRunningTime="2026-10-30 21:17:47.492022159 +0000 UTC m=+2350.556798557"
Oct 30 21:17:48.388222 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:48.388185    5713 generic.go:356] "Generic (PLEG): container finished" podID="2c710850-22af-4868-863b-99b41649b9b7" containerID="f14de084b4e59c5b10af5ded67c3cbba9c64401cc2e0d577edf5beec2dcccb49" exitCode=0
Oct 30 21:17:48.388656 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:48.388276    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-781058-dlclg" event={"ID":"2c710850-22af-4868-863b-99b41649b9b7","Type":"ContainerDied","Data":"f14de084b4e59c5b10af5ded67c3cbba9c64401cc2e0d577edf5beec2dcccb49"}
Oct 30 21:17:49.543880 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:49.543852    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-781058-dlclg"
Oct 30 21:17:49.591803 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:49.591770    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c710850-22af-4868-863b-99b41649b9b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c710850-22af-4868-863b-99b41649b9b7-maas-api-openshift-ca\") pod \"2c710850-22af-4868-863b-99b41649b9b7\" (UID: \"2c710850-22af-4868-863b-99b41649b9b7\") "
Oct 30 21:17:49.591977 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:49.591894    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c710850-22af-4868-863b-99b41649b9b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c710850-22af-4868-863b-99b41649b9b7-maas-api-xks-ca\") pod \"2c710850-22af-4868-863b-99b41649b9b7\" (UID: \"2c710850-22af-4868-863b-99b41649b9b7\") "
Oct 30 21:17:49.591977 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:49.591929    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c710850-22af-4868-863b-99b41649b9b7-kube-api-access-qppzz\" (UniqueName: \"kubernetes.io/projected/2c710850-22af-4868-863b-99b41649b9b7-kube-api-access-qppzz\") pod \"2c710850-22af-4868-863b-99b41649b9b7\" (UID: \"2c710850-22af-4868-863b-99b41649b9b7\") "
Oct 30 21:17:49.592212 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:49.592186    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c710850-22af-4868-863b-99b41649b9b7-maas-api-openshift-ca" pod "2c710850-22af-4868-863b-99b41649b9b7" (UID: "2c710850-22af-4868-863b-99b41649b9b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:17:49.592564 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:49.592538    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c710850-22af-4868-863b-99b41649b9b7-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:17:49.594183 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:49.594141    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c710850-22af-4868-863b-99b41649b9b7-maas-api-xks-ca" pod "2c710850-22af-4868-863b-99b41649b9b7" (UID: "2c710850-22af-4868-863b-99b41649b9b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:17:49.594302 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:49.594248    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c710850-22af-4868-863b-99b41649b9b7-kube-api-access-qppzz" pod "2c710850-22af-4868-863b-99b41649b9b7" (UID: "2c710850-22af-4868-863b-99b41649b9b7"). InnerVolumeSpecName "kube-api-access-qppzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:17:49.694425 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:49.694377    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c710850-22af-4868-863b-99b41649b9b7-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:17:49.694425 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:49.694418    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qppzz\" (UniqueName: \"kubernetes.io/projected/2c710850-22af-4868-863b-99b41649b9b7-kube-api-access-qppzz\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:17:50.147837 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:50.147776    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk"
Oct 30 21:17:50.200512 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:50.200477    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/237d222b-50be-4b97-b53d-49187434c09b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/237d222b-50be-4b97-b53d-49187434c09b-istio-podinfo\") pod \"237d222b-50be-4b97-b53d-49187434c09b\" (UID: \"237d222b-50be-4b97-b53d-49187434c09b\") "
Oct 30 21:17:50.200731 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:50.200544    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/237d222b-50be-4b97-b53d-49187434c09b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/237d222b-50be-4b97-b53d-49187434c09b-workload-socket\") pod \"237d222b-50be-4b97-b53d-49187434c09b\" (UID: \"237d222b-50be-4b97-b53d-49187434c09b\") "
Oct 30 21:17:50.200731 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:50.200598    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/237d222b-50be-4b97-b53d-49187434c09b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/237d222b-50be-4b97-b53d-49187434c09b-workload-certs\") pod \"237d222b-50be-4b97-b53d-49187434c09b\" (UID: \"237d222b-50be-4b97-b53d-49187434c09b\") "
Oct 30 21:17:50.200731 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:50.200664    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/237d222b-50be-4b97-b53d-49187434c09b-istio-token\" (UniqueName: \"kubernetes.io/projected/237d222b-50be-4b97-b53d-49187434c09b-istio-token\") pod \"237d222b-50be-4b97-b53d-49187434c09b\" (UID: \"237d222b-50be-4b97-b53d-49187434c09b\") "
Oct 30 21:17:50.200731 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:50.200721    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/237d222b-50be-4b97-b53d-49187434c09b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/237d222b-50be-4b97-b53d-49187434c09b-istio-envoy\") pod \"237d222b-50be-4b97-b53d-49187434c09b\" (UID: \"237d222b-50be-4b97-b53d-49187434c09b\") "
Oct 30 21:17:50.201089 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:50.200761    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/237d222b-50be-4b97-b53d-49187434c09b-kube-api-access-zp6ng\" (UniqueName: \"kubernetes.io/projected/237d222b-50be-4b97-b53d-49187434c09b-kube-api-access-zp6ng\") pod \"237d222b-50be-4b97-b53d-49187434c09b\" (UID: \"237d222b-50be-4b97-b53d-49187434c09b\") "
Oct 30 21:17:50.201089 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:50.200861    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/237d222b-50be-4b97-b53d-49187434c09b-workload-socket" pod "237d222b-50be-4b97-b53d-49187434c09b" (UID: "237d222b-50be-4b97-b53d-49187434c09b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:17:50.201089 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:50.200927    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/237d222b-50be-4b97-b53d-49187434c09b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/237d222b-50be-4b97-b53d-49187434c09b-istiod-ca-cert\") pod \"237d222b-50be-4b97-b53d-49187434c09b\" (UID: \"237d222b-50be-4b97-b53d-49187434c09b\") "
Oct 30 21:17:50.201089 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:50.200973    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/237d222b-50be-4b97-b53d-49187434c09b-workload-certs" pod "237d222b-50be-4b97-b53d-49187434c09b" (UID: "237d222b-50be-4b97-b53d-49187434c09b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:17:50.201089 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:50.201007    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/237d222b-50be-4b97-b53d-49187434c09b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/237d222b-50be-4b97-b53d-49187434c09b-credential-socket\") pod \"237d222b-50be-4b97-b53d-49187434c09b\" (UID: \"237d222b-50be-4b97-b53d-49187434c09b\") "
Oct 30 21:17:50.201089 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:50.201056    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/237d222b-50be-4b97-b53d-49187434c09b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/237d222b-50be-4b97-b53d-49187434c09b-istio-data\") pod \"237d222b-50be-4b97-b53d-49187434c09b\" (UID: \"237d222b-50be-4b97-b53d-49187434c09b\") "
Oct 30 21:17:50.201519 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:50.201235    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/237d222b-50be-4b97-b53d-49187434c09b-credential-socket" pod "237d222b-50be-4b97-b53d-49187434c09b" (UID: "237d222b-50be-4b97-b53d-49187434c09b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:17:50.201654 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:50.201573    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/237d222b-50be-4b97-b53d-49187434c09b-istio-data" pod "237d222b-50be-4b97-b53d-49187434c09b" (UID: "237d222b-50be-4b97-b53d-49187434c09b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:17:50.201654 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:50.201587    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/237d222b-50be-4b97-b53d-49187434c09b-istiod-ca-cert" pod "237d222b-50be-4b97-b53d-49187434c09b" (UID: "237d222b-50be-4b97-b53d-49187434c09b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:17:50.201816 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:50.201720    5713 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/237d222b-50be-4b97-b53d-49187434c09b-istiod-ca-cert\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:17:50.201816 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:50.201744    5713 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/237d222b-50be-4b97-b53d-49187434c09b-credential-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:17:50.201816 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:50.201759    5713 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/237d222b-50be-4b97-b53d-49187434c09b-istio-data\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:17:50.201816 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:50.201773    5713 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/237d222b-50be-4b97-b53d-49187434c09b-workload-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:17:50.201816 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:50.201804    5713 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/237d222b-50be-4b97-b53d-49187434c09b-workload-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:17:50.203196 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:50.203167    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/237d222b-50be-4b97-b53d-49187434c09b-istio-podinfo" pod "237d222b-50be-4b97-b53d-49187434c09b" (UID: "237d222b-50be-4b97-b53d-49187434c09b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:17:50.203323 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:50.203202    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/237d222b-50be-4b97-b53d-49187434c09b-istio-token" pod "237d222b-50be-4b97-b53d-49187434c09b" (UID: "237d222b-50be-4b97-b53d-49187434c09b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:17:50.203605 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:50.203575    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/237d222b-50be-4b97-b53d-49187434c09b-kube-api-access-zp6ng" pod "237d222b-50be-4b97-b53d-49187434c09b" (UID: "237d222b-50be-4b97-b53d-49187434c09b"). InnerVolumeSpecName "kube-api-access-zp6ng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:17:50.203667 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:50.203596    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/237d222b-50be-4b97-b53d-49187434c09b-istio-envoy" pod "237d222b-50be-4b97-b53d-49187434c09b" (UID: "237d222b-50be-4b97-b53d-49187434c09b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:17:50.303904 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:50.303782    5713 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/237d222b-50be-4b97-b53d-49187434c09b-istio-podinfo\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:17:50.303904 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:50.303854    5713 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/237d222b-50be-4b97-b53d-49187434c09b-istio-token\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:17:50.303904 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:50.303870    5713 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/237d222b-50be-4b97-b53d-49187434c09b-istio-envoy\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:17:50.303904 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:50.303885    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zp6ng\" (UniqueName: \"kubernetes.io/projected/237d222b-50be-4b97-b53d-49187434c09b-kube-api-access-zp6ng\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:17:50.393200 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:50.393125    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-781058/distinct-2-w0-781058-kserve-7f6c654757-ppmhm"]
Oct 30 21:17:50.406983 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:50.406958    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-781058-dlclg"
Oct 30 21:17:50.406983 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:50.406968    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-781058-dlclg" event={"ID":"2c710850-22af-4868-863b-99b41649b9b7","Type":"ContainerDied","Data":"a9e6ed513b3855ec5998d423901a87fda28c7dece4fa4239336d501cbb1e0a74"}
Oct 30 21:17:50.407244 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:50.407019    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9e6ed513b3855ec5998d423901a87fda28c7dece4fa4239336d501cbb1e0a74"
Oct 30 21:17:50.407244 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:50.407037    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-781058/premium-w0-781058-kserve-7f5c8964c9-z5fmk"]
Oct 30 21:17:50.409522 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:50.409488    5713 generic.go:356] "Generic (PLEG): container finished" podID="237d222b-50be-4b97-b53d-49187434c09b" containerID="332957b8ee59b70dc1edf253b8a8fe242676318b11019866f8f09563dbb41904" exitCode=0
Oct 30 21:17:50.409643 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:50.409564    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk" event={"ID":"237d222b-50be-4b97-b53d-49187434c09b","Type":"ContainerDied","Data":"332957b8ee59b70dc1edf253b8a8fe242676318b11019866f8f09563dbb41904"}
Oct 30 21:17:50.409643 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:50.409606    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk" event={"ID":"237d222b-50be-4b97-b53d-49187434c09b","Type":"ContainerDied","Data":"e7c993d7d4761ad0a18fcf483bedb6a62ad9701ddc15eaa9afed646bbe024c72"}
Oct 30 21:17:50.409643 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:50.409635    5713 scope.go:122] "RemoveContainer" containerID="332957b8ee59b70dc1edf253b8a8fe242676318b11019866f8f09563dbb41904"
Oct 30 21:17:50.409852 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:50.409708    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk"
Oct 30 21:17:50.419500 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:50.419478    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-781058/simulator-w0-781058-kserve-55b954cb4b-4k7z6"]
Oct 30 21:17:50.426024 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:50.426003    5713 scope.go:122] "RemoveContainer" containerID="05163563985b8d23bde1a51105a5b2d6ba5a9dd9196c54c6ca0a059993cd8339"
Oct 30 21:17:50.436488 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:50.436469    5713 scope.go:122] "RemoveContainer" containerID="332957b8ee59b70dc1edf253b8a8fe242676318b11019866f8f09563dbb41904"
Oct 30 21:17:50.436833 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:17:50.436808    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"332957b8ee59b70dc1edf253b8a8fe242676318b11019866f8f09563dbb41904\": container with ID starting with 332957b8ee59b70dc1edf253b8a8fe242676318b11019866f8f09563dbb41904 not found: ID does not exist" containerID="332957b8ee59b70dc1edf253b8a8fe242676318b11019866f8f09563dbb41904"
Oct 30 21:17:50.436909 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:50.436845    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"332957b8ee59b70dc1edf253b8a8fe242676318b11019866f8f09563dbb41904"} err="failed to get container status \"332957b8ee59b70dc1edf253b8a8fe242676318b11019866f8f09563dbb41904\": rpc error: code = NotFound desc = could not find container \"332957b8ee59b70dc1edf253b8a8fe242676318b11019866f8f09563dbb41904\": container with ID starting with 332957b8ee59b70dc1edf253b8a8fe242676318b11019866f8f09563dbb41904 not found: ID does not exist"
Oct 30 21:17:50.436909 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:50.436870    5713 scope.go:122] "RemoveContainer" containerID="05163563985b8d23bde1a51105a5b2d6ba5a9dd9196c54c6ca0a059993cd8339"
Oct 30 21:17:50.437169 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:17:50.437151    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05163563985b8d23bde1a51105a5b2d6ba5a9dd9196c54c6ca0a059993cd8339\": container with ID starting with 05163563985b8d23bde1a51105a5b2d6ba5a9dd9196c54c6ca0a059993cd8339 not found: ID does not exist" containerID="05163563985b8d23bde1a51105a5b2d6ba5a9dd9196c54c6ca0a059993cd8339"
Oct 30 21:17:50.437237 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:50.437174    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05163563985b8d23bde1a51105a5b2d6ba5a9dd9196c54c6ca0a059993cd8339"} err="failed to get container status \"05163563985b8d23bde1a51105a5b2d6ba5a9dd9196c54c6ca0a059993cd8339\": rpc error: code = NotFound desc = could not find container \"05163563985b8d23bde1a51105a5b2d6ba5a9dd9196c54c6ca0a059993cd8339\": container with ID starting with 05163563985b8d23bde1a51105a5b2d6ba5a9dd9196c54c6ca0a059993cd8339 not found: ID does not exist"
Oct 30 21:17:50.622164 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:50.622062    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk"]
Oct 30 21:17:50.625450 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:50.625423    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-cb6404-openshift-default-747ccd9449-jtrqk"]
Oct 30 21:17:51.064231 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:51.064203    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="237d222b-50be-4b97-b53d-49187434c09b" path="/var/lib/kubelet/pods/237d222b-50be-4b97-b53d-49187434c09b/volumes"
Oct 30 21:17:54.461128 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:17:54.461092    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:17:57.656662 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:17:57.656620    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-781058-6d867c875-285pt"]
Oct 30 21:18:00.092814 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:00.089875    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-781058/simulator-w0-781058-kserve-55b954cb4b-4k7z6" podUID="92c43343-2820-4097-b829-9aae96b670de" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:18:00.186816 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:00.185865    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-781058/premium-w0-781058-kserve-7f5c8964c9-z5fmk" podUID="57626f4f-844e-4ec9-ab89-359d015923af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 30 21:18:02.658909 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:02.658856    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-781058-6d867c875-285pt" podUID="e725a39a-4b17-4c3c-9166-e82091523ae5" containerName="payload-pre-processing" containerID="cri-o://22165e39e06ba7f0d34899a19dee4963c66747c782ba94213f5f9b922f724d34" gracePeriod=25
Oct 30 21:18:04.344367 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:04.344269    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-103ce74c-hcszl"]
Oct 30 21:18:04.345178 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:04.345155    5713 cpu_manager.go:414] "containerMap: removing container" podUID="2c710850-22af-4868-863b-99b41649b9b7" containerName="revoke-keys"
Oct 30 21:18:04.345304 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:04.345179    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c710850-22af-4868-863b-99b41649b9b7" containerName="revoke-keys"
Oct 30 21:18:04.345304 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:04.345219    5713 cpu_manager.go:414] "containerMap: removing container" podUID="237d222b-50be-4b97-b53d-49187434c09b" containerName="istio-proxy"
Oct 30 21:18:04.345304 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:04.345227    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="237d222b-50be-4b97-b53d-49187434c09b" containerName="istio-proxy"
Oct 30 21:18:04.345468 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:04.345453    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c710850-22af-4868-863b-99b41649b9b7" containerName="revoke-keys"
Oct 30 21:18:04.345524 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:04.345471    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="237d222b-50be-4b97-b53d-49187434c09b" containerName="istio-proxy"
Oct 30 21:18:04.345524 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:04.345487    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="237d222b-50be-4b97-b53d-49187434c09b" containerName="istio-proxy"
Oct 30 21:18:04.348000 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:04.347976    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-103ce74c-hcszl"
Oct 30 21:18:04.405335 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:04.405286    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a34d3155-xkmp5"]
Oct 30 21:18:04.407054 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:04.406548    5713 cpu_manager.go:414] "containerMap: removing container" podUID="237d222b-50be-4b97-b53d-49187434c09b" containerName="istio-proxy"
Oct 30 21:18:04.407054 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:04.406579    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="237d222b-50be-4b97-b53d-49187434c09b" containerName="istio-proxy"
Oct 30 21:18:04.410045 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:04.410019    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a34d3155-xkmp5"
Oct 30 21:18:04.448869 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:04.448685    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:18:04.448869 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:04.448702    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:18:04.448869 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:04.448831    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:18:04.451169 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:04.451144    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:18:04.452811 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:04.452772    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-103ce74c-hcszl"]
Oct 30 21:18:04.472130 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:04.472101    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a34d3155-xkmp5"]
Oct 30 21:18:04.495638 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:04.495607    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ddb1527-fa02-4407-a861-5e850c65f7ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-103ce74c-hcszl\" (UID: \"2ddb1527-fa02-4407-a861-5e850c65f7ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-103ce74c-hcszl"
Oct 30 21:18:04.495846 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:04.495659    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb8n7\" (UniqueName: \"kubernetes.io/projected/2ddb1527-fa02-4407-a861-5e850c65f7ae-kube-api-access-xb8n7\") pod \"maas-api-revoke-sub-103ce74c-hcszl\" (UID: \"2ddb1527-fa02-4407-a861-5e850c65f7ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-103ce74c-hcszl"
Oct 30 21:18:04.495908 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:04.495850    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ca575a1-4c51-4967-9798-fc18d5a7752d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a34d3155-xkmp5\" (UID: \"4ca575a1-4c51-4967-9798-fc18d5a7752d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a34d3155-xkmp5"
Oct 30 21:18:04.495964 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:04.495907    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jmv7\" (UniqueName: \"kubernetes.io/projected/4ca575a1-4c51-4967-9798-fc18d5a7752d-kube-api-access-6jmv7\") pod \"maas-api-revoke-sub-a34d3155-xkmp5\" (UID: \"4ca575a1-4c51-4967-9798-fc18d5a7752d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a34d3155-xkmp5"
Oct 30 21:18:04.496003 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:04.495961    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ddb1527-fa02-4407-a861-5e850c65f7ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-103ce74c-hcszl\" (UID: \"2ddb1527-fa02-4407-a861-5e850c65f7ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-103ce74c-hcszl"
Oct 30 21:18:04.496053 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:04.496043    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ca575a1-4c51-4967-9798-fc18d5a7752d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a34d3155-xkmp5\" (UID: \"4ca575a1-4c51-4967-9798-fc18d5a7752d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a34d3155-xkmp5"
Oct 30 21:18:04.597295 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:04.597197    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ca575a1-4c51-4967-9798-fc18d5a7752d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a34d3155-xkmp5\" (UID: \"4ca575a1-4c51-4967-9798-fc18d5a7752d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a34d3155-xkmp5"
Oct 30 21:18:04.597295 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:04.597266    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jmv7\" (UniqueName: \"kubernetes.io/projected/4ca575a1-4c51-4967-9798-fc18d5a7752d-kube-api-access-6jmv7\") pod \"maas-api-revoke-sub-a34d3155-xkmp5\" (UID: \"4ca575a1-4c51-4967-9798-fc18d5a7752d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a34d3155-xkmp5"
Oct 30 21:18:04.597295 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:04.597294    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ddb1527-fa02-4407-a861-5e850c65f7ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-103ce74c-hcszl\" (UID: \"2ddb1527-fa02-4407-a861-5e850c65f7ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-103ce74c-hcszl"
Oct 30 21:18:04.597596 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:04.597379    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ca575a1-4c51-4967-9798-fc18d5a7752d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a34d3155-xkmp5\" (UID: \"4ca575a1-4c51-4967-9798-fc18d5a7752d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a34d3155-xkmp5"
Oct 30 21:18:04.597596 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:04.597496    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ddb1527-fa02-4407-a861-5e850c65f7ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-103ce74c-hcszl\" (UID: \"2ddb1527-fa02-4407-a861-5e850c65f7ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-103ce74c-hcszl"
Oct 30 21:18:04.597596 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:04.597543    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb8n7\" (UniqueName: \"kubernetes.io/projected/2ddb1527-fa02-4407-a861-5e850c65f7ae-kube-api-access-xb8n7\") pod \"maas-api-revoke-sub-103ce74c-hcszl\" (UID: \"2ddb1527-fa02-4407-a861-5e850c65f7ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-103ce74c-hcszl"
Oct 30 21:18:04.598131 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:04.598104    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ca575a1-4c51-4967-9798-fc18d5a7752d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a34d3155-xkmp5\" (UID: \"4ca575a1-4c51-4967-9798-fc18d5a7752d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a34d3155-xkmp5"
Oct 30 21:18:04.598541 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:04.598521    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ddb1527-fa02-4407-a861-5e850c65f7ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-103ce74c-hcszl\" (UID: \"2ddb1527-fa02-4407-a861-5e850c65f7ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-103ce74c-hcszl"
Oct 30 21:18:04.600237 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:04.600212    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ca575a1-4c51-4967-9798-fc18d5a7752d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a34d3155-xkmp5\" (UID: \"4ca575a1-4c51-4967-9798-fc18d5a7752d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a34d3155-xkmp5"
Oct 30 21:18:04.600561 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:04.600539    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ddb1527-fa02-4407-a861-5e850c65f7ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-103ce74c-hcszl\" (UID: \"2ddb1527-fa02-4407-a861-5e850c65f7ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-103ce74c-hcszl"
Oct 30 21:18:04.654757 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:04.654722    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb8n7\" (UniqueName: \"kubernetes.io/projected/2ddb1527-fa02-4407-a861-5e850c65f7ae-kube-api-access-xb8n7\") pod \"maas-api-revoke-sub-103ce74c-hcszl\" (UID: \"2ddb1527-fa02-4407-a861-5e850c65f7ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-103ce74c-hcszl"
Oct 30 21:18:04.660233 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:04.660206    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jmv7\" (UniqueName: \"kubernetes.io/projected/4ca575a1-4c51-4967-9798-fc18d5a7752d-kube-api-access-6jmv7\") pod \"maas-api-revoke-sub-a34d3155-xkmp5\" (UID: \"4ca575a1-4c51-4967-9798-fc18d5a7752d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a34d3155-xkmp5"
Oct 30 21:18:04.662016 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:04.661993    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-103ce74c-hcszl"
Oct 30 21:18:04.751573 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:04.751437    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a34d3155-xkmp5"
Oct 30 21:18:05.045836 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:05.045808    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-103ce74c-hcszl"]
Oct 30 21:18:05.047632 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:18:05.047603    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ddb1527_fa02_4407_a861_5e850c65f7ae.slice/crio-dcc600a72e05d9485e18d4f46f33ea7d779b324d467d707477d6059725a74109 WatchSource:0}: Error finding container dcc600a72e05d9485e18d4f46f33ea7d779b324d467d707477d6059725a74109: Status 404 returned error can't find the container with id dcc600a72e05d9485e18d4f46f33ea7d779b324d467d707477d6059725a74109
Oct 30 21:18:05.145385 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:05.145354    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a34d3155-xkmp5"]
Oct 30 21:18:05.146434 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:18:05.146397    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ca575a1_4c51_4967_9798_fc18d5a7752d.slice/crio-7a3c0d3ca57b1ca056024c2f19daedddeec6f21f21e2b75da5ec09fe68534d73 WatchSource:0}: Error finding container 7a3c0d3ca57b1ca056024c2f19daedddeec6f21f21e2b75da5ec09fe68534d73: Status 404 returned error can't find the container with id 7a3c0d3ca57b1ca056024c2f19daedddeec6f21f21e2b75da5ec09fe68534d73
Oct 30 21:18:05.418611 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:05.418509    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-781058/distinct-2-w0-781058-kserve-7f6c654757-ppmhm" podUID="5e818303-0e9f-4ba0-a635-b80b92c9e36d" containerName="main" containerID="cri-o://d274c2e68c23dcac393ac0a2d3e75375c212cda703367c93087c24fe7a7708b3" gracePeriod=45
Oct 30 21:18:05.432418 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:05.432381    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-781058/premium-w0-781058-kserve-7f5c8964c9-z5fmk" podUID="57626f4f-844e-4ec9-ab89-359d015923af" containerName="main" containerID="cri-o://c0ef753417cc6374a5d6ff21837df58e88d0c1db900d40b9a68655d8266088b9" gracePeriod=45
Oct 30 21:18:05.442073 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:05.442032    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-781058/simulator-w0-781058-kserve-55b954cb4b-4k7z6" podUID="92c43343-2820-4097-b829-9aae96b670de" containerName="main" containerID="cri-o://50a55ff6235c501e4a11d2ce850bce47b190ac21bd7f8c8e0ddab4e97b439c6c" gracePeriod=45
Oct 30 21:18:05.664234 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:05.664206    5713 generic.go:356] "Generic (PLEG): container finished" podID="92c43343-2820-4097-b829-9aae96b670de" containerID="50a55ff6235c501e4a11d2ce850bce47b190ac21bd7f8c8e0ddab4e97b439c6c" exitCode=0
Oct 30 21:18:05.664362 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:05.664278    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-781058/simulator-w0-781058-kserve-55b954cb4b-4k7z6" event={"ID":"92c43343-2820-4097-b829-9aae96b670de","Type":"ContainerDied","Data":"50a55ff6235c501e4a11d2ce850bce47b190ac21bd7f8c8e0ddab4e97b439c6c"}
Oct 30 21:18:05.665951 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:05.665928    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-103ce74c-hcszl" event={"ID":"2ddb1527-fa02-4407-a861-5e850c65f7ae","Type":"ContainerStarted","Data":"2b3a49e1fcd7eaed2980b586981883d2afb42ff68aaff949048e56b567aa6580"}
Oct 30 21:18:05.666043 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:05.665973    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-103ce74c-hcszl" event={"ID":"2ddb1527-fa02-4407-a861-5e850c65f7ae","Type":"ContainerStarted","Data":"dcc600a72e05d9485e18d4f46f33ea7d779b324d467d707477d6059725a74109"}
Oct 30 21:18:05.668652 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:05.668596    5713 generic.go:356] "Generic (PLEG): container finished" podID="5e818303-0e9f-4ba0-a635-b80b92c9e36d" containerID="d274c2e68c23dcac393ac0a2d3e75375c212cda703367c93087c24fe7a7708b3" exitCode=0
Oct 30 21:18:05.668744 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:05.668668    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-781058/distinct-2-w0-781058-kserve-7f6c654757-ppmhm" event={"ID":"5e818303-0e9f-4ba0-a635-b80b92c9e36d","Type":"ContainerDied","Data":"d274c2e68c23dcac393ac0a2d3e75375c212cda703367c93087c24fe7a7708b3"}
Oct 30 21:18:05.671825 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:05.671457    5713 generic.go:356] "Generic (PLEG): container finished" podID="57626f4f-844e-4ec9-ab89-359d015923af" containerID="c0ef753417cc6374a5d6ff21837df58e88d0c1db900d40b9a68655d8266088b9" exitCode=0
Oct 30 21:18:05.671941 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:05.671594    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-781058/premium-w0-781058-kserve-7f5c8964c9-z5fmk" event={"ID":"57626f4f-844e-4ec9-ab89-359d015923af","Type":"ContainerDied","Data":"c0ef753417cc6374a5d6ff21837df58e88d0c1db900d40b9a68655d8266088b9"}
Oct 30 21:18:05.676467 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:05.676442    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a34d3155-xkmp5" event={"ID":"4ca575a1-4c51-4967-9798-fc18d5a7752d","Type":"ContainerStarted","Data":"7a3c0d3ca57b1ca056024c2f19daedddeec6f21f21e2b75da5ec09fe68534d73"}
Oct 30 21:18:05.758530 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:05.758498    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-781058/premium-w0-781058-kserve-7f5c8964c9-z5fmk"
Oct 30 21:18:05.803713 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:05.803620    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-781058/distinct-2-w0-781058-kserve-7f6c654757-ppmhm"
Oct 30 21:18:05.804509 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:05.804454    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-103ce74c-hcszl" podStartSLOduration=1.804435485 podStartE2EDuration="1.804435485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:18:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:18:05.794577025 +0000 UTC m=+2368.859353425" watchObservedRunningTime="2026-10-30 21:18:05.804435485 +0000 UTC m=+2368.869211884"
Oct 30 21:18:05.809103 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:05.809081    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-781058/simulator-w0-781058-kserve-55b954cb4b-4k7z6"
Oct 30 21:18:05.923362 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:05.923266    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57626f4f-844e-4ec9-ab89-359d015923af-home\" (UniqueName: \"kubernetes.io/empty-dir/57626f4f-844e-4ec9-ab89-359d015923af-home\") pod \"57626f4f-844e-4ec9-ab89-359d015923af\" (UID: \"57626f4f-844e-4ec9-ab89-359d015923af\") "
Oct 30 21:18:05.923362 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:05.923315    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57626f4f-844e-4ec9-ab89-359d015923af-tls-certs\" (UniqueName: \"kubernetes.io/secret/57626f4f-844e-4ec9-ab89-359d015923af-tls-certs\") pod \"57626f4f-844e-4ec9-ab89-359d015923af\" (UID: \"57626f4f-844e-4ec9-ab89-359d015923af\") "
Oct 30 21:18:05.923610 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:05.923424    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92c43343-2820-4097-b829-9aae96b670de-tls-certs\" (UniqueName: \"kubernetes.io/secret/92c43343-2820-4097-b829-9aae96b670de-tls-certs\") pod \"92c43343-2820-4097-b829-9aae96b670de\" (UID: \"92c43343-2820-4097-b829-9aae96b670de\") "
Oct 30 21:18:05.923610 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:05.923479    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e818303-0e9f-4ba0-a635-b80b92c9e36d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e818303-0e9f-4ba0-a635-b80b92c9e36d-dshm\") pod \"5e818303-0e9f-4ba0-a635-b80b92c9e36d\" (UID: \"5e818303-0e9f-4ba0-a635-b80b92c9e36d\") "
Oct 30 21:18:05.923610 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:05.923517    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e818303-0e9f-4ba0-a635-b80b92c9e36d-kube-api-access-krnzf\" (UniqueName: \"kubernetes.io/projected/5e818303-0e9f-4ba0-a635-b80b92c9e36d-kube-api-access-krnzf\") pod \"5e818303-0e9f-4ba0-a635-b80b92c9e36d\" (UID: \"5e818303-0e9f-4ba0-a635-b80b92c9e36d\") "
Oct 30 21:18:05.923610 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:05.923573    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57626f4f-844e-4ec9-ab89-359d015923af-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57626f4f-844e-4ec9-ab89-359d015923af-model-cache\") pod \"57626f4f-844e-4ec9-ab89-359d015923af\" (UID: \"57626f4f-844e-4ec9-ab89-359d015923af\") "
Oct 30 21:18:05.923610 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:05.923607    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e818303-0e9f-4ba0-a635-b80b92c9e36d-home\" (UniqueName: \"kubernetes.io/empty-dir/5e818303-0e9f-4ba0-a635-b80b92c9e36d-home\") pod \"5e818303-0e9f-4ba0-a635-b80b92c9e36d\" (UID: \"5e818303-0e9f-4ba0-a635-b80b92c9e36d\") "
Oct 30 21:18:05.923931 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:05.923638    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57626f4f-844e-4ec9-ab89-359d015923af-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57626f4f-844e-4ec9-ab89-359d015923af-tmp-dir\") pod \"57626f4f-844e-4ec9-ab89-359d015923af\" (UID: \"57626f4f-844e-4ec9-ab89-359d015923af\") "
Oct 30 21:18:05.923931 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:05.923692    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57626f4f-844e-4ec9-ab89-359d015923af-dshm\" (UniqueName: \"kubernetes.io/empty-dir/57626f4f-844e-4ec9-ab89-359d015923af-dshm\") pod \"57626f4f-844e-4ec9-ab89-359d015923af\" (UID: \"57626f4f-844e-4ec9-ab89-359d015923af\") "
Oct 30 21:18:05.923931 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:05.923728    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e818303-0e9f-4ba0-a635-b80b92c9e36d-tls-certs\" (UniqueName: \"kubernetes.io/secret/5e818303-0e9f-4ba0-a635-b80b92c9e36d-tls-certs\") pod \"5e818303-0e9f-4ba0-a635-b80b92c9e36d\" (UID: \"5e818303-0e9f-4ba0-a635-b80b92c9e36d\") "
Oct 30 21:18:05.923931 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:05.923755    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92c43343-2820-4097-b829-9aae96b670de-home\" (UniqueName: \"kubernetes.io/empty-dir/92c43343-2820-4097-b829-9aae96b670de-home\") pod \"92c43343-2820-4097-b829-9aae96b670de\" (UID: \"92c43343-2820-4097-b829-9aae96b670de\") "
Oct 30 21:18:05.923931 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:05.923843    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e818303-0e9f-4ba0-a635-b80b92c9e36d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e818303-0e9f-4ba0-a635-b80b92c9e36d-model-cache\") pod \"5e818303-0e9f-4ba0-a635-b80b92c9e36d\" (UID: \"5e818303-0e9f-4ba0-a635-b80b92c9e36d\") "
Oct 30 21:18:05.923931 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:05.923885    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92c43343-2820-4097-b829-9aae96b670de-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92c43343-2820-4097-b829-9aae96b670de-tmp-dir\") pod \"92c43343-2820-4097-b829-9aae96b670de\" (UID: \"92c43343-2820-4097-b829-9aae96b670de\") "
Oct 30 21:18:05.924291 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:05.923965    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e818303-0e9f-4ba0-a635-b80b92c9e36d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e818303-0e9f-4ba0-a635-b80b92c9e36d-tmp-dir\") pod \"5e818303-0e9f-4ba0-a635-b80b92c9e36d\" (UID: \"5e818303-0e9f-4ba0-a635-b80b92c9e36d\") "
Oct 30 21:18:05.924291 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:05.923992    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57626f4f-844e-4ec9-ab89-359d015923af-kube-api-access-rxdvh\" (UniqueName: \"kubernetes.io/projected/57626f4f-844e-4ec9-ab89-359d015923af-kube-api-access-rxdvh\") pod \"57626f4f-844e-4ec9-ab89-359d015923af\" (UID: \"57626f4f-844e-4ec9-ab89-359d015923af\") "
Oct 30 21:18:05.924291 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:05.924038    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92c43343-2820-4097-b829-9aae96b670de-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92c43343-2820-4097-b829-9aae96b670de-model-cache\") pod \"92c43343-2820-4097-b829-9aae96b670de\" (UID: \"92c43343-2820-4097-b829-9aae96b670de\") "
Oct 30 21:18:05.924291 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:05.924083    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92c43343-2820-4097-b829-9aae96b670de-kube-api-access-h8kbk\" (UniqueName: \"kubernetes.io/projected/92c43343-2820-4097-b829-9aae96b670de-kube-api-access-h8kbk\") pod \"92c43343-2820-4097-b829-9aae96b670de\" (UID: \"92c43343-2820-4097-b829-9aae96b670de\") "
Oct 30 21:18:05.924291 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:05.924124    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92c43343-2820-4097-b829-9aae96b670de-dshm\" (UniqueName: \"kubernetes.io/empty-dir/92c43343-2820-4097-b829-9aae96b670de-dshm\") pod \"92c43343-2820-4097-b829-9aae96b670de\" (UID: \"92c43343-2820-4097-b829-9aae96b670de\") "
Oct 30 21:18:05.924592 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:05.924492    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57626f4f-844e-4ec9-ab89-359d015923af-model-cache" pod "57626f4f-844e-4ec9-ab89-359d015923af" (UID: "57626f4f-844e-4ec9-ab89-359d015923af"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:18:05.924903 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:05.924878    5713 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57626f4f-844e-4ec9-ab89-359d015923af-model-cache\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:18:05.925128 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:05.925104    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e818303-0e9f-4ba0-a635-b80b92c9e36d-tmp-dir" pod "5e818303-0e9f-4ba0-a635-b80b92c9e36d" (UID: "5e818303-0e9f-4ba0-a635-b80b92c9e36d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:18:05.927621 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:05.927597    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57626f4f-844e-4ec9-ab89-359d015923af-kube-api-access-rxdvh" pod "57626f4f-844e-4ec9-ab89-359d015923af" (UID: "57626f4f-844e-4ec9-ab89-359d015923af"). InnerVolumeSpecName "kube-api-access-rxdvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:18:05.927872 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:05.927850    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92c43343-2820-4097-b829-9aae96b670de-model-cache" pod "92c43343-2820-4097-b829-9aae96b670de" (UID: "92c43343-2820-4097-b829-9aae96b670de"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:18:05.927961 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:05.927904    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e818303-0e9f-4ba0-a635-b80b92c9e36d-home" pod "5e818303-0e9f-4ba0-a635-b80b92c9e36d" (UID: "5e818303-0e9f-4ba0-a635-b80b92c9e36d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:18:05.928259 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:05.928051    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57626f4f-844e-4ec9-ab89-359d015923af-home" pod "57626f4f-844e-4ec9-ab89-359d015923af" (UID: "57626f4f-844e-4ec9-ab89-359d015923af"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:18:05.930169 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:05.929985    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92c43343-2820-4097-b829-9aae96b670de-home" pod "92c43343-2820-4097-b829-9aae96b670de" (UID: "92c43343-2820-4097-b829-9aae96b670de"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:18:05.930296 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:05.930260    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92c43343-2820-4097-b829-9aae96b670de-kube-api-access-h8kbk" pod "92c43343-2820-4097-b829-9aae96b670de" (UID: "92c43343-2820-4097-b829-9aae96b670de"). InnerVolumeSpecName "kube-api-access-h8kbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:18:05.930474 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:05.930422    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e818303-0e9f-4ba0-a635-b80b92c9e36d-model-cache" pod "5e818303-0e9f-4ba0-a635-b80b92c9e36d" (UID: "5e818303-0e9f-4ba0-a635-b80b92c9e36d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:18:05.932908 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:05.932869    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57626f4f-844e-4ec9-ab89-359d015923af-tmp-dir" pod "57626f4f-844e-4ec9-ab89-359d015923af" (UID: "57626f4f-844e-4ec9-ab89-359d015923af"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:18:05.933864 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:05.933721    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e818303-0e9f-4ba0-a635-b80b92c9e36d-dshm" pod "5e818303-0e9f-4ba0-a635-b80b92c9e36d" (UID: "5e818303-0e9f-4ba0-a635-b80b92c9e36d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:18:05.933864 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:05.933740    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92c43343-2820-4097-b829-9aae96b670de-tls-certs" pod "92c43343-2820-4097-b829-9aae96b670de" (UID: "92c43343-2820-4097-b829-9aae96b670de"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:18:05.933864 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:05.933827    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57626f4f-844e-4ec9-ab89-359d015923af-tls-certs" pod "57626f4f-844e-4ec9-ab89-359d015923af" (UID: "57626f4f-844e-4ec9-ab89-359d015923af"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:18:05.934219 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:05.934193    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92c43343-2820-4097-b829-9aae96b670de-tmp-dir" pod "92c43343-2820-4097-b829-9aae96b670de" (UID: "92c43343-2820-4097-b829-9aae96b670de"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:18:05.934996 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:05.934897    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92c43343-2820-4097-b829-9aae96b670de-dshm" pod "92c43343-2820-4097-b829-9aae96b670de" (UID: "92c43343-2820-4097-b829-9aae96b670de"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:18:05.935777 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:05.935750    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57626f4f-844e-4ec9-ab89-359d015923af-dshm" pod "57626f4f-844e-4ec9-ab89-359d015923af" (UID: "57626f4f-844e-4ec9-ab89-359d015923af"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:18:05.935906 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:05.935839    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e818303-0e9f-4ba0-a635-b80b92c9e36d-tls-certs" pod "5e818303-0e9f-4ba0-a635-b80b92c9e36d" (UID: "5e818303-0e9f-4ba0-a635-b80b92c9e36d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:18:05.935906 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:05.935884    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e818303-0e9f-4ba0-a635-b80b92c9e36d-kube-api-access-krnzf" pod "5e818303-0e9f-4ba0-a635-b80b92c9e36d" (UID: "5e818303-0e9f-4ba0-a635-b80b92c9e36d"). InnerVolumeSpecName "kube-api-access-krnzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:18:06.026396 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:06.026360    5713 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e818303-0e9f-4ba0-a635-b80b92c9e36d-model-cache\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:18:06.026396 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:06.026397    5713 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92c43343-2820-4097-b829-9aae96b670de-tmp-dir\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:18:06.026673 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:06.026413    5713 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e818303-0e9f-4ba0-a635-b80b92c9e36d-tmp-dir\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:18:06.026673 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:06.026428    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxdvh\" (UniqueName: \"kubernetes.io/projected/57626f4f-844e-4ec9-ab89-359d015923af-kube-api-access-rxdvh\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:18:06.026673 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:06.026442    5713 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92c43343-2820-4097-b829-9aae96b670de-model-cache\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:18:06.026673 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:06.026457    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8kbk\" (UniqueName: \"kubernetes.io/projected/92c43343-2820-4097-b829-9aae96b670de-kube-api-access-h8kbk\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:18:06.026673 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:06.026470    5713 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92c43343-2820-4097-b829-9aae96b670de-dshm\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:18:06.026673 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:06.026483    5713 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57626f4f-844e-4ec9-ab89-359d015923af-home\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:18:06.026673 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:06.026498    5713 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57626f4f-844e-4ec9-ab89-359d015923af-tls-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:18:06.026673 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:06.026510    5713 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92c43343-2820-4097-b829-9aae96b670de-tls-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:18:06.026673 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:06.026523    5713 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e818303-0e9f-4ba0-a635-b80b92c9e36d-dshm\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:18:06.026673 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:06.026536    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krnzf\" (UniqueName: \"kubernetes.io/projected/5e818303-0e9f-4ba0-a635-b80b92c9e36d-kube-api-access-krnzf\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:18:06.026673 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:06.026549    5713 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e818303-0e9f-4ba0-a635-b80b92c9e36d-home\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:18:06.026673 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:06.026562    5713 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57626f4f-844e-4ec9-ab89-359d015923af-tmp-dir\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:18:06.026673 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:06.026574    5713 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57626f4f-844e-4ec9-ab89-359d015923af-dshm\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:18:06.026673 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:06.026586    5713 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e818303-0e9f-4ba0-a635-b80b92c9e36d-tls-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:18:06.026673 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:06.026599    5713 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92c43343-2820-4097-b829-9aae96b670de-home\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:18:06.686434 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:06.686398    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-781058/simulator-w0-781058-kserve-55b954cb4b-4k7z6"
Oct 30 21:18:06.686434 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:06.686408    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-781058/simulator-w0-781058-kserve-55b954cb4b-4k7z6" event={"ID":"92c43343-2820-4097-b829-9aae96b670de","Type":"ContainerDied","Data":"19eacedf1b007cfbcce22b68742b9023b0f3bfca56abe034c6a377e6127a72ae"}
Oct 30 21:18:06.687042 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:06.686478    5713 scope.go:122] "RemoveContainer" containerID="50a55ff6235c501e4a11d2ce850bce47b190ac21bd7f8c8e0ddab4e97b439c6c"
Oct 30 21:18:06.688296 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:06.688269    5713 generic.go:356] "Generic (PLEG): container finished" podID="2ddb1527-fa02-4407-a861-5e850c65f7ae" containerID="2b3a49e1fcd7eaed2980b586981883d2afb42ff68aaff949048e56b567aa6580" exitCode=0
Oct 30 21:18:06.688421 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:06.688362    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-103ce74c-hcszl" event={"ID":"2ddb1527-fa02-4407-a861-5e850c65f7ae","Type":"ContainerDied","Data":"2b3a49e1fcd7eaed2980b586981883d2afb42ff68aaff949048e56b567aa6580"}
Oct 30 21:18:06.690277 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:06.690217    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-781058/distinct-2-w0-781058-kserve-7f6c654757-ppmhm"
Oct 30 21:18:06.690277 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:06.690229    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-781058/distinct-2-w0-781058-kserve-7f6c654757-ppmhm" event={"ID":"5e818303-0e9f-4ba0-a635-b80b92c9e36d","Type":"ContainerDied","Data":"9ae3ab343b880d6ca1d77d18eb1e1fc6f0c7995109fae2526ee365a334fc5d5a"}
Oct 30 21:18:06.691964 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:06.691941    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-781058/premium-w0-781058-kserve-7f5c8964c9-z5fmk"
Oct 30 21:18:06.692102 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:06.691976    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-781058/premium-w0-781058-kserve-7f5c8964c9-z5fmk" event={"ID":"57626f4f-844e-4ec9-ab89-359d015923af","Type":"ContainerDied","Data":"6b16125500de29985750c17c820341af752b3999a467dc864fb84845849fcfeb"}
Oct 30 21:18:06.693964 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:06.693937    5713 generic.go:356] "Generic (PLEG): container finished" podID="4ca575a1-4c51-4967-9798-fc18d5a7752d" containerID="b2909abd83ca144cbdc9fac802b0462261d8d2ca99f0355eff29fe1877da048a" exitCode=0
Oct 30 21:18:06.694071 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:06.694014    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a34d3155-xkmp5" event={"ID":"4ca575a1-4c51-4967-9798-fc18d5a7752d","Type":"ContainerDied","Data":"b2909abd83ca144cbdc9fac802b0462261d8d2ca99f0355eff29fe1877da048a"}
Oct 30 21:18:06.702880 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:06.702854    5713 scope.go:122] "RemoveContainer" containerID="d274c2e68c23dcac393ac0a2d3e75375c212cda703367c93087c24fe7a7708b3"
Oct 30 21:18:06.714730 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:06.714705    5713 scope.go:122] "RemoveContainer" containerID="c0ef753417cc6374a5d6ff21837df58e88d0c1db900d40b9a68655d8266088b9"
Oct 30 21:18:06.963337 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:06.963303    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-781058/simulator-w0-781058-kserve-55b954cb4b-4k7z6"]
Oct 30 21:18:06.966564 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:06.966534    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-781058/simulator-w0-781058-kserve-55b954cb4b-4k7z6"]
Oct 30 21:18:07.063838 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:07.063806    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92c43343-2820-4097-b829-9aae96b670de" path="/var/lib/kubelet/pods/92c43343-2820-4097-b829-9aae96b670de/volumes"
Oct 30 21:18:07.186769 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:07.186736    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-781058/distinct-2-w0-781058-kserve-7f6c654757-ppmhm"]
Oct 30 21:18:07.191780 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:07.191750    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-781058/distinct-2-w0-781058-kserve-7f6c654757-ppmhm"]
Oct 30 21:18:07.296956 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:07.296865    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-781058/premium-w0-781058-kserve-7f5c8964c9-z5fmk"]
Oct 30 21:18:07.300465 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:07.300437    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-781058/premium-w0-781058-kserve-7f5c8964c9-z5fmk"]
Oct 30 21:18:07.877646 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:07.877616    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-103ce74c-hcszl"
Oct 30 21:18:07.917415 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:07.917382    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a34d3155-xkmp5"
Oct 30 21:18:07.947622 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:07.947591    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ddb1527-fa02-4407-a861-5e850c65f7ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ddb1527-fa02-4407-a861-5e850c65f7ae-maas-api-xks-ca\") pod \"2ddb1527-fa02-4407-a861-5e850c65f7ae\" (UID: \"2ddb1527-fa02-4407-a861-5e850c65f7ae\") "
Oct 30 21:18:07.947854 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:07.947647    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ca575a1-4c51-4967-9798-fc18d5a7752d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ca575a1-4c51-4967-9798-fc18d5a7752d-maas-api-openshift-ca\") pod \"4ca575a1-4c51-4967-9798-fc18d5a7752d\" (UID: \"4ca575a1-4c51-4967-9798-fc18d5a7752d\") "
Oct 30 21:18:07.947854 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:07.947675    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ca575a1-4c51-4967-9798-fc18d5a7752d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ca575a1-4c51-4967-9798-fc18d5a7752d-maas-api-xks-ca\") pod \"4ca575a1-4c51-4967-9798-fc18d5a7752d\" (UID: \"4ca575a1-4c51-4967-9798-fc18d5a7752d\") "
Oct 30 21:18:07.947854 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:07.947774    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ddb1527-fa02-4407-a861-5e850c65f7ae-kube-api-access-xb8n7\" (UniqueName: \"kubernetes.io/projected/2ddb1527-fa02-4407-a861-5e850c65f7ae-kube-api-access-xb8n7\") pod \"2ddb1527-fa02-4407-a861-5e850c65f7ae\" (UID: \"2ddb1527-fa02-4407-a861-5e850c65f7ae\") "
Oct 30 21:18:07.948037 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:07.947888    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ca575a1-4c51-4967-9798-fc18d5a7752d-kube-api-access-6jmv7\" (UniqueName: \"kubernetes.io/projected/4ca575a1-4c51-4967-9798-fc18d5a7752d-kube-api-access-6jmv7\") pod \"4ca575a1-4c51-4967-9798-fc18d5a7752d\" (UID: \"4ca575a1-4c51-4967-9798-fc18d5a7752d\") "
Oct 30 21:18:07.948037 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:07.947940    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ddb1527-fa02-4407-a861-5e850c65f7ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ddb1527-fa02-4407-a861-5e850c65f7ae-maas-api-openshift-ca\") pod \"2ddb1527-fa02-4407-a861-5e850c65f7ae\" (UID: \"2ddb1527-fa02-4407-a861-5e850c65f7ae\") "
Oct 30 21:18:07.948157 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:07.948093    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ca575a1-4c51-4967-9798-fc18d5a7752d-maas-api-openshift-ca" pod "4ca575a1-4c51-4967-9798-fc18d5a7752d" (UID: "4ca575a1-4c51-4967-9798-fc18d5a7752d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:18:07.948411 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:07.948387    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ddb1527-fa02-4407-a861-5e850c65f7ae-maas-api-openshift-ca" pod "2ddb1527-fa02-4407-a861-5e850c65f7ae" (UID: "2ddb1527-fa02-4407-a861-5e850c65f7ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:18:07.948822 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:07.948781    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ddb1527-fa02-4407-a861-5e850c65f7ae-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:18:07.948921 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:07.948829    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ca575a1-4c51-4967-9798-fc18d5a7752d-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:18:07.950749 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:07.950723    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ddb1527-fa02-4407-a861-5e850c65f7ae-maas-api-xks-ca" pod "2ddb1527-fa02-4407-a861-5e850c65f7ae" (UID: "2ddb1527-fa02-4407-a861-5e850c65f7ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:18:07.950865 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:07.950761    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ddb1527-fa02-4407-a861-5e850c65f7ae-kube-api-access-xb8n7" pod "2ddb1527-fa02-4407-a861-5e850c65f7ae" (UID: "2ddb1527-fa02-4407-a861-5e850c65f7ae"). InnerVolumeSpecName "kube-api-access-xb8n7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:18:07.951096 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:07.951074    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ca575a1-4c51-4967-9798-fc18d5a7752d-kube-api-access-6jmv7" pod "4ca575a1-4c51-4967-9798-fc18d5a7752d" (UID: "4ca575a1-4c51-4967-9798-fc18d5a7752d"). InnerVolumeSpecName "kube-api-access-6jmv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:18:07.951185 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:07.951170    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ca575a1-4c51-4967-9798-fc18d5a7752d-maas-api-xks-ca" pod "4ca575a1-4c51-4967-9798-fc18d5a7752d" (UID: "4ca575a1-4c51-4967-9798-fc18d5a7752d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:18:08.050064 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:08.050019    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ddb1527-fa02-4407-a861-5e850c65f7ae-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:18:08.050064 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:08.050057    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ca575a1-4c51-4967-9798-fc18d5a7752d-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:18:08.050285 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:08.050070    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xb8n7\" (UniqueName: \"kubernetes.io/projected/2ddb1527-fa02-4407-a861-5e850c65f7ae-kube-api-access-xb8n7\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:18:08.050285 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:08.050086    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jmv7\" (UniqueName: \"kubernetes.io/projected/4ca575a1-4c51-4967-9798-fc18d5a7752d-kube-api-access-6jmv7\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:18:08.717052 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:08.717024    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a34d3155-xkmp5"
Oct 30 21:18:08.717257 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:08.717060    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a34d3155-xkmp5" event={"ID":"4ca575a1-4c51-4967-9798-fc18d5a7752d","Type":"ContainerDied","Data":"7a3c0d3ca57b1ca056024c2f19daedddeec6f21f21e2b75da5ec09fe68534d73"}
Oct 30 21:18:08.717257 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:08.717119    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a3c0d3ca57b1ca056024c2f19daedddeec6f21f21e2b75da5ec09fe68534d73"
Oct 30 21:18:08.718893 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:08.718866    5713 generic.go:356] "Generic (PLEG): container finished" podID="fbb39bfd-2716-4ded-9e69-6131383afdb9" containerID="ae817ded885d8d06555408698bfad5fe2bb80a8ef99e711f5330eda0f7763afe" exitCode=0
Oct 30 21:18:08.719047 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:08.718901    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x" event={"ID":"fbb39bfd-2716-4ded-9e69-6131383afdb9","Type":"ContainerDied","Data":"ae817ded885d8d06555408698bfad5fe2bb80a8ef99e711f5330eda0f7763afe"}
Oct 30 21:18:08.719364 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:08.719341    5713 scope.go:122] "RemoveContainer" containerID="ae817ded885d8d06555408698bfad5fe2bb80a8ef99e711f5330eda0f7763afe"
Oct 30 21:18:08.720437 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:08.720414    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-103ce74c-hcszl" event={"ID":"2ddb1527-fa02-4407-a861-5e850c65f7ae","Type":"ContainerDied","Data":"dcc600a72e05d9485e18d4f46f33ea7d779b324d467d707477d6059725a74109"}
Oct 30 21:18:08.720533 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:08.720459    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dcc600a72e05d9485e18d4f46f33ea7d779b324d467d707477d6059725a74109"
Oct 30 21:18:08.720533 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:08.720473    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-103ce74c-hcszl"
Oct 30 21:18:08.720995 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:08.720969    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:18:08.721101 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:08.721049    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:18:08.721101 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:08.721093    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:18:09.064382 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:09.064297    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57626f4f-844e-4ec9-ab89-359d015923af" path="/var/lib/kubelet/pods/57626f4f-844e-4ec9-ab89-359d015923af/volumes"
Oct 30 21:18:09.064927 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:09.064899    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e818303-0e9f-4ba0-a635-b80b92c9e36d" path="/var/lib/kubelet/pods/5e818303-0e9f-4ba0-a635-b80b92c9e36d/volumes"
Oct 30 21:18:09.344875 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:09.340940    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x"
Oct 30 21:18:09.731024 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:09.730988    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x" event={"ID":"fbb39bfd-2716-4ded-9e69-6131383afdb9","Type":"ContainerStarted","Data":"4f5c7a7b969f241216c7caa753d8eb44f3df9c8acfe8f66d904e8f018377425d"}
Oct 30 21:18:10.523188 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:10.523147    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x"
Oct 30 21:18:10.524539 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:10.524515    5713 patch_prober.go:29] interesting pod/e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body=
Oct 30 21:18:10.524615 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:10.524565    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x" podUID="fbb39bfd-2716-4ded-9e69-6131383afdb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused"
Oct 30 21:18:10.580240 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:10.580199    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-781058-6d867c875-285pt" podUID="e725a39a-4b17-4c3c-9166-e82091523ae5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:18:11.523395 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:11.523357    5713 patch_prober.go:29] interesting pod/e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body=
Oct 30 21:18:11.523870 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:11.523410    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x" podUID="fbb39bfd-2716-4ded-9e69-6131383afdb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused"
Oct 30 21:18:12.523975 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:12.523936    5713 patch_prober.go:29] interesting pod/e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body=
Oct 30 21:18:12.524486 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:12.524002    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x" podUID="fbb39bfd-2716-4ded-9e69-6131383afdb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused"
Oct 30 21:18:13.523155 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:13.523121    5713 patch_prober.go:29] interesting pod/e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body=
Oct 30 21:18:13.523342 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:13.523176    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x" podUID="fbb39bfd-2716-4ded-9e69-6131383afdb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused"
Oct 30 21:18:14.523405 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:14.523365    5713 patch_prober.go:29] interesting pod/e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body=
Oct 30 21:18:14.523825 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:14.523420    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x" podUID="fbb39bfd-2716-4ded-9e69-6131383afdb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused"
Oct 30 21:18:15.528517 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:15.528480    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x"
Oct 30 21:18:15.528982 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:15.528961    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x"
Oct 30 21:18:15.529625 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:15.529609    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x"
Oct 30 21:18:20.579752 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:20.579706    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-781058-6d867c875-285pt" podUID="e725a39a-4b17-4c3c-9166-e82091523ae5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:18:21.540874 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:21.540840    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2b425d-hdhgj"]
Oct 30 21:18:21.590655 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:21.590617    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2b425d-hdhgj"]
Oct 30 21:18:22.864857 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:22.864818    5713 generic.go:356] "Generic (PLEG): container finished" podID="e725a39a-4b17-4c3c-9166-e82091523ae5" containerID="22165e39e06ba7f0d34899a19dee4963c66747c782ba94213f5f9b922f724d34" exitCode=0
Oct 30 21:18:22.865252 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:22.864852    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-781058-6d867c875-285pt" event={"ID":"e725a39a-4b17-4c3c-9166-e82091523ae5","Type":"ContainerDied","Data":"22165e39e06ba7f0d34899a19dee4963c66747c782ba94213f5f9b922f724d34"}
Oct 30 21:18:22.939164 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:22.939136    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-781058-6d867c875-285pt"
Oct 30 21:18:23.054447 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:23.054350    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e725a39a-4b17-4c3c-9166-e82091523ae5-kube-api-access-47vl5\" (UniqueName: \"kubernetes.io/projected/e725a39a-4b17-4c3c-9166-e82091523ae5-kube-api-access-47vl5\") pod \"e725a39a-4b17-4c3c-9166-e82091523ae5\" (UID: \"e725a39a-4b17-4c3c-9166-e82091523ae5\") "
Oct 30 21:18:23.054447 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:23.054408    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e725a39a-4b17-4c3c-9166-e82091523ae5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e725a39a-4b17-4c3c-9166-e82091523ae5-plugins-config-volume\") pod \"e725a39a-4b17-4c3c-9166-e82091523ae5\" (UID: \"e725a39a-4b17-4c3c-9166-e82091523ae5\") "
Oct 30 21:18:23.055588 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:23.055509    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e725a39a-4b17-4c3c-9166-e82091523ae5-plugins-config-volume" pod "e725a39a-4b17-4c3c-9166-e82091523ae5" (UID: "e725a39a-4b17-4c3c-9166-e82091523ae5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:18:23.076890 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:23.059944    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e725a39a-4b17-4c3c-9166-e82091523ae5-kube-api-access-47vl5" pod "e725a39a-4b17-4c3c-9166-e82091523ae5" (UID: "e725a39a-4b17-4c3c-9166-e82091523ae5"). InnerVolumeSpecName "kube-api-access-47vl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:18:23.077622 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:23.077580    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94effd3f-fc12-433c-aad6-82f48b4ec4fa" path="/var/lib/kubelet/pods/94effd3f-fc12-433c-aad6-82f48b4ec4fa/volumes"
Oct 30 21:18:23.157491 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:23.157344    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47vl5\" (UniqueName: \"kubernetes.io/projected/e725a39a-4b17-4c3c-9166-e82091523ae5-kube-api-access-47vl5\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:18:23.157756 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:23.157735    5713 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e725a39a-4b17-4c3c-9166-e82091523ae5-plugins-config-volume\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:18:23.874746 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:23.874709    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-781058-6d867c875-285pt"
Oct 30 21:18:23.875228 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:23.874749    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-781058-6d867c875-285pt" event={"ID":"e725a39a-4b17-4c3c-9166-e82091523ae5","Type":"ContainerDied","Data":"3d25cd85d76da174d487463a3abab28bcd806c23b6f6b3df7f2ed8e9bcee791f"}
Oct 30 21:18:23.875228 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:23.874828    5713 scope.go:122] "RemoveContainer" containerID="22165e39e06ba7f0d34899a19dee4963c66747c782ba94213f5f9b922f724d34"
Oct 30 21:18:24.041586 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:24.041551    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-781058-6d867c875-285pt"]
Oct 30 21:18:24.090679 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:24.090650    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-781058-6d867c875-285pt"]
Oct 30 21:18:24.459160 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:18:24.459119    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:18:25.063329 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:25.063297    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e725a39a-4b17-4c3c-9166-e82091523ae5" path="/var/lib/kubelet/pods/e725a39a-4b17-4c3c-9166-e82091523ae5/volumes"
Oct 30 21:18:28.566272 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:28.566239    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq"]
Oct 30 21:18:28.566685 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:28.566493    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq" podUID="63ace536-4242-41d2-b1a0-3b10efad3333" containerName="istio-proxy" containerID="cri-o://983ffde06d9cc0123227140305489c5db6ae4b16afed321ad2f93946fa8b52a3" gracePeriod=30
Oct 30 21:18:31.825800 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:31.825750    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f4cd36b-555c2"]
Oct 30 21:18:31.826374 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:31.826359    5713 cpu_manager.go:414] "containerMap: removing container" podUID="92c43343-2820-4097-b829-9aae96b670de" containerName="main"
Oct 30 21:18:31.826422 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:31.826375    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92c43343-2820-4097-b829-9aae96b670de" containerName="main"
Oct 30 21:18:31.826422 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:31.826392    5713 cpu_manager.go:414] "containerMap: removing container" podUID="2ddb1527-fa02-4407-a861-5e850c65f7ae" containerName="revoke-keys"
Oct 30 21:18:31.826422 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:31.826397    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ddb1527-fa02-4407-a861-5e850c65f7ae" containerName="revoke-keys"
Oct 30 21:18:31.826422 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:31.826417    5713 cpu_manager.go:414] "containerMap: removing container" podUID="57626f4f-844e-4ec9-ab89-359d015923af" containerName="main"
Oct 30 21:18:31.826548 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:31.826424    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57626f4f-844e-4ec9-ab89-359d015923af" containerName="main"
Oct 30 21:18:31.826548 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:31.826439    5713 cpu_manager.go:414] "containerMap: removing container" podUID="e725a39a-4b17-4c3c-9166-e82091523ae5" containerName="payload-pre-processing"
Oct 30 21:18:31.826548 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:31.826444    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e725a39a-4b17-4c3c-9166-e82091523ae5" containerName="payload-pre-processing"
Oct 30 21:18:31.826548 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:31.826465    5713 cpu_manager.go:414] "containerMap: removing container" podUID="4ca575a1-4c51-4967-9798-fc18d5a7752d" containerName="revoke-keys"
Oct 30 21:18:31.826548 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:31.826469    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ca575a1-4c51-4967-9798-fc18d5a7752d" containerName="revoke-keys"
Oct 30 21:18:31.826548 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:31.826479    5713 cpu_manager.go:414] "containerMap: removing container" podUID="5e818303-0e9f-4ba0-a635-b80b92c9e36d" containerName="main"
Oct 30 21:18:31.826548 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:31.826484    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e818303-0e9f-4ba0-a635-b80b92c9e36d" containerName="main"
Oct 30 21:18:31.826764 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:31.826557    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ca575a1-4c51-4967-9798-fc18d5a7752d" containerName="revoke-keys"
Oct 30 21:18:31.826764 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:31.826564    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e818303-0e9f-4ba0-a635-b80b92c9e36d" containerName="main"
Oct 30 21:18:31.826764 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:31.826571    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="92c43343-2820-4097-b829-9aae96b670de" containerName="main"
Oct 30 21:18:31.826764 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:31.826578    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ddb1527-fa02-4407-a861-5e850c65f7ae" containerName="revoke-keys"
Oct 30 21:18:31.826764 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:31.826587    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="57626f4f-844e-4ec9-ab89-359d015923af" containerName="main"
Oct 30 21:18:31.826764 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:31.826594    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="e725a39a-4b17-4c3c-9166-e82091523ae5" containerName="payload-pre-processing"
Oct 30 21:18:31.830004 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:31.829987    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f4cd36b-555c2"
Oct 30 21:18:31.878217 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:31.878181    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51fb32f8-14a4-4d70-9cea-051fcb96f989-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f4cd36b-555c2\" (UID: \"51fb32f8-14a4-4d70-9cea-051fcb96f989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f4cd36b-555c2"
Oct 30 21:18:31.878410 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:31.878235    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51fb32f8-14a4-4d70-9cea-051fcb96f989-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f4cd36b-555c2\" (UID: \"51fb32f8-14a4-4d70-9cea-051fcb96f989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f4cd36b-555c2"
Oct 30 21:18:31.878489 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:31.878414    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrmvb\" (UniqueName: \"kubernetes.io/projected/51fb32f8-14a4-4d70-9cea-051fcb96f989-kube-api-access-qrmvb\") pod \"maas-api-revoke-sub-3f4cd36b-555c2\" (UID: \"51fb32f8-14a4-4d70-9cea-051fcb96f989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f4cd36b-555c2"
Oct 30 21:18:31.880281 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:31.880261    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac25069b-rb2gz"]
Oct 30 21:18:31.884482 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:31.884463    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac25069b-rb2gz"
Oct 30 21:18:31.884892 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:31.884871    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f4cd36b-555c2"]
Oct 30 21:18:31.926533 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:31.926496    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:18:31.931143 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:31.931107    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:18:31.931314 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:31.931180    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:18:31.931314 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:31.931223    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:18:31.943093 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:31.943060    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac25069b-rb2gz"]
Oct 30 21:18:31.980243 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:31.980211    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrmvb\" (UniqueName: \"kubernetes.io/projected/51fb32f8-14a4-4d70-9cea-051fcb96f989-kube-api-access-qrmvb\") pod \"maas-api-revoke-sub-3f4cd36b-555c2\" (UID: \"51fb32f8-14a4-4d70-9cea-051fcb96f989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f4cd36b-555c2"
Oct 30 21:18:31.980415 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:31.980323    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40901aeb-4a8b-4e5b-9756-6594247fc30e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac25069b-rb2gz\" (UID: \"40901aeb-4a8b-4e5b-9756-6594247fc30e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac25069b-rb2gz"
Oct 30 21:18:31.980518 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:31.980496    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51fb32f8-14a4-4d70-9cea-051fcb96f989-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f4cd36b-555c2\" (UID: \"51fb32f8-14a4-4d70-9cea-051fcb96f989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f4cd36b-555c2"
Oct 30 21:18:31.980569 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:31.980545    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51fb32f8-14a4-4d70-9cea-051fcb96f989-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f4cd36b-555c2\" (UID: \"51fb32f8-14a4-4d70-9cea-051fcb96f989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f4cd36b-555c2"
Oct 30 21:18:31.980640 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:31.980626    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n84k\" (UniqueName: \"kubernetes.io/projected/40901aeb-4a8b-4e5b-9756-6594247fc30e-kube-api-access-6n84k\") pod \"maas-api-revoke-sub-ac25069b-rb2gz\" (UID: \"40901aeb-4a8b-4e5b-9756-6594247fc30e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac25069b-rb2gz"
Oct 30 21:18:31.980682 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:31.980671    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40901aeb-4a8b-4e5b-9756-6594247fc30e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac25069b-rb2gz\" (UID: \"40901aeb-4a8b-4e5b-9756-6594247fc30e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac25069b-rb2gz"
Oct 30 21:18:31.981365 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:31.981336    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51fb32f8-14a4-4d70-9cea-051fcb96f989-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f4cd36b-555c2\" (UID: \"51fb32f8-14a4-4d70-9cea-051fcb96f989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f4cd36b-555c2"
Oct 30 21:18:31.983229 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:31.983197    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51fb32f8-14a4-4d70-9cea-051fcb96f989-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f4cd36b-555c2\" (UID: \"51fb32f8-14a4-4d70-9cea-051fcb96f989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f4cd36b-555c2"
Oct 30 21:18:32.037119 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:32.037084    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrmvb\" (UniqueName: \"kubernetes.io/projected/51fb32f8-14a4-4d70-9cea-051fcb96f989-kube-api-access-qrmvb\") pod \"maas-api-revoke-sub-3f4cd36b-555c2\" (UID: \"51fb32f8-14a4-4d70-9cea-051fcb96f989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f4cd36b-555c2"
Oct 30 21:18:32.082534 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:32.082442    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40901aeb-4a8b-4e5b-9756-6594247fc30e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac25069b-rb2gz\" (UID: \"40901aeb-4a8b-4e5b-9756-6594247fc30e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac25069b-rb2gz"
Oct 30 21:18:32.082733 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:32.082705    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n84k\" (UniqueName: \"kubernetes.io/projected/40901aeb-4a8b-4e5b-9756-6594247fc30e-kube-api-access-6n84k\") pod \"maas-api-revoke-sub-ac25069b-rb2gz\" (UID: \"40901aeb-4a8b-4e5b-9756-6594247fc30e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac25069b-rb2gz"
Oct 30 21:18:32.082954 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:32.082760    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40901aeb-4a8b-4e5b-9756-6594247fc30e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac25069b-rb2gz\" (UID: \"40901aeb-4a8b-4e5b-9756-6594247fc30e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac25069b-rb2gz"
Oct 30 21:18:32.083377 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:32.083351    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40901aeb-4a8b-4e5b-9756-6594247fc30e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac25069b-rb2gz\" (UID: \"40901aeb-4a8b-4e5b-9756-6594247fc30e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac25069b-rb2gz"
Oct 30 21:18:32.085104 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:32.085085    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40901aeb-4a8b-4e5b-9756-6594247fc30e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac25069b-rb2gz\" (UID: \"40901aeb-4a8b-4e5b-9756-6594247fc30e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac25069b-rb2gz"
Oct 30 21:18:32.139223 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:32.139190    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n84k\" (UniqueName: \"kubernetes.io/projected/40901aeb-4a8b-4e5b-9756-6594247fc30e-kube-api-access-6n84k\") pod \"maas-api-revoke-sub-ac25069b-rb2gz\" (UID: \"40901aeb-4a8b-4e5b-9756-6594247fc30e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac25069b-rb2gz"
Oct 30 21:18:32.144036 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:32.144011    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f4cd36b-555c2"
Oct 30 21:18:32.199517 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:32.198295    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac25069b-rb2gz"
Oct 30 21:18:32.523639 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:18:32.523610    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51fb32f8_14a4_4d70_9cea_051fcb96f989.slice/crio-eae45d0f1138ff0901845c889ce93735e1fde8da0fc1f3ad6ecaa7fe5d9caefd WatchSource:0}: Error finding container eae45d0f1138ff0901845c889ce93735e1fde8da0fc1f3ad6ecaa7fe5d9caefd: Status 404 returned error can't find the container with id eae45d0f1138ff0901845c889ce93735e1fde8da0fc1f3ad6ecaa7fe5d9caefd
Oct 30 21:18:32.523929 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:32.523905    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f4cd36b-555c2"]
Oct 30 21:18:32.625348 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:32.625312    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac25069b-rb2gz"]
Oct 30 21:18:32.625963 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:18:32.625939    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40901aeb_4a8b_4e5b_9756_6594247fc30e.slice/crio-f831f4138c11ff1024ddc6f377383c007198227defb20c808c7ff9a7abf7697e WatchSource:0}: Error finding container f831f4138c11ff1024ddc6f377383c007198227defb20c808c7ff9a7abf7697e: Status 404 returned error can't find the container with id f831f4138c11ff1024ddc6f377383c007198227defb20c808c7ff9a7abf7697e
Oct 30 21:18:32.962118 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:32.962080    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac25069b-rb2gz" event={"ID":"40901aeb-4a8b-4e5b-9756-6594247fc30e","Type":"ContainerStarted","Data":"f831f4138c11ff1024ddc6f377383c007198227defb20c808c7ff9a7abf7697e"}
Oct 30 21:18:32.963930 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:32.963908    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f4cd36b-555c2" event={"ID":"51fb32f8-14a4-4d70-9cea-051fcb96f989","Type":"ContainerStarted","Data":"346d32966dbefe5e0cbc3530b88e8c0c96ac817ceba60b93c76d2cba1bbace0b"}
Oct 30 21:18:32.964047 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:32.963945    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f4cd36b-555c2" event={"ID":"51fb32f8-14a4-4d70-9cea-051fcb96f989","Type":"ContainerStarted","Data":"eae45d0f1138ff0901845c889ce93735e1fde8da0fc1f3ad6ecaa7fe5d9caefd"}
Oct 30 21:18:33.078820 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:33.078737    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f4cd36b-555c2" podStartSLOduration=2.078722454 podStartE2EDuration="2.078722454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:18:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:18:33.077769575 +0000 UTC m=+2396.142545974" watchObservedRunningTime="2026-10-30 21:18:33.078722454 +0000 UTC m=+2396.143498852"
Oct 30 21:18:33.837186 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:33.837161    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq"
Oct 30 21:18:33.904670 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:33.904633    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63ace536-4242-41d2-b1a0-3b10efad3333-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63ace536-4242-41d2-b1a0-3b10efad3333-istiod-ca-cert\") pod \"63ace536-4242-41d2-b1a0-3b10efad3333\" (UID: \"63ace536-4242-41d2-b1a0-3b10efad3333\") "
Oct 30 21:18:33.904670 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:33.904676    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63ace536-4242-41d2-b1a0-3b10efad3333-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63ace536-4242-41d2-b1a0-3b10efad3333-workload-certs\") pod \"63ace536-4242-41d2-b1a0-3b10efad3333\" (UID: \"63ace536-4242-41d2-b1a0-3b10efad3333\") "
Oct 30 21:18:33.904999 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:33.904744    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63ace536-4242-41d2-b1a0-3b10efad3333-kube-api-access-f2nwl\" (UniqueName: \"kubernetes.io/projected/63ace536-4242-41d2-b1a0-3b10efad3333-kube-api-access-f2nwl\") pod \"63ace536-4242-41d2-b1a0-3b10efad3333\" (UID: \"63ace536-4242-41d2-b1a0-3b10efad3333\") "
Oct 30 21:18:33.904999 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:33.904822    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/63ace536-4242-41d2-b1a0-3b10efad3333-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63ace536-4242-41d2-b1a0-3b10efad3333-istio-podinfo\") pod \"63ace536-4242-41d2-b1a0-3b10efad3333\" (UID: \"63ace536-4242-41d2-b1a0-3b10efad3333\") "
Oct 30 21:18:33.904999 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:33.904882    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63ace536-4242-41d2-b1a0-3b10efad3333-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63ace536-4242-41d2-b1a0-3b10efad3333-credential-socket\") pod \"63ace536-4242-41d2-b1a0-3b10efad3333\" (UID: \"63ace536-4242-41d2-b1a0-3b10efad3333\") "
Oct 30 21:18:33.904999 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:33.904916    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63ace536-4242-41d2-b1a0-3b10efad3333-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63ace536-4242-41d2-b1a0-3b10efad3333-workload-socket\") pod \"63ace536-4242-41d2-b1a0-3b10efad3333\" (UID: \"63ace536-4242-41d2-b1a0-3b10efad3333\") "
Oct 30 21:18:33.904999 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:33.904996    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63ace536-4242-41d2-b1a0-3b10efad3333-istio-token\" (UniqueName: \"kubernetes.io/projected/63ace536-4242-41d2-b1a0-3b10efad3333-istio-token\") pod \"63ace536-4242-41d2-b1a0-3b10efad3333\" (UID: \"63ace536-4242-41d2-b1a0-3b10efad3333\") "
Oct 30 21:18:33.905278 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:33.905025    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63ace536-4242-41d2-b1a0-3b10efad3333-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63ace536-4242-41d2-b1a0-3b10efad3333-istio-envoy\") pod \"63ace536-4242-41d2-b1a0-3b10efad3333\" (UID: \"63ace536-4242-41d2-b1a0-3b10efad3333\") "
Oct 30 21:18:33.905278 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:33.905066    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63ace536-4242-41d2-b1a0-3b10efad3333-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63ace536-4242-41d2-b1a0-3b10efad3333-istio-data\") pod \"63ace536-4242-41d2-b1a0-3b10efad3333\" (UID: \"63ace536-4242-41d2-b1a0-3b10efad3333\") "
Oct 30 21:18:33.905278 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:33.905149    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63ace536-4242-41d2-b1a0-3b10efad3333-workload-certs" pod "63ace536-4242-41d2-b1a0-3b10efad3333" (UID: "63ace536-4242-41d2-b1a0-3b10efad3333"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:18:33.905278 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:33.905204    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63ace536-4242-41d2-b1a0-3b10efad3333-credential-socket" pod "63ace536-4242-41d2-b1a0-3b10efad3333" (UID: "63ace536-4242-41d2-b1a0-3b10efad3333"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:18:33.905278 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:33.905241    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63ace536-4242-41d2-b1a0-3b10efad3333-istiod-ca-cert" pod "63ace536-4242-41d2-b1a0-3b10efad3333" (UID: "63ace536-4242-41d2-b1a0-3b10efad3333"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:18:33.905551 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:33.905469    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63ace536-4242-41d2-b1a0-3b10efad3333-istio-data" pod "63ace536-4242-41d2-b1a0-3b10efad3333" (UID: "63ace536-4242-41d2-b1a0-3b10efad3333"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:18:33.905551 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:33.905509    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63ace536-4242-41d2-b1a0-3b10efad3333-workload-socket" pod "63ace536-4242-41d2-b1a0-3b10efad3333" (UID: "63ace536-4242-41d2-b1a0-3b10efad3333"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:18:33.905981 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:33.905957    5713 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63ace536-4242-41d2-b1a0-3b10efad3333-istio-data\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:18:33.906104 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:33.905986    5713 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63ace536-4242-41d2-b1a0-3b10efad3333-istiod-ca-cert\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:18:33.906104 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:33.906007    5713 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63ace536-4242-41d2-b1a0-3b10efad3333-workload-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:18:33.906104 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:33.906023    5713 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63ace536-4242-41d2-b1a0-3b10efad3333-credential-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:18:33.906104 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:33.906039    5713 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63ace536-4242-41d2-b1a0-3b10efad3333-workload-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:18:33.907374 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:33.907352    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63ace536-4242-41d2-b1a0-3b10efad3333-istio-token" pod "63ace536-4242-41d2-b1a0-3b10efad3333" (UID: "63ace536-4242-41d2-b1a0-3b10efad3333"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:18:33.907457 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:33.907397    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63ace536-4242-41d2-b1a0-3b10efad3333-kube-api-access-f2nwl" pod "63ace536-4242-41d2-b1a0-3b10efad3333" (UID: "63ace536-4242-41d2-b1a0-3b10efad3333"). InnerVolumeSpecName "kube-api-access-f2nwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:18:33.907888 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:33.907864    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63ace536-4242-41d2-b1a0-3b10efad3333-istio-podinfo" pod "63ace536-4242-41d2-b1a0-3b10efad3333" (UID: "63ace536-4242-41d2-b1a0-3b10efad3333"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:18:33.907888 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:33.907870    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63ace536-4242-41d2-b1a0-3b10efad3333-istio-envoy" pod "63ace536-4242-41d2-b1a0-3b10efad3333" (UID: "63ace536-4242-41d2-b1a0-3b10efad3333"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:18:33.973471 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:33.973431    5713 generic.go:356] "Generic (PLEG): container finished" podID="40901aeb-4a8b-4e5b-9756-6594247fc30e" containerID="f7a39a388676aa241a71f37c4b8390dd39e2568691d2ae4575173bc1a1b5c885" exitCode=0
Oct 30 21:18:33.973920 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:33.973519    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac25069b-rb2gz" event={"ID":"40901aeb-4a8b-4e5b-9756-6594247fc30e","Type":"ContainerDied","Data":"f7a39a388676aa241a71f37c4b8390dd39e2568691d2ae4575173bc1a1b5c885"}
Oct 30 21:18:33.975512 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:33.975486    5713 generic.go:356] "Generic (PLEG): container finished" podID="51fb32f8-14a4-4d70-9cea-051fcb96f989" containerID="346d32966dbefe5e0cbc3530b88e8c0c96ac817ceba60b93c76d2cba1bbace0b" exitCode=0
Oct 30 21:18:33.975662 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:33.975574    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f4cd36b-555c2" event={"ID":"51fb32f8-14a4-4d70-9cea-051fcb96f989","Type":"ContainerDied","Data":"346d32966dbefe5e0cbc3530b88e8c0c96ac817ceba60b93c76d2cba1bbace0b"}
Oct 30 21:18:33.977468 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:33.977445    5713 generic.go:356] "Generic (PLEG): container finished" podID="63ace536-4242-41d2-b1a0-3b10efad3333" containerID="983ffde06d9cc0123227140305489c5db6ae4b16afed321ad2f93946fa8b52a3" exitCode=0
Oct 30 21:18:33.977594 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:33.977488    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq"
Oct 30 21:18:33.977594 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:33.977525    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq" event={"ID":"63ace536-4242-41d2-b1a0-3b10efad3333","Type":"ContainerDied","Data":"983ffde06d9cc0123227140305489c5db6ae4b16afed321ad2f93946fa8b52a3"}
Oct 30 21:18:33.977594 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:33.977576    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq" event={"ID":"63ace536-4242-41d2-b1a0-3b10efad3333","Type":"ContainerDied","Data":"f9a194e253aab9210f3266c1d97a16be2231fec122d0f0ed3b2cc69813c31e38"}
Oct 30 21:18:33.977735 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:33.977606    5713 scope.go:122] "RemoveContainer" containerID="983ffde06d9cc0123227140305489c5db6ae4b16afed321ad2f93946fa8b52a3"
Oct 30 21:18:33.991556 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:33.991530    5713 scope.go:122] "RemoveContainer" containerID="b1dd9a119d562d2f3f9f7ca2671288b163d12504b86500f6e47a34b420812bdf"
Oct 30 21:18:34.004271 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:34.003991    5713 scope.go:122] "RemoveContainer" containerID="983ffde06d9cc0123227140305489c5db6ae4b16afed321ad2f93946fa8b52a3"
Oct 30 21:18:34.004980 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:18:34.004953    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"983ffde06d9cc0123227140305489c5db6ae4b16afed321ad2f93946fa8b52a3\": container with ID starting with 983ffde06d9cc0123227140305489c5db6ae4b16afed321ad2f93946fa8b52a3 not found: ID does not exist" containerID="983ffde06d9cc0123227140305489c5db6ae4b16afed321ad2f93946fa8b52a3"
Oct 30 21:18:34.005064 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:34.004993    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"983ffde06d9cc0123227140305489c5db6ae4b16afed321ad2f93946fa8b52a3"} err="failed to get container status \"983ffde06d9cc0123227140305489c5db6ae4b16afed321ad2f93946fa8b52a3\": rpc error: code = NotFound desc = could not find container \"983ffde06d9cc0123227140305489c5db6ae4b16afed321ad2f93946fa8b52a3\": container with ID starting with 983ffde06d9cc0123227140305489c5db6ae4b16afed321ad2f93946fa8b52a3 not found: ID does not exist"
Oct 30 21:18:34.005064 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:34.005018    5713 scope.go:122] "RemoveContainer" containerID="b1dd9a119d562d2f3f9f7ca2671288b163d12504b86500f6e47a34b420812bdf"
Oct 30 21:18:34.005365 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:18:34.005345    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1dd9a119d562d2f3f9f7ca2671288b163d12504b86500f6e47a34b420812bdf\": container with ID starting with b1dd9a119d562d2f3f9f7ca2671288b163d12504b86500f6e47a34b420812bdf not found: ID does not exist" containerID="b1dd9a119d562d2f3f9f7ca2671288b163d12504b86500f6e47a34b420812bdf"
Oct 30 21:18:34.005420 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:34.005373    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1dd9a119d562d2f3f9f7ca2671288b163d12504b86500f6e47a34b420812bdf"} err="failed to get container status \"b1dd9a119d562d2f3f9f7ca2671288b163d12504b86500f6e47a34b420812bdf\": rpc error: code = NotFound desc = could not find container \"b1dd9a119d562d2f3f9f7ca2671288b163d12504b86500f6e47a34b420812bdf\": container with ID starting with b1dd9a119d562d2f3f9f7ca2671288b163d12504b86500f6e47a34b420812bdf not found: ID does not exist"
Oct 30 21:18:34.007223 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:34.007203    5713 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63ace536-4242-41d2-b1a0-3b10efad3333-istio-podinfo\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:18:34.007223 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:34.007224    5713 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63ace536-4242-41d2-b1a0-3b10efad3333-istio-token\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:18:34.007370 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:34.007234    5713 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63ace536-4242-41d2-b1a0-3b10efad3333-istio-envoy\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:18:34.007370 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:34.007243    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2nwl\" (UniqueName: \"kubernetes.io/projected/63ace536-4242-41d2-b1a0-3b10efad3333-kube-api-access-f2nwl\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:18:34.342768 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:34.342711    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq"]
Oct 30 21:18:34.346184 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:34.346157    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b425d-openshift-default-f547dbcbf-hscpq"]
Oct 30 21:18:35.068930 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:35.068895    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63ace536-4242-41d2-b1a0-3b10efad3333" path="/var/lib/kubelet/pods/63ace536-4242-41d2-b1a0-3b10efad3333/volumes"
Oct 30 21:18:35.178722 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:35.178697    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac25069b-rb2gz"
Oct 30 21:18:35.180762 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:35.180743    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f4cd36b-555c2"
Oct 30 21:18:35.224474 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:35.224443    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51fb32f8-14a4-4d70-9cea-051fcb96f989-kube-api-access-qrmvb\" (UniqueName: \"kubernetes.io/projected/51fb32f8-14a4-4d70-9cea-051fcb96f989-kube-api-access-qrmvb\") pod \"51fb32f8-14a4-4d70-9cea-051fcb96f989\" (UID: \"51fb32f8-14a4-4d70-9cea-051fcb96f989\") "
Oct 30 21:18:35.224644 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:35.224503    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40901aeb-4a8b-4e5b-9756-6594247fc30e-kube-api-access-6n84k\" (UniqueName: \"kubernetes.io/projected/40901aeb-4a8b-4e5b-9756-6594247fc30e-kube-api-access-6n84k\") pod \"40901aeb-4a8b-4e5b-9756-6594247fc30e\" (UID: \"40901aeb-4a8b-4e5b-9756-6594247fc30e\") "
Oct 30 21:18:35.224644 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:35.224615    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51fb32f8-14a4-4d70-9cea-051fcb96f989-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51fb32f8-14a4-4d70-9cea-051fcb96f989-maas-api-xks-ca\") pod \"51fb32f8-14a4-4d70-9cea-051fcb96f989\" (UID: \"51fb32f8-14a4-4d70-9cea-051fcb96f989\") "
Oct 30 21:18:35.224771 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:35.224729    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51fb32f8-14a4-4d70-9cea-051fcb96f989-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51fb32f8-14a4-4d70-9cea-051fcb96f989-maas-api-openshift-ca\") pod \"51fb32f8-14a4-4d70-9cea-051fcb96f989\" (UID: \"51fb32f8-14a4-4d70-9cea-051fcb96f989\") "
Oct 30 21:18:35.224890 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:35.224767    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40901aeb-4a8b-4e5b-9756-6594247fc30e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40901aeb-4a8b-4e5b-9756-6594247fc30e-maas-api-xks-ca\") pod \"40901aeb-4a8b-4e5b-9756-6594247fc30e\" (UID: \"40901aeb-4a8b-4e5b-9756-6594247fc30e\") "
Oct 30 21:18:35.224957 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:35.224908    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40901aeb-4a8b-4e5b-9756-6594247fc30e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40901aeb-4a8b-4e5b-9756-6594247fc30e-maas-api-openshift-ca\") pod \"40901aeb-4a8b-4e5b-9756-6594247fc30e\" (UID: \"40901aeb-4a8b-4e5b-9756-6594247fc30e\") "
Oct 30 21:18:35.225359 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:35.225263    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40901aeb-4a8b-4e5b-9756-6594247fc30e-maas-api-openshift-ca" pod "40901aeb-4a8b-4e5b-9756-6594247fc30e" (UID: "40901aeb-4a8b-4e5b-9756-6594247fc30e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:18:35.225359 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:35.225234    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51fb32f8-14a4-4d70-9cea-051fcb96f989-maas-api-openshift-ca" pod "51fb32f8-14a4-4d70-9cea-051fcb96f989" (UID: "51fb32f8-14a4-4d70-9cea-051fcb96f989"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:18:35.225714 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:35.225689    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51fb32f8-14a4-4d70-9cea-051fcb96f989-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:18:35.225819 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:35.225721    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40901aeb-4a8b-4e5b-9756-6594247fc30e-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:18:35.226989 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:35.226963    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51fb32f8-14a4-4d70-9cea-051fcb96f989-maas-api-xks-ca" pod "51fb32f8-14a4-4d70-9cea-051fcb96f989" (UID: "51fb32f8-14a4-4d70-9cea-051fcb96f989"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:18:35.227082 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:35.226999    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40901aeb-4a8b-4e5b-9756-6594247fc30e-kube-api-access-6n84k" pod "40901aeb-4a8b-4e5b-9756-6594247fc30e" (UID: "40901aeb-4a8b-4e5b-9756-6594247fc30e"). InnerVolumeSpecName "kube-api-access-6n84k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:18:35.227082 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:35.227030    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51fb32f8-14a4-4d70-9cea-051fcb96f989-kube-api-access-qrmvb" pod "51fb32f8-14a4-4d70-9cea-051fcb96f989" (UID: "51fb32f8-14a4-4d70-9cea-051fcb96f989"). InnerVolumeSpecName "kube-api-access-qrmvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:18:35.227082 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:35.227063    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40901aeb-4a8b-4e5b-9756-6594247fc30e-maas-api-xks-ca" pod "40901aeb-4a8b-4e5b-9756-6594247fc30e" (UID: "40901aeb-4a8b-4e5b-9756-6594247fc30e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:18:35.327438 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:35.327361    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrmvb\" (UniqueName: \"kubernetes.io/projected/51fb32f8-14a4-4d70-9cea-051fcb96f989-kube-api-access-qrmvb\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:18:35.327438 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:35.327393    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n84k\" (UniqueName: \"kubernetes.io/projected/40901aeb-4a8b-4e5b-9756-6594247fc30e-kube-api-access-6n84k\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:18:35.327438 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:35.327404    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51fb32f8-14a4-4d70-9cea-051fcb96f989-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:18:35.327438 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:35.327413    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40901aeb-4a8b-4e5b-9756-6594247fc30e-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:18:35.998339 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:35.998300    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f4cd36b-555c2"
Oct 30 21:18:35.998520 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:35.998336    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f4cd36b-555c2" event={"ID":"51fb32f8-14a4-4d70-9cea-051fcb96f989","Type":"ContainerDied","Data":"eae45d0f1138ff0901845c889ce93735e1fde8da0fc1f3ad6ecaa7fe5d9caefd"}
Oct 30 21:18:35.998520 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:35.998384    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eae45d0f1138ff0901845c889ce93735e1fde8da0fc1f3ad6ecaa7fe5d9caefd"
Oct 30 21:18:35.999847 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:35.999818    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac25069b-rb2gz" event={"ID":"40901aeb-4a8b-4e5b-9756-6594247fc30e","Type":"ContainerDied","Data":"f831f4138c11ff1024ddc6f377383c007198227defb20c808c7ff9a7abf7697e"}
Oct 30 21:18:35.999847 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:35.999839    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac25069b-rb2gz"
Oct 30 21:18:36.000033 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:35.999858    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f831f4138c11ff1024ddc6f377383c007198227defb20c808c7ff9a7abf7697e"
Oct 30 21:18:37.607028 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:37.606992    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d4597073-8ff0-4a4e-933c-c64e934fe977/prometheus/0.log"
Oct 30 21:18:37.614133 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:37.614108    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d4597073-8ff0-4a4e-933c-c64e934fe977/prometheus/0.log"
Oct 30 21:18:37.633434 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:37.633401    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r7lpf_764461d5-e0b2-4856-a97e-0c5d3d34a497/ovn-acl-logging/0.log"
Oct 30 21:18:37.641543 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:37.641517    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r7lpf_764461d5-e0b2-4856-a97e-0c5d3d34a497/ovn-acl-logging/0.log"
Oct 30 21:18:39.175442 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:39.175406    5713 scope.go:122] "RemoveContainer" containerID="f58c4f7fbb4b78ce6c3de1cc5dd1ff4b3327ea7a65c1574dfec5ad788c23a165"
Oct 30 21:18:39.185737 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:39.185714    5713 scope.go:122] "RemoveContainer" containerID="62f60bb9996c98bb3b8d6919834b8dd0c6b6e4e12c1fa96250e7bab7c77510e8"
Oct 30 21:18:39.204444 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:39.204422    5713 scope.go:122] "RemoveContainer" containerID="e66f28c3bdda8ff572c884c025956f2845957102575d2bd249c5e2db7382c889"
Oct 30 21:18:54.460775 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:18:54.460736    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:18:55.043583 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:55.043532    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-d1b889bd/e2e-lifecycle-model-d1b889bd-kserve-6bf5bc48d5-7cw4n"]
Oct 30 21:18:55.044949 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:55.044920    5713 cpu_manager.go:414] "containerMap: removing container" podUID="51fb32f8-14a4-4d70-9cea-051fcb96f989" containerName="revoke-keys"
Oct 30 21:18:55.044949 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:55.044948    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51fb32f8-14a4-4d70-9cea-051fcb96f989" containerName="revoke-keys"
Oct 30 21:18:55.045165 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:55.044991    5713 cpu_manager.go:414] "containerMap: removing container" podUID="63ace536-4242-41d2-b1a0-3b10efad3333" containerName="istio-proxy"
Oct 30 21:18:55.045165 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:55.044999    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63ace536-4242-41d2-b1a0-3b10efad3333" containerName="istio-proxy"
Oct 30 21:18:55.045165 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:55.045040    5713 cpu_manager.go:414] "containerMap: removing container" podUID="40901aeb-4a8b-4e5b-9756-6594247fc30e" containerName="revoke-keys"
Oct 30 21:18:55.045165 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:55.045049    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40901aeb-4a8b-4e5b-9756-6594247fc30e" containerName="revoke-keys"
Oct 30 21:18:55.045389 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:55.045372    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="51fb32f8-14a4-4d70-9cea-051fcb96f989" containerName="revoke-keys"
Oct 30 21:18:55.045457 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:55.045409    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="63ace536-4242-41d2-b1a0-3b10efad3333" containerName="istio-proxy"
Oct 30 21:18:55.045457 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:55.045429    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="63ace536-4242-41d2-b1a0-3b10efad3333" containerName="istio-proxy"
Oct 30 21:18:55.045457 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:55.045448    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="40901aeb-4a8b-4e5b-9756-6594247fc30e" containerName="revoke-keys"
Oct 30 21:18:55.056955 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:55.056924    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d1b889bd/e2e-lifecycle-model-d1b889bd-kserve-6bf5bc48d5-7cw4n"
Oct 30 21:18:55.080193 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:55.080154    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4-tmp-dir\") pod \"e2e-lifecycle-model-d1b889bd-kserve-6bf5bc48d5-7cw4n\" (UID: \"2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4\") " pod="ai-tenant-e2e-mt-d1b889bd/e2e-lifecycle-model-d1b889bd-kserve-6bf5bc48d5-7cw4n"
Oct 30 21:18:55.080369 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:55.080282    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4-model-cache\") pod \"e2e-lifecycle-model-d1b889bd-kserve-6bf5bc48d5-7cw4n\" (UID: \"2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4\") " pod="ai-tenant-e2e-mt-d1b889bd/e2e-lifecycle-model-d1b889bd-kserve-6bf5bc48d5-7cw4n"
Oct 30 21:18:55.080430 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:55.080367    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4-tls-certs\") pod \"e2e-lifecycle-model-d1b889bd-kserve-6bf5bc48d5-7cw4n\" (UID: \"2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4\") " pod="ai-tenant-e2e-mt-d1b889bd/e2e-lifecycle-model-d1b889bd-kserve-6bf5bc48d5-7cw4n"
Oct 30 21:18:55.080515 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:55.080494    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4-home\") pod \"e2e-lifecycle-model-d1b889bd-kserve-6bf5bc48d5-7cw4n\" (UID: \"2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4\") " pod="ai-tenant-e2e-mt-d1b889bd/e2e-lifecycle-model-d1b889bd-kserve-6bf5bc48d5-7cw4n"
Oct 30 21:18:55.080571 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:55.080560    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49l52\" (UniqueName: \"kubernetes.io/projected/2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4-kube-api-access-49l52\") pod \"e2e-lifecycle-model-d1b889bd-kserve-6bf5bc48d5-7cw4n\" (UID: \"2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4\") " pod="ai-tenant-e2e-mt-d1b889bd/e2e-lifecycle-model-d1b889bd-kserve-6bf5bc48d5-7cw4n"
Oct 30 21:18:55.080661 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:55.080635    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4-dshm\") pod \"e2e-lifecycle-model-d1b889bd-kserve-6bf5bc48d5-7cw4n\" (UID: \"2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4\") " pod="ai-tenant-e2e-mt-d1b889bd/e2e-lifecycle-model-d1b889bd-kserve-6bf5bc48d5-7cw4n"
Oct 30 21:18:55.104914 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:55.104881    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-d1b889bd/e2e-lifecycle-model-d1b889bd-kserve-6bf5bc48d5-7cw4n"]
Oct 30 21:18:55.148251 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:55.148210    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-d1b889bd\"/\"openshift-service-ca.crt\""
Oct 30 21:18:55.148451 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:55.148283    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-d1b889bd\"/\"default-dockercfg-kxp9r\""
Oct 30 21:18:55.148451 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:55.148290    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-d1b889bd\"/\"kube-root-ca.crt\""
Oct 30 21:18:55.158970 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:55.158943    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-d1b889bd\"/\"e2e-lifecycle-model-d1b889bd-kserve-self-signed-certs\""
Oct 30 21:18:55.181840 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:55.181810    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4-tmp-dir\") pod \"e2e-lifecycle-model-d1b889bd-kserve-6bf5bc48d5-7cw4n\" (UID: \"2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4\") " pod="ai-tenant-e2e-mt-d1b889bd/e2e-lifecycle-model-d1b889bd-kserve-6bf5bc48d5-7cw4n"
Oct 30 21:18:55.182048 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:55.181888    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4-model-cache\") pod \"e2e-lifecycle-model-d1b889bd-kserve-6bf5bc48d5-7cw4n\" (UID: \"2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4\") " pod="ai-tenant-e2e-mt-d1b889bd/e2e-lifecycle-model-d1b889bd-kserve-6bf5bc48d5-7cw4n"
Oct 30 21:18:55.182048 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:55.181940    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4-tls-certs\") pod \"e2e-lifecycle-model-d1b889bd-kserve-6bf5bc48d5-7cw4n\" (UID: \"2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4\") " pod="ai-tenant-e2e-mt-d1b889bd/e2e-lifecycle-model-d1b889bd-kserve-6bf5bc48d5-7cw4n"
Oct 30 21:18:55.182048 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:55.182037    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4-home\") pod \"e2e-lifecycle-model-d1b889bd-kserve-6bf5bc48d5-7cw4n\" (UID: \"2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4\") " pod="ai-tenant-e2e-mt-d1b889bd/e2e-lifecycle-model-d1b889bd-kserve-6bf5bc48d5-7cw4n"
Oct 30 21:18:55.182215 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:55.182096    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49l52\" (UniqueName: \"kubernetes.io/projected/2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4-kube-api-access-49l52\") pod \"e2e-lifecycle-model-d1b889bd-kserve-6bf5bc48d5-7cw4n\" (UID: \"2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4\") " pod="ai-tenant-e2e-mt-d1b889bd/e2e-lifecycle-model-d1b889bd-kserve-6bf5bc48d5-7cw4n"
Oct 30 21:18:55.182215 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:55.182124    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4-dshm\") pod \"e2e-lifecycle-model-d1b889bd-kserve-6bf5bc48d5-7cw4n\" (UID: \"2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4\") " pod="ai-tenant-e2e-mt-d1b889bd/e2e-lifecycle-model-d1b889bd-kserve-6bf5bc48d5-7cw4n"
Oct 30 21:18:55.182323 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:55.182208    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4-tmp-dir\") pod \"e2e-lifecycle-model-d1b889bd-kserve-6bf5bc48d5-7cw4n\" (UID: \"2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4\") " pod="ai-tenant-e2e-mt-d1b889bd/e2e-lifecycle-model-d1b889bd-kserve-6bf5bc48d5-7cw4n"
Oct 30 21:18:55.182435 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:55.182413    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4-model-cache\") pod \"e2e-lifecycle-model-d1b889bd-kserve-6bf5bc48d5-7cw4n\" (UID: \"2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4\") " pod="ai-tenant-e2e-mt-d1b889bd/e2e-lifecycle-model-d1b889bd-kserve-6bf5bc48d5-7cw4n"
Oct 30 21:18:55.182489 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:55.182426    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4-home\") pod \"e2e-lifecycle-model-d1b889bd-kserve-6bf5bc48d5-7cw4n\" (UID: \"2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4\") " pod="ai-tenant-e2e-mt-d1b889bd/e2e-lifecycle-model-d1b889bd-kserve-6bf5bc48d5-7cw4n"
Oct 30 21:18:55.184640 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:55.184611    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4-dshm\") pod \"e2e-lifecycle-model-d1b889bd-kserve-6bf5bc48d5-7cw4n\" (UID: \"2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4\") " pod="ai-tenant-e2e-mt-d1b889bd/e2e-lifecycle-model-d1b889bd-kserve-6bf5bc48d5-7cw4n"
Oct 30 21:18:55.184856 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:55.184841    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4-tls-certs\") pod \"e2e-lifecycle-model-d1b889bd-kserve-6bf5bc48d5-7cw4n\" (UID: \"2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4\") " pod="ai-tenant-e2e-mt-d1b889bd/e2e-lifecycle-model-d1b889bd-kserve-6bf5bc48d5-7cw4n"
Oct 30 21:18:55.239315 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:55.239283    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49l52\" (UniqueName: \"kubernetes.io/projected/2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4-kube-api-access-49l52\") pod \"e2e-lifecycle-model-d1b889bd-kserve-6bf5bc48d5-7cw4n\" (UID: \"2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4\") " pod="ai-tenant-e2e-mt-d1b889bd/e2e-lifecycle-model-d1b889bd-kserve-6bf5bc48d5-7cw4n"
Oct 30 21:18:55.370079 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:55.369991    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d1b889bd/e2e-lifecycle-model-d1b889bd-kserve-6bf5bc48d5-7cw4n"
Oct 30 21:18:55.749124 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:55.749093    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-d1b889bd/e2e-lifecycle-model-d1b889bd-kserve-6bf5bc48d5-7cw4n"]
Oct 30 21:18:55.750691 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:18:55.750657    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ac9ab3e_0d2f_469b_8538_3ec0ace4ecc4.slice/crio-fad24492435369d437e948c391dc52aa2ce1429e3ac61a2871df4ac16b762f47 WatchSource:0}: Error finding container fad24492435369d437e948c391dc52aa2ce1429e3ac61a2871df4ac16b762f47: Status 404 returned error can't find the container with id fad24492435369d437e948c391dc52aa2ce1429e3ac61a2871df4ac16b762f47
Oct 30 21:18:56.182810 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:56.182744    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d1b889bd/e2e-lifecycle-model-d1b889bd-kserve-6bf5bc48d5-7cw4n" event={"ID":"2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4","Type":"ContainerStarted","Data":"4f9633f73e188265e5106159856da1af2f1af108745645bcf89f191e54061ca0"}
Oct 30 21:18:56.184917 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:56.184869    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d1b889bd/e2e-lifecycle-model-d1b889bd-kserve-6bf5bc48d5-7cw4n" event={"ID":"2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4","Type":"ContainerStarted","Data":"fad24492435369d437e948c391dc52aa2ce1429e3ac61a2871df4ac16b762f47"}
Oct 30 21:18:56.315826 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:56.315742    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-d1b889bd/e2e-lifecycle-model-d1b889bd-kserve-6bf5bc48d5-7cw4n" podStartSLOduration=2.315729203 podStartE2EDuration="2.315729203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:18:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:18:56.314489437 +0000 UTC m=+2419.379265835" watchObservedRunningTime="2026-10-30 21:18:56.315729203 +0000 UTC m=+2419.380505600"
Oct 30 21:18:56.370921 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:18:56.370884    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-d1b889bd/e2e-lifecycle-model-d1b889bd-kserve-6bf5bc48d5-7cw4n"
Oct 30 21:19:00.360404 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:00.360369    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-2f9eff/test-model-2f9eff-kserve-f8c48b795-svkpt"]
Oct 30 21:19:00.360982 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:00.360968    5713 cpu_manager.go:414] "containerMap: removing container" podUID="63ace536-4242-41d2-b1a0-3b10efad3333" containerName="istio-proxy"
Oct 30 21:19:00.361029 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:00.360982    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63ace536-4242-41d2-b1a0-3b10efad3333" containerName="istio-proxy"
Oct 30 21:19:00.364651 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:00.364634    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2f9eff/test-model-2f9eff-kserve-f8c48b795-svkpt"
Oct 30 21:19:00.448638 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:00.448604    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c2ba144-eb1e-40c2-92db-0e5181b50515-model-cache\") pod \"test-model-2f9eff-kserve-f8c48b795-svkpt\" (UID: \"4c2ba144-eb1e-40c2-92db-0e5181b50515\") " pod="ai-tenant-e2e-inf-a-2f9eff/test-model-2f9eff-kserve-f8c48b795-svkpt"
Oct 30 21:19:00.448822 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:00.448642    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk76p\" (UniqueName: \"kubernetes.io/projected/4c2ba144-eb1e-40c2-92db-0e5181b50515-kube-api-access-hk76p\") pod \"test-model-2f9eff-kserve-f8c48b795-svkpt\" (UID: \"4c2ba144-eb1e-40c2-92db-0e5181b50515\") " pod="ai-tenant-e2e-inf-a-2f9eff/test-model-2f9eff-kserve-f8c48b795-svkpt"
Oct 30 21:19:00.448822 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:00.448750    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c2ba144-eb1e-40c2-92db-0e5181b50515-tls-certs\") pod \"test-model-2f9eff-kserve-f8c48b795-svkpt\" (UID: \"4c2ba144-eb1e-40c2-92db-0e5181b50515\") " pod="ai-tenant-e2e-inf-a-2f9eff/test-model-2f9eff-kserve-f8c48b795-svkpt"
Oct 30 21:19:00.448822 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:00.448821    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c2ba144-eb1e-40c2-92db-0e5181b50515-dshm\") pod \"test-model-2f9eff-kserve-f8c48b795-svkpt\" (UID: \"4c2ba144-eb1e-40c2-92db-0e5181b50515\") " pod="ai-tenant-e2e-inf-a-2f9eff/test-model-2f9eff-kserve-f8c48b795-svkpt"
Oct 30 21:19:00.448960 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:00.448909    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c2ba144-eb1e-40c2-92db-0e5181b50515-tmp-dir\") pod \"test-model-2f9eff-kserve-f8c48b795-svkpt\" (UID: \"4c2ba144-eb1e-40c2-92db-0e5181b50515\") " pod="ai-tenant-e2e-inf-a-2f9eff/test-model-2f9eff-kserve-f8c48b795-svkpt"
Oct 30 21:19:00.449017 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:00.449003    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c2ba144-eb1e-40c2-92db-0e5181b50515-home\") pod \"test-model-2f9eff-kserve-f8c48b795-svkpt\" (UID: \"4c2ba144-eb1e-40c2-92db-0e5181b50515\") " pod="ai-tenant-e2e-inf-a-2f9eff/test-model-2f9eff-kserve-f8c48b795-svkpt"
Oct 30 21:19:00.467472 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:00.467442    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-2f9eff\"/\"openshift-service-ca.crt\""
Oct 30 21:19:00.467472 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:00.467447    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-2f9eff\"/\"default-dockercfg-s6l5f\""
Oct 30 21:19:00.467669 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:00.467503    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-2f9eff\"/\"kube-root-ca.crt\""
Oct 30 21:19:00.467669 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:00.467543    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-2f9eff\"/\"test-model-2f9eff-kserve-self-signed-certs\""
Oct 30 21:19:00.470959 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:00.470932    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-2f9eff/test-model-2f9eff-kserve-f8c48b795-svkpt"]
Oct 30 21:19:00.549766 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:00.549732    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c2ba144-eb1e-40c2-92db-0e5181b50515-dshm\") pod \"test-model-2f9eff-kserve-f8c48b795-svkpt\" (UID: \"4c2ba144-eb1e-40c2-92db-0e5181b50515\") " pod="ai-tenant-e2e-inf-a-2f9eff/test-model-2f9eff-kserve-f8c48b795-svkpt"
Oct 30 21:19:00.549963 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:00.549834    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c2ba144-eb1e-40c2-92db-0e5181b50515-tmp-dir\") pod \"test-model-2f9eff-kserve-f8c48b795-svkpt\" (UID: \"4c2ba144-eb1e-40c2-92db-0e5181b50515\") " pod="ai-tenant-e2e-inf-a-2f9eff/test-model-2f9eff-kserve-f8c48b795-svkpt"
Oct 30 21:19:00.549963 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:00.549941    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c2ba144-eb1e-40c2-92db-0e5181b50515-home\") pod \"test-model-2f9eff-kserve-f8c48b795-svkpt\" (UID: \"4c2ba144-eb1e-40c2-92db-0e5181b50515\") " pod="ai-tenant-e2e-inf-a-2f9eff/test-model-2f9eff-kserve-f8c48b795-svkpt"
Oct 30 21:19:00.550079 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:00.550059    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c2ba144-eb1e-40c2-92db-0e5181b50515-model-cache\") pod \"test-model-2f9eff-kserve-f8c48b795-svkpt\" (UID: \"4c2ba144-eb1e-40c2-92db-0e5181b50515\") " pod="ai-tenant-e2e-inf-a-2f9eff/test-model-2f9eff-kserve-f8c48b795-svkpt"
Oct 30 21:19:00.550158 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:00.550104    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk76p\" (UniqueName: \"kubernetes.io/projected/4c2ba144-eb1e-40c2-92db-0e5181b50515-kube-api-access-hk76p\") pod \"test-model-2f9eff-kserve-f8c48b795-svkpt\" (UID: \"4c2ba144-eb1e-40c2-92db-0e5181b50515\") " pod="ai-tenant-e2e-inf-a-2f9eff/test-model-2f9eff-kserve-f8c48b795-svkpt"
Oct 30 21:19:00.550229 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:00.550165    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c2ba144-eb1e-40c2-92db-0e5181b50515-tls-certs\") pod \"test-model-2f9eff-kserve-f8c48b795-svkpt\" (UID: \"4c2ba144-eb1e-40c2-92db-0e5181b50515\") " pod="ai-tenant-e2e-inf-a-2f9eff/test-model-2f9eff-kserve-f8c48b795-svkpt"
Oct 30 21:19:00.550330 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:00.550299    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c2ba144-eb1e-40c2-92db-0e5181b50515-tmp-dir\") pod \"test-model-2f9eff-kserve-f8c48b795-svkpt\" (UID: \"4c2ba144-eb1e-40c2-92db-0e5181b50515\") " pod="ai-tenant-e2e-inf-a-2f9eff/test-model-2f9eff-kserve-f8c48b795-svkpt"
Oct 30 21:19:00.550396 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:00.550344    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c2ba144-eb1e-40c2-92db-0e5181b50515-home\") pod \"test-model-2f9eff-kserve-f8c48b795-svkpt\" (UID: \"4c2ba144-eb1e-40c2-92db-0e5181b50515\") " pod="ai-tenant-e2e-inf-a-2f9eff/test-model-2f9eff-kserve-f8c48b795-svkpt"
Oct 30 21:19:00.550396 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:00.550359    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c2ba144-eb1e-40c2-92db-0e5181b50515-model-cache\") pod \"test-model-2f9eff-kserve-f8c48b795-svkpt\" (UID: \"4c2ba144-eb1e-40c2-92db-0e5181b50515\") " pod="ai-tenant-e2e-inf-a-2f9eff/test-model-2f9eff-kserve-f8c48b795-svkpt"
Oct 30 21:19:00.552285 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:00.552266    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c2ba144-eb1e-40c2-92db-0e5181b50515-dshm\") pod \"test-model-2f9eff-kserve-f8c48b795-svkpt\" (UID: \"4c2ba144-eb1e-40c2-92db-0e5181b50515\") " pod="ai-tenant-e2e-inf-a-2f9eff/test-model-2f9eff-kserve-f8c48b795-svkpt"
Oct 30 21:19:00.552600 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:00.552584    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c2ba144-eb1e-40c2-92db-0e5181b50515-tls-certs\") pod \"test-model-2f9eff-kserve-f8c48b795-svkpt\" (UID: \"4c2ba144-eb1e-40c2-92db-0e5181b50515\") " pod="ai-tenant-e2e-inf-a-2f9eff/test-model-2f9eff-kserve-f8c48b795-svkpt"
Oct 30 21:19:00.608433 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:00.608398    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk76p\" (UniqueName: \"kubernetes.io/projected/4c2ba144-eb1e-40c2-92db-0e5181b50515-kube-api-access-hk76p\") pod \"test-model-2f9eff-kserve-f8c48b795-svkpt\" (UID: \"4c2ba144-eb1e-40c2-92db-0e5181b50515\") " pod="ai-tenant-e2e-inf-a-2f9eff/test-model-2f9eff-kserve-f8c48b795-svkpt"
Oct 30 21:19:00.678352 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:00.678280    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2f9eff/test-model-2f9eff-kserve-f8c48b795-svkpt"
Oct 30 21:19:01.057391 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:19:01.057357    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4c2ba144_eb1e_40c2_92db_0e5181b50515.slice/crio-94835c401eb4e87a48fb5ba0dd29c8e7d89e81f614657377a92070779874b0c0 WatchSource:0}: Error finding container 94835c401eb4e87a48fb5ba0dd29c8e7d89e81f614657377a92070779874b0c0: Status 404 returned error can't find the container with id 94835c401eb4e87a48fb5ba0dd29c8e7d89e81f614657377a92070779874b0c0
Oct 30 21:19:01.065119 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:01.065095    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-2f9eff/test-model-2f9eff-kserve-f8c48b795-svkpt"]
Oct 30 21:19:01.225371 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:01.225330    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2f9eff/test-model-2f9eff-kserve-f8c48b795-svkpt" event={"ID":"4c2ba144-eb1e-40c2-92db-0e5181b50515","Type":"ContainerStarted","Data":"7611df1deaf8818615087d9c6ae3735318191db071c182ce15f4049668478a90"}
Oct 30 21:19:01.225533 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:01.225383    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2f9eff/test-model-2f9eff-kserve-f8c48b795-svkpt" event={"ID":"4c2ba144-eb1e-40c2-92db-0e5181b50515","Type":"ContainerStarted","Data":"94835c401eb4e87a48fb5ba0dd29c8e7d89e81f614657377a92070779874b0c0"}
Oct 30 21:19:01.342894 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:01.342772    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-2f9eff/test-model-2f9eff-kserve-f8c48b795-svkpt" podStartSLOduration=1.3427561080000001 podStartE2EDuration="1.342756108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:19:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:01.340227317 +0000 UTC m=+2424.405003715" watchObservedRunningTime="2026-10-30 21:19:01.342756108 +0000 UTC m=+2424.407532506"
Oct 30 21:19:01.679108 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:01.679012    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-2f9eff/test-model-2f9eff-kserve-f8c48b795-svkpt"
Oct 30 21:19:03.572754 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:03.572717    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-781058-dlclg"]
Oct 30 21:19:03.622046 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:03.622012    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-781058-dlclg"]
Oct 30 21:19:05.064256 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:05.064220    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c710850-22af-4868-863b-99b41649b9b7" path="/var/lib/kubelet/pods/2c710850-22af-4868-863b-99b41649b9b7/volumes"
Oct 30 21:19:05.370480 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:05.370400    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-d1b889bd/e2e-lifecycle-model-d1b889bd-kserve-6bf5bc48d5-7cw4n"
Oct 30 21:19:05.383473 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:05.383441    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-d1b889bd/e2e-lifecycle-model-d1b889bd-kserve-6bf5bc48d5-7cw4n"
Oct 30 21:19:06.276415 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:06.276388    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-d1b889bd/e2e-lifecycle-model-d1b889bd-kserve-6bf5bc48d5-7cw4n"
Oct 30 21:19:10.411103 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:10.411055    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv"]
Oct 30 21:19:10.411620 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:10.411313    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv" podUID="f02b6741-5ee5-49d1-8d70-775cadd1b75d" containerName="istio-proxy" containerID="cri-o://3da89a0b2ec5b12d7de5cbb79bc162d51563dced32e513eaa8d36b659c65e6ca" gracePeriod=30
Oct 30 21:19:10.625783 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:10.625742    5713 patch_prober.go:29] interesting pod/e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:19:10.625971 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:10.625824    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv" podUID="f02b6741-5ee5-49d1-8d70-775cadd1b75d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:19:10.679191 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:10.679119    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-2f9eff/test-model-2f9eff-kserve-f8c48b795-svkpt"
Oct 30 21:19:10.691502 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:10.691478    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-2f9eff/test-model-2f9eff-kserve-f8c48b795-svkpt"
Oct 30 21:19:11.317736 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:11.317703    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-2f9eff/test-model-2f9eff-kserve-f8c48b795-svkpt"
Oct 30 21:19:15.677705 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:15.677678    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv"
Oct 30 21:19:15.766508 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:15.766466    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f02b6741-5ee5-49d1-8d70-775cadd1b75d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f02b6741-5ee5-49d1-8d70-775cadd1b75d-istio-envoy\") pod \"f02b6741-5ee5-49d1-8d70-775cadd1b75d\" (UID: \"f02b6741-5ee5-49d1-8d70-775cadd1b75d\") "
Oct 30 21:19:15.766691 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:15.766559    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f02b6741-5ee5-49d1-8d70-775cadd1b75d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f02b6741-5ee5-49d1-8d70-775cadd1b75d-workload-socket\") pod \"f02b6741-5ee5-49d1-8d70-775cadd1b75d\" (UID: \"f02b6741-5ee5-49d1-8d70-775cadd1b75d\") "
Oct 30 21:19:15.766691 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:15.766588    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f02b6741-5ee5-49d1-8d70-775cadd1b75d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f02b6741-5ee5-49d1-8d70-775cadd1b75d-workload-certs\") pod \"f02b6741-5ee5-49d1-8d70-775cadd1b75d\" (UID: \"f02b6741-5ee5-49d1-8d70-775cadd1b75d\") "
Oct 30 21:19:15.766691 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:15.766613    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f02b6741-5ee5-49d1-8d70-775cadd1b75d-istio-token\" (UniqueName: \"kubernetes.io/projected/f02b6741-5ee5-49d1-8d70-775cadd1b75d-istio-token\") pod \"f02b6741-5ee5-49d1-8d70-775cadd1b75d\" (UID: \"f02b6741-5ee5-49d1-8d70-775cadd1b75d\") "
Oct 30 21:19:15.766691 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:15.766664    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f02b6741-5ee5-49d1-8d70-775cadd1b75d-kube-api-access-sx6ld\" (UniqueName: \"kubernetes.io/projected/f02b6741-5ee5-49d1-8d70-775cadd1b75d-kube-api-access-sx6ld\") pod \"f02b6741-5ee5-49d1-8d70-775cadd1b75d\" (UID: \"f02b6741-5ee5-49d1-8d70-775cadd1b75d\") "
Oct 30 21:19:15.766938 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:15.766725    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f02b6741-5ee5-49d1-8d70-775cadd1b75d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f02b6741-5ee5-49d1-8d70-775cadd1b75d-credential-socket\") pod \"f02b6741-5ee5-49d1-8d70-775cadd1b75d\" (UID: \"f02b6741-5ee5-49d1-8d70-775cadd1b75d\") "
Oct 30 21:19:15.766938 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:15.766889    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f02b6741-5ee5-49d1-8d70-775cadd1b75d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f02b6741-5ee5-49d1-8d70-775cadd1b75d-istio-data\") pod \"f02b6741-5ee5-49d1-8d70-775cadd1b75d\" (UID: \"f02b6741-5ee5-49d1-8d70-775cadd1b75d\") "
Oct 30 21:19:15.766938 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:15.766889    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f02b6741-5ee5-49d1-8d70-775cadd1b75d-workload-certs" pod "f02b6741-5ee5-49d1-8d70-775cadd1b75d" (UID: "f02b6741-5ee5-49d1-8d70-775cadd1b75d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:19:15.766938 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:15.766928    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f02b6741-5ee5-49d1-8d70-775cadd1b75d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f02b6741-5ee5-49d1-8d70-775cadd1b75d-istiod-ca-cert\") pod \"f02b6741-5ee5-49d1-8d70-775cadd1b75d\" (UID: \"f02b6741-5ee5-49d1-8d70-775cadd1b75d\") "
Oct 30 21:19:15.767152 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:15.766977    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f02b6741-5ee5-49d1-8d70-775cadd1b75d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f02b6741-5ee5-49d1-8d70-775cadd1b75d-istio-podinfo\") pod \"f02b6741-5ee5-49d1-8d70-775cadd1b75d\" (UID: \"f02b6741-5ee5-49d1-8d70-775cadd1b75d\") "
Oct 30 21:19:15.767152 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:15.767075    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f02b6741-5ee5-49d1-8d70-775cadd1b75d-credential-socket" pod "f02b6741-5ee5-49d1-8d70-775cadd1b75d" (UID: "f02b6741-5ee5-49d1-8d70-775cadd1b75d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:19:15.767152 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:15.767093    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f02b6741-5ee5-49d1-8d70-775cadd1b75d-istio-data" pod "f02b6741-5ee5-49d1-8d70-775cadd1b75d" (UID: "f02b6741-5ee5-49d1-8d70-775cadd1b75d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:19:15.767489 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:15.767201    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f02b6741-5ee5-49d1-8d70-775cadd1b75d-workload-socket" pod "f02b6741-5ee5-49d1-8d70-775cadd1b75d" (UID: "f02b6741-5ee5-49d1-8d70-775cadd1b75d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:19:15.767489 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:15.767352    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f02b6741-5ee5-49d1-8d70-775cadd1b75d-istiod-ca-cert" pod "f02b6741-5ee5-49d1-8d70-775cadd1b75d" (UID: "f02b6741-5ee5-49d1-8d70-775cadd1b75d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:19:15.767766 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:15.767740    5713 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f02b6741-5ee5-49d1-8d70-775cadd1b75d-workload-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:19:15.767900 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:15.767771    5713 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f02b6741-5ee5-49d1-8d70-775cadd1b75d-workload-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:19:15.767900 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:15.767826    5713 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f02b6741-5ee5-49d1-8d70-775cadd1b75d-credential-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:19:15.767900 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:15.767843    5713 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f02b6741-5ee5-49d1-8d70-775cadd1b75d-istio-data\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:19:15.767900 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:15.767874    5713 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f02b6741-5ee5-49d1-8d70-775cadd1b75d-istiod-ca-cert\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:19:15.769116 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:15.769088    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f02b6741-5ee5-49d1-8d70-775cadd1b75d-istio-envoy" pod "f02b6741-5ee5-49d1-8d70-775cadd1b75d" (UID: "f02b6741-5ee5-49d1-8d70-775cadd1b75d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:19:15.769219 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:15.769137    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f02b6741-5ee5-49d1-8d70-775cadd1b75d-kube-api-access-sx6ld" pod "f02b6741-5ee5-49d1-8d70-775cadd1b75d" (UID: "f02b6741-5ee5-49d1-8d70-775cadd1b75d"). InnerVolumeSpecName "kube-api-access-sx6ld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:19:15.769219 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:15.769203    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f02b6741-5ee5-49d1-8d70-775cadd1b75d-istio-podinfo" pod "f02b6741-5ee5-49d1-8d70-775cadd1b75d" (UID: "f02b6741-5ee5-49d1-8d70-775cadd1b75d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:19:15.769642 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:15.769620    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f02b6741-5ee5-49d1-8d70-775cadd1b75d-istio-token" pod "f02b6741-5ee5-49d1-8d70-775cadd1b75d" (UID: "f02b6741-5ee5-49d1-8d70-775cadd1b75d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:19:15.869557 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:15.869519    5713 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f02b6741-5ee5-49d1-8d70-775cadd1b75d-istio-envoy\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:19:15.869557 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:15.869559    5713 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f02b6741-5ee5-49d1-8d70-775cadd1b75d-istio-token\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:19:15.869760 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:15.869575    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sx6ld\" (UniqueName: \"kubernetes.io/projected/f02b6741-5ee5-49d1-8d70-775cadd1b75d-kube-api-access-sx6ld\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:19:15.869760 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:15.869591    5713 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f02b6741-5ee5-49d1-8d70-775cadd1b75d-istio-podinfo\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:19:16.345953 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:16.345920    5713 generic.go:356] "Generic (PLEG): container finished" podID="f02b6741-5ee5-49d1-8d70-775cadd1b75d" containerID="3da89a0b2ec5b12d7de5cbb79bc162d51563dced32e513eaa8d36b659c65e6ca" exitCode=0
Oct 30 21:19:16.346130 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:16.345966    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv"
Oct 30 21:19:16.346130 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:16.346001    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv" event={"ID":"f02b6741-5ee5-49d1-8d70-775cadd1b75d","Type":"ContainerDied","Data":"3da89a0b2ec5b12d7de5cbb79bc162d51563dced32e513eaa8d36b659c65e6ca"}
Oct 30 21:19:16.346130 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:16.346060    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv" event={"ID":"f02b6741-5ee5-49d1-8d70-775cadd1b75d","Type":"ContainerDied","Data":"91ce0b08b77bb7cad1b152c4b68870724f3ddf6f306fe83b2d678311d9e2e34b"}
Oct 30 21:19:16.346130 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:16.346085    5713 scope.go:122] "RemoveContainer" containerID="3da89a0b2ec5b12d7de5cbb79bc162d51563dced32e513eaa8d36b659c65e6ca"
Oct 30 21:19:16.357652 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:16.357635    5713 scope.go:122] "RemoveContainer" containerID="3da89a0b2ec5b12d7de5cbb79bc162d51563dced32e513eaa8d36b659c65e6ca"
Oct 30 21:19:16.357932 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:19:16.357915    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3da89a0b2ec5b12d7de5cbb79bc162d51563dced32e513eaa8d36b659c65e6ca\": container with ID starting with 3da89a0b2ec5b12d7de5cbb79bc162d51563dced32e513eaa8d36b659c65e6ca not found: ID does not exist" containerID="3da89a0b2ec5b12d7de5cbb79bc162d51563dced32e513eaa8d36b659c65e6ca"
Oct 30 21:19:16.357980 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:16.357941    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3da89a0b2ec5b12d7de5cbb79bc162d51563dced32e513eaa8d36b659c65e6ca"} err="failed to get container status \"3da89a0b2ec5b12d7de5cbb79bc162d51563dced32e513eaa8d36b659c65e6ca\": rpc error: code = NotFound desc = could not find container \"3da89a0b2ec5b12d7de5cbb79bc162d51563dced32e513eaa8d36b659c65e6ca\": container with ID starting with 3da89a0b2ec5b12d7de5cbb79bc162d51563dced32e513eaa8d36b659c65e6ca not found: ID does not exist"
Oct 30 21:19:16.561112 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:16.561042    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv"]
Oct 30 21:19:16.609983 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:16.609914    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-781058-openshift-default-55fb495b7-xp7jv"]
Oct 30 21:19:17.063800 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:17.063757    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f02b6741-5ee5-49d1-8d70-775cadd1b75d" path="/var/lib/kubelet/pods/f02b6741-5ee5-49d1-8d70-775cadd1b75d/volumes"
Oct 30 21:19:19.632530 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:19.632494    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-08df98/test-model-08df98-kserve-5d5c94f6c7-4cqkf"]
Oct 30 21:19:19.633147 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:19.633131    5713 cpu_manager.go:414] "containerMap: removing container" podUID="f02b6741-5ee5-49d1-8d70-775cadd1b75d" containerName="istio-proxy"
Oct 30 21:19:19.633192 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:19.633148    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f02b6741-5ee5-49d1-8d70-775cadd1b75d" containerName="istio-proxy"
Oct 30 21:19:19.633252 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:19.633243    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="f02b6741-5ee5-49d1-8d70-775cadd1b75d" containerName="istio-proxy"
Oct 30 21:19:19.638306 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:19.638288    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-08df98/test-model-08df98-kserve-5d5c94f6c7-4cqkf"
Oct 30 21:19:19.694459 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:19.694424    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-08df98/test-model-08df98-kserve-5d5c94f6c7-4cqkf"]
Oct 30 21:19:19.719115 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:19.719082    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8861d60-e55d-41cc-9d3e-114ce28bbe01-tls-certs\") pod \"test-model-08df98-kserve-5d5c94f6c7-4cqkf\" (UID: \"f8861d60-e55d-41cc-9d3e-114ce28bbe01\") " pod="ai-tenant-e2e-inf-b-08df98/test-model-08df98-kserve-5d5c94f6c7-4cqkf"
Oct 30 21:19:19.719313 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:19.719130    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8861d60-e55d-41cc-9d3e-114ce28bbe01-tmp-dir\") pod \"test-model-08df98-kserve-5d5c94f6c7-4cqkf\" (UID: \"f8861d60-e55d-41cc-9d3e-114ce28bbe01\") " pod="ai-tenant-e2e-inf-b-08df98/test-model-08df98-kserve-5d5c94f6c7-4cqkf"
Oct 30 21:19:19.719313 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:19.719160    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqnrb\" (UniqueName: \"kubernetes.io/projected/f8861d60-e55d-41cc-9d3e-114ce28bbe01-kube-api-access-xqnrb\") pod \"test-model-08df98-kserve-5d5c94f6c7-4cqkf\" (UID: \"f8861d60-e55d-41cc-9d3e-114ce28bbe01\") " pod="ai-tenant-e2e-inf-b-08df98/test-model-08df98-kserve-5d5c94f6c7-4cqkf"
Oct 30 21:19:19.719422 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:19.719337    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8861d60-e55d-41cc-9d3e-114ce28bbe01-dshm\") pod \"test-model-08df98-kserve-5d5c94f6c7-4cqkf\" (UID: \"f8861d60-e55d-41cc-9d3e-114ce28bbe01\") " pod="ai-tenant-e2e-inf-b-08df98/test-model-08df98-kserve-5d5c94f6c7-4cqkf"
Oct 30 21:19:19.719422 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:19.719388    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8861d60-e55d-41cc-9d3e-114ce28bbe01-model-cache\") pod \"test-model-08df98-kserve-5d5c94f6c7-4cqkf\" (UID: \"f8861d60-e55d-41cc-9d3e-114ce28bbe01\") " pod="ai-tenant-e2e-inf-b-08df98/test-model-08df98-kserve-5d5c94f6c7-4cqkf"
Oct 30 21:19:19.719509 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:19.719461    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8861d60-e55d-41cc-9d3e-114ce28bbe01-home\") pod \"test-model-08df98-kserve-5d5c94f6c7-4cqkf\" (UID: \"f8861d60-e55d-41cc-9d3e-114ce28bbe01\") " pod="ai-tenant-e2e-inf-b-08df98/test-model-08df98-kserve-5d5c94f6c7-4cqkf"
Oct 30 21:19:19.738501 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:19.738469    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-08df98\"/\"kube-root-ca.crt\""
Oct 30 21:19:19.738664 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:19.738577    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-08df98\"/\"default-dockercfg-8srlv\""
Oct 30 21:19:19.739988 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:19.739973    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-08df98\"/\"openshift-service-ca.crt\""
Oct 30 21:19:19.740037 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:19.739994    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-08df98\"/\"test-model-08df98-kserve-self-signed-certs\""
Oct 30 21:19:19.820970 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:19.820934    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8861d60-e55d-41cc-9d3e-114ce28bbe01-tls-certs\") pod \"test-model-08df98-kserve-5d5c94f6c7-4cqkf\" (UID: \"f8861d60-e55d-41cc-9d3e-114ce28bbe01\") " pod="ai-tenant-e2e-inf-b-08df98/test-model-08df98-kserve-5d5c94f6c7-4cqkf"
Oct 30 21:19:19.821179 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:19.820984    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8861d60-e55d-41cc-9d3e-114ce28bbe01-tmp-dir\") pod \"test-model-08df98-kserve-5d5c94f6c7-4cqkf\" (UID: \"f8861d60-e55d-41cc-9d3e-114ce28bbe01\") " pod="ai-tenant-e2e-inf-b-08df98/test-model-08df98-kserve-5d5c94f6c7-4cqkf"
Oct 30 21:19:19.821179 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:19.821012    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqnrb\" (UniqueName: \"kubernetes.io/projected/f8861d60-e55d-41cc-9d3e-114ce28bbe01-kube-api-access-xqnrb\") pod \"test-model-08df98-kserve-5d5c94f6c7-4cqkf\" (UID: \"f8861d60-e55d-41cc-9d3e-114ce28bbe01\") " pod="ai-tenant-e2e-inf-b-08df98/test-model-08df98-kserve-5d5c94f6c7-4cqkf"
Oct 30 21:19:19.821179 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:19.821130    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8861d60-e55d-41cc-9d3e-114ce28bbe01-dshm\") pod \"test-model-08df98-kserve-5d5c94f6c7-4cqkf\" (UID: \"f8861d60-e55d-41cc-9d3e-114ce28bbe01\") " pod="ai-tenant-e2e-inf-b-08df98/test-model-08df98-kserve-5d5c94f6c7-4cqkf"
Oct 30 21:19:19.821366 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:19.821179    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8861d60-e55d-41cc-9d3e-114ce28bbe01-model-cache\") pod \"test-model-08df98-kserve-5d5c94f6c7-4cqkf\" (UID: \"f8861d60-e55d-41cc-9d3e-114ce28bbe01\") " pod="ai-tenant-e2e-inf-b-08df98/test-model-08df98-kserve-5d5c94f6c7-4cqkf"
Oct 30 21:19:19.821366 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:19.821214    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8861d60-e55d-41cc-9d3e-114ce28bbe01-home\") pod \"test-model-08df98-kserve-5d5c94f6c7-4cqkf\" (UID: \"f8861d60-e55d-41cc-9d3e-114ce28bbe01\") " pod="ai-tenant-e2e-inf-b-08df98/test-model-08df98-kserve-5d5c94f6c7-4cqkf"
Oct 30 21:19:19.821497 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:19.821458    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8861d60-e55d-41cc-9d3e-114ce28bbe01-tmp-dir\") pod \"test-model-08df98-kserve-5d5c94f6c7-4cqkf\" (UID: \"f8861d60-e55d-41cc-9d3e-114ce28bbe01\") " pod="ai-tenant-e2e-inf-b-08df98/test-model-08df98-kserve-5d5c94f6c7-4cqkf"
Oct 30 21:19:19.821605 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:19.821577    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8861d60-e55d-41cc-9d3e-114ce28bbe01-model-cache\") pod \"test-model-08df98-kserve-5d5c94f6c7-4cqkf\" (UID: \"f8861d60-e55d-41cc-9d3e-114ce28bbe01\") " pod="ai-tenant-e2e-inf-b-08df98/test-model-08df98-kserve-5d5c94f6c7-4cqkf"
Oct 30 21:19:19.821826 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:19.821641    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8861d60-e55d-41cc-9d3e-114ce28bbe01-home\") pod \"test-model-08df98-kserve-5d5c94f6c7-4cqkf\" (UID: \"f8861d60-e55d-41cc-9d3e-114ce28bbe01\") " pod="ai-tenant-e2e-inf-b-08df98/test-model-08df98-kserve-5d5c94f6c7-4cqkf"
Oct 30 21:19:19.823884 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:19.823863    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8861d60-e55d-41cc-9d3e-114ce28bbe01-tls-certs\") pod \"test-model-08df98-kserve-5d5c94f6c7-4cqkf\" (UID: \"f8861d60-e55d-41cc-9d3e-114ce28bbe01\") " pod="ai-tenant-e2e-inf-b-08df98/test-model-08df98-kserve-5d5c94f6c7-4cqkf"
Oct 30 21:19:19.823980 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:19.823946    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8861d60-e55d-41cc-9d3e-114ce28bbe01-dshm\") pod \"test-model-08df98-kserve-5d5c94f6c7-4cqkf\" (UID: \"f8861d60-e55d-41cc-9d3e-114ce28bbe01\") " pod="ai-tenant-e2e-inf-b-08df98/test-model-08df98-kserve-5d5c94f6c7-4cqkf"
Oct 30 21:19:19.878405 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:19.878367    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqnrb\" (UniqueName: \"kubernetes.io/projected/f8861d60-e55d-41cc-9d3e-114ce28bbe01-kube-api-access-xqnrb\") pod \"test-model-08df98-kserve-5d5c94f6c7-4cqkf\" (UID: \"f8861d60-e55d-41cc-9d3e-114ce28bbe01\") " pod="ai-tenant-e2e-inf-b-08df98/test-model-08df98-kserve-5d5c94f6c7-4cqkf"
Oct 30 21:19:19.950971 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:19.950943    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-08df98/test-model-08df98-kserve-5d5c94f6c7-4cqkf"
Oct 30 21:19:20.331262 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:20.331235    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-08df98/test-model-08df98-kserve-5d5c94f6c7-4cqkf"]
Oct 30 21:19:20.332434 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:19:20.332411    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf8861d60_e55d_41cc_9d3e_114ce28bbe01.slice/crio-ab998f1583aa120cee557f86a8e3fff0827f10c9326ddea4bea599d9900ed4c1 WatchSource:0}: Error finding container ab998f1583aa120cee557f86a8e3fff0827f10c9326ddea4bea599d9900ed4c1: Status 404 returned error can't find the container with id ab998f1583aa120cee557f86a8e3fff0827f10c9326ddea4bea599d9900ed4c1
Oct 30 21:19:20.379106 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:20.379076    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-08df98/test-model-08df98-kserve-5d5c94f6c7-4cqkf" event={"ID":"f8861d60-e55d-41cc-9d3e-114ce28bbe01","Type":"ContainerStarted","Data":"ab998f1583aa120cee557f86a8e3fff0827f10c9326ddea4bea599d9900ed4c1"}
Oct 30 21:19:21.388731 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:21.388691    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-08df98/test-model-08df98-kserve-5d5c94f6c7-4cqkf" event={"ID":"f8861d60-e55d-41cc-9d3e-114ce28bbe01","Type":"ContainerStarted","Data":"1e2fde808bb29c728bbae00d0f2e78fab595ba8b8df38afcff2de9e27d236583"}
Oct 30 21:19:21.506582 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:21.506517    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-08df98/test-model-08df98-kserve-5d5c94f6c7-4cqkf" podStartSLOduration=2.506496339 podStartE2EDuration="2.506496339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:19:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:21.504901741 +0000 UTC m=+2444.569678140" watchObservedRunningTime="2026-10-30 21:19:21.506496339 +0000 UTC m=+2444.571272740"
Oct 30 21:19:21.951546 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:21.951501    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-08df98/test-model-08df98-kserve-5d5c94f6c7-4cqkf"
Oct 30 21:19:24.459121 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:19:24.459080    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:19:25.771190 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:25.771154    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz"]
Oct 30 21:19:25.771737 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:25.771369    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz" podUID="caf52803-d06e-4366-ab90-4e06f15a3ce3" containerName="istio-proxy" containerID="cri-o://dbb59b8d8c5d66ed12e8a6cab98ad7f72e48010f04a62672d6d7c41f02d0b9bc" gracePeriod=30
Oct 30 21:19:26.557157 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:26.557117    5713 patch_prober.go:29] interesting pod/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:19:26.557345 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:26.557191    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz" podUID="caf52803-d06e-4366-ab90-4e06f15a3ce3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:19:29.951447 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:29.951409    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-08df98/test-model-08df98-kserve-5d5c94f6c7-4cqkf"
Oct 30 21:19:29.964867 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:29.964842    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-08df98/test-model-08df98-kserve-5d5c94f6c7-4cqkf"
Oct 30 21:19:30.468626 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:30.468598    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-08df98/test-model-08df98-kserve-5d5c94f6c7-4cqkf"
Oct 30 21:19:30.540690 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:30.540625    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-f0fe13/test-model-f0fe13-kserve-c746dc965-tkz9t"]
Oct 30 21:19:30.546571 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:30.546538    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f0fe13/test-model-f0fe13-kserve-c746dc965-tkz9t"
Oct 30 21:19:30.599228 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:30.599186    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f0fe13\"/\"test-model-f0fe13-kserve-self-signed-certs\""
Oct 30 21:19:30.599228 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:30.599185    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f0fe13\"/\"openshift-service-ca.crt\""
Oct 30 21:19:30.602145 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:30.602115    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f0fe13/test-model-f0fe13-kserve-c746dc965-tkz9t"]
Oct 30 21:19:30.619833 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:30.619767    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f0fe13\"/\"kube-root-ca.crt\""
Oct 30 21:19:30.646129 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:30.646093    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f0fe13\"/\"default-dockercfg-fcpmc\""
Oct 30 21:19:30.655179 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:30.655141    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4c9a7f4-79ff-4f0b-855a-d0850f5a3115-tmp-dir\") pod \"test-model-f0fe13-kserve-c746dc965-tkz9t\" (UID: \"f4c9a7f4-79ff-4f0b-855a-d0850f5a3115\") " pod="ai-tenant-e2e-inf-a-f0fe13/test-model-f0fe13-kserve-c746dc965-tkz9t"
Oct 30 21:19:30.655336 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:30.655208    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4c9a7f4-79ff-4f0b-855a-d0850f5a3115-tls-certs\") pod \"test-model-f0fe13-kserve-c746dc965-tkz9t\" (UID: \"f4c9a7f4-79ff-4f0b-855a-d0850f5a3115\") " pod="ai-tenant-e2e-inf-a-f0fe13/test-model-f0fe13-kserve-c746dc965-tkz9t"
Oct 30 21:19:30.655387 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:30.655335    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqp6c\" (UniqueName: \"kubernetes.io/projected/f4c9a7f4-79ff-4f0b-855a-d0850f5a3115-kube-api-access-xqp6c\") pod \"test-model-f0fe13-kserve-c746dc965-tkz9t\" (UID: \"f4c9a7f4-79ff-4f0b-855a-d0850f5a3115\") " pod="ai-tenant-e2e-inf-a-f0fe13/test-model-f0fe13-kserve-c746dc965-tkz9t"
Oct 30 21:19:30.655500 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:30.655468    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4c9a7f4-79ff-4f0b-855a-d0850f5a3115-home\") pod \"test-model-f0fe13-kserve-c746dc965-tkz9t\" (UID: \"f4c9a7f4-79ff-4f0b-855a-d0850f5a3115\") " pod="ai-tenant-e2e-inf-a-f0fe13/test-model-f0fe13-kserve-c746dc965-tkz9t"
Oct 30 21:19:30.655642 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:30.655508    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4c9a7f4-79ff-4f0b-855a-d0850f5a3115-dshm\") pod \"test-model-f0fe13-kserve-c746dc965-tkz9t\" (UID: \"f4c9a7f4-79ff-4f0b-855a-d0850f5a3115\") " pod="ai-tenant-e2e-inf-a-f0fe13/test-model-f0fe13-kserve-c746dc965-tkz9t"
Oct 30 21:19:30.655642 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:30.655558    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4c9a7f4-79ff-4f0b-855a-d0850f5a3115-model-cache\") pod \"test-model-f0fe13-kserve-c746dc965-tkz9t\" (UID: \"f4c9a7f4-79ff-4f0b-855a-d0850f5a3115\") " pod="ai-tenant-e2e-inf-a-f0fe13/test-model-f0fe13-kserve-c746dc965-tkz9t"
Oct 30 21:19:30.756516 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:30.756434    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4c9a7f4-79ff-4f0b-855a-d0850f5a3115-tmp-dir\") pod \"test-model-f0fe13-kserve-c746dc965-tkz9t\" (UID: \"f4c9a7f4-79ff-4f0b-855a-d0850f5a3115\") " pod="ai-tenant-e2e-inf-a-f0fe13/test-model-f0fe13-kserve-c746dc965-tkz9t"
Oct 30 21:19:30.756516 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:30.756481    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4c9a7f4-79ff-4f0b-855a-d0850f5a3115-tls-certs\") pod \"test-model-f0fe13-kserve-c746dc965-tkz9t\" (UID: \"f4c9a7f4-79ff-4f0b-855a-d0850f5a3115\") " pod="ai-tenant-e2e-inf-a-f0fe13/test-model-f0fe13-kserve-c746dc965-tkz9t"
Oct 30 21:19:30.756774 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:30.756557    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqp6c\" (UniqueName: \"kubernetes.io/projected/f4c9a7f4-79ff-4f0b-855a-d0850f5a3115-kube-api-access-xqp6c\") pod \"test-model-f0fe13-kserve-c746dc965-tkz9t\" (UID: \"f4c9a7f4-79ff-4f0b-855a-d0850f5a3115\") " pod="ai-tenant-e2e-inf-a-f0fe13/test-model-f0fe13-kserve-c746dc965-tkz9t"
Oct 30 21:19:30.756774 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:30.756629    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4c9a7f4-79ff-4f0b-855a-d0850f5a3115-home\") pod \"test-model-f0fe13-kserve-c746dc965-tkz9t\" (UID: \"f4c9a7f4-79ff-4f0b-855a-d0850f5a3115\") " pod="ai-tenant-e2e-inf-a-f0fe13/test-model-f0fe13-kserve-c746dc965-tkz9t"
Oct 30 21:19:30.756774 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:30.756649    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4c9a7f4-79ff-4f0b-855a-d0850f5a3115-dshm\") pod \"test-model-f0fe13-kserve-c746dc965-tkz9t\" (UID: \"f4c9a7f4-79ff-4f0b-855a-d0850f5a3115\") " pod="ai-tenant-e2e-inf-a-f0fe13/test-model-f0fe13-kserve-c746dc965-tkz9t"
Oct 30 21:19:30.756774 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:30.756690    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4c9a7f4-79ff-4f0b-855a-d0850f5a3115-model-cache\") pod \"test-model-f0fe13-kserve-c746dc965-tkz9t\" (UID: \"f4c9a7f4-79ff-4f0b-855a-d0850f5a3115\") " pod="ai-tenant-e2e-inf-a-f0fe13/test-model-f0fe13-kserve-c746dc965-tkz9t"
Oct 30 21:19:30.757021 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:30.756837    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4c9a7f4-79ff-4f0b-855a-d0850f5a3115-tmp-dir\") pod \"test-model-f0fe13-kserve-c746dc965-tkz9t\" (UID: \"f4c9a7f4-79ff-4f0b-855a-d0850f5a3115\") " pod="ai-tenant-e2e-inf-a-f0fe13/test-model-f0fe13-kserve-c746dc965-tkz9t"
Oct 30 21:19:30.757021 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:30.756978    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4c9a7f4-79ff-4f0b-855a-d0850f5a3115-home\") pod \"test-model-f0fe13-kserve-c746dc965-tkz9t\" (UID: \"f4c9a7f4-79ff-4f0b-855a-d0850f5a3115\") " pod="ai-tenant-e2e-inf-a-f0fe13/test-model-f0fe13-kserve-c746dc965-tkz9t"
Oct 30 21:19:30.757133 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:30.757114    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4c9a7f4-79ff-4f0b-855a-d0850f5a3115-model-cache\") pod \"test-model-f0fe13-kserve-c746dc965-tkz9t\" (UID: \"f4c9a7f4-79ff-4f0b-855a-d0850f5a3115\") " pod="ai-tenant-e2e-inf-a-f0fe13/test-model-f0fe13-kserve-c746dc965-tkz9t"
Oct 30 21:19:30.759628 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:30.759602    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4c9a7f4-79ff-4f0b-855a-d0850f5a3115-tls-certs\") pod \"test-model-f0fe13-kserve-c746dc965-tkz9t\" (UID: \"f4c9a7f4-79ff-4f0b-855a-d0850f5a3115\") " pod="ai-tenant-e2e-inf-a-f0fe13/test-model-f0fe13-kserve-c746dc965-tkz9t"
Oct 30 21:19:30.759823 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:30.759665    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4c9a7f4-79ff-4f0b-855a-d0850f5a3115-dshm\") pod \"test-model-f0fe13-kserve-c746dc965-tkz9t\" (UID: \"f4c9a7f4-79ff-4f0b-855a-d0850f5a3115\") " pod="ai-tenant-e2e-inf-a-f0fe13/test-model-f0fe13-kserve-c746dc965-tkz9t"
Oct 30 21:19:30.815121 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:30.815087    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqp6c\" (UniqueName: \"kubernetes.io/projected/f4c9a7f4-79ff-4f0b-855a-d0850f5a3115-kube-api-access-xqp6c\") pod \"test-model-f0fe13-kserve-c746dc965-tkz9t\" (UID: \"f4c9a7f4-79ff-4f0b-855a-d0850f5a3115\") " pod="ai-tenant-e2e-inf-a-f0fe13/test-model-f0fe13-kserve-c746dc965-tkz9t"
Oct 30 21:19:30.866238 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:30.866197    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f0fe13/test-model-f0fe13-kserve-c746dc965-tkz9t"
Oct 30 21:19:31.115044 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:31.115018    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz"
Oct 30 21:19:31.166374 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:31.166341    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caf52803-d06e-4366-ab90-4e06f15a3ce3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caf52803-d06e-4366-ab90-4e06f15a3ce3-istio-data\") pod \"caf52803-d06e-4366-ab90-4e06f15a3ce3\" (UID: \"caf52803-d06e-4366-ab90-4e06f15a3ce3\") "
Oct 30 21:19:31.166562 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:31.166391    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caf52803-d06e-4366-ab90-4e06f15a3ce3-kube-api-access-27rm7\" (UniqueName: \"kubernetes.io/projected/caf52803-d06e-4366-ab90-4e06f15a3ce3-kube-api-access-27rm7\") pod \"caf52803-d06e-4366-ab90-4e06f15a3ce3\" (UID: \"caf52803-d06e-4366-ab90-4e06f15a3ce3\") "
Oct 30 21:19:31.166562 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:31.166466    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caf52803-d06e-4366-ab90-4e06f15a3ce3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf52803-d06e-4366-ab90-4e06f15a3ce3-workload-socket\") pod \"caf52803-d06e-4366-ab90-4e06f15a3ce3\" (UID: \"caf52803-d06e-4366-ab90-4e06f15a3ce3\") "
Oct 30 21:19:31.166562 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:31.166505    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caf52803-d06e-4366-ab90-4e06f15a3ce3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caf52803-d06e-4366-ab90-4e06f15a3ce3-istio-envoy\") pod \"caf52803-d06e-4366-ab90-4e06f15a3ce3\" (UID: \"caf52803-d06e-4366-ab90-4e06f15a3ce3\") "
Oct 30 21:19:31.166715 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:31.166694    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caf52803-d06e-4366-ab90-4e06f15a3ce3-istio-token\" (UniqueName: \"kubernetes.io/projected/caf52803-d06e-4366-ab90-4e06f15a3ce3-istio-token\") pod \"caf52803-d06e-4366-ab90-4e06f15a3ce3\" (UID: \"caf52803-d06e-4366-ab90-4e06f15a3ce3\") "
Oct 30 21:19:31.166764 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:31.166723    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caf52803-d06e-4366-ab90-4e06f15a3ce3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caf52803-d06e-4366-ab90-4e06f15a3ce3-workload-certs\") pod \"caf52803-d06e-4366-ab90-4e06f15a3ce3\" (UID: \"caf52803-d06e-4366-ab90-4e06f15a3ce3\") "
Oct 30 21:19:31.166830 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:31.166783    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/caf52803-d06e-4366-ab90-4e06f15a3ce3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caf52803-d06e-4366-ab90-4e06f15a3ce3-istio-podinfo\") pod \"caf52803-d06e-4366-ab90-4e06f15a3ce3\" (UID: \"caf52803-d06e-4366-ab90-4e06f15a3ce3\") "
Oct 30 21:19:31.166900 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:31.166890    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/caf52803-d06e-4366-ab90-4e06f15a3ce3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caf52803-d06e-4366-ab90-4e06f15a3ce3-istiod-ca-cert\") pod \"caf52803-d06e-4366-ab90-4e06f15a3ce3\" (UID: \"caf52803-d06e-4366-ab90-4e06f15a3ce3\") "
Oct 30 21:19:31.166964 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:31.166697    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caf52803-d06e-4366-ab90-4e06f15a3ce3-istio-data" pod "caf52803-d06e-4366-ab90-4e06f15a3ce3" (UID: "caf52803-d06e-4366-ab90-4e06f15a3ce3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:19:31.166964 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:31.166782    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caf52803-d06e-4366-ab90-4e06f15a3ce3-workload-socket" pod "caf52803-d06e-4366-ab90-4e06f15a3ce3" (UID: "caf52803-d06e-4366-ab90-4e06f15a3ce3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:19:31.166964 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:31.166930    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caf52803-d06e-4366-ab90-4e06f15a3ce3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf52803-d06e-4366-ab90-4e06f15a3ce3-credential-socket\") pod \"caf52803-d06e-4366-ab90-4e06f15a3ce3\" (UID: \"caf52803-d06e-4366-ab90-4e06f15a3ce3\") "
Oct 30 21:19:31.166964 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:31.166940    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caf52803-d06e-4366-ab90-4e06f15a3ce3-workload-certs" pod "caf52803-d06e-4366-ab90-4e06f15a3ce3" (UID: "caf52803-d06e-4366-ab90-4e06f15a3ce3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:19:31.167185 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:31.167162    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caf52803-d06e-4366-ab90-4e06f15a3ce3-credential-socket" pod "caf52803-d06e-4366-ab90-4e06f15a3ce3" (UID: "caf52803-d06e-4366-ab90-4e06f15a3ce3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:19:31.167230 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:31.167187    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caf52803-d06e-4366-ab90-4e06f15a3ce3-istiod-ca-cert" pod "caf52803-d06e-4366-ab90-4e06f15a3ce3" (UID: "caf52803-d06e-4366-ab90-4e06f15a3ce3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:19:31.167881 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:31.167854    5713 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caf52803-d06e-4366-ab90-4e06f15a3ce3-workload-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:19:31.167881 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:31.167883    5713 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caf52803-d06e-4366-ab90-4e06f15a3ce3-istiod-ca-cert\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:19:31.168084 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:31.167898    5713 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf52803-d06e-4366-ab90-4e06f15a3ce3-credential-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:19:31.168084 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:31.167915    5713 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caf52803-d06e-4366-ab90-4e06f15a3ce3-istio-data\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:19:31.168084 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:31.167929    5713 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf52803-d06e-4366-ab90-4e06f15a3ce3-workload-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:19:31.169257 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:31.169235    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caf52803-d06e-4366-ab90-4e06f15a3ce3-kube-api-access-27rm7" pod "caf52803-d06e-4366-ab90-4e06f15a3ce3" (UID: "caf52803-d06e-4366-ab90-4e06f15a3ce3"). InnerVolumeSpecName "kube-api-access-27rm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:19:31.169345 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:31.169257    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/caf52803-d06e-4366-ab90-4e06f15a3ce3-istio-podinfo" pod "caf52803-d06e-4366-ab90-4e06f15a3ce3" (UID: "caf52803-d06e-4366-ab90-4e06f15a3ce3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:19:31.169471 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:31.169453    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caf52803-d06e-4366-ab90-4e06f15a3ce3-istio-token" pod "caf52803-d06e-4366-ab90-4e06f15a3ce3" (UID: "caf52803-d06e-4366-ab90-4e06f15a3ce3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:19:31.169525 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:31.169452    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caf52803-d06e-4366-ab90-4e06f15a3ce3-istio-envoy" pod "caf52803-d06e-4366-ab90-4e06f15a3ce3" (UID: "caf52803-d06e-4366-ab90-4e06f15a3ce3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:19:31.266890 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:31.266853    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f0fe13/test-model-f0fe13-kserve-c746dc965-tkz9t"]
Oct 30 21:19:31.269280 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:31.269255    5713 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caf52803-d06e-4366-ab90-4e06f15a3ce3-istio-token\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:19:31.269280 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:31.269280    5713 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caf52803-d06e-4366-ab90-4e06f15a3ce3-istio-podinfo\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:19:31.269498 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:31.269290    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27rm7\" (UniqueName: \"kubernetes.io/projected/caf52803-d06e-4366-ab90-4e06f15a3ce3-kube-api-access-27rm7\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:19:31.269498 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:31.269300    5713 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caf52803-d06e-4366-ab90-4e06f15a3ce3-istio-envoy\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:19:31.301586 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:19:31.301555    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf4c9a7f4_79ff_4f0b_855a_d0850f5a3115.slice/crio-90550b95ba0bda4247fb9a8180460e3c393a543a52598882dfcf74a7713ba045 WatchSource:0}: Error finding container 90550b95ba0bda4247fb9a8180460e3c393a543a52598882dfcf74a7713ba045: Status 404 returned error can't find the container with id 90550b95ba0bda4247fb9a8180460e3c393a543a52598882dfcf74a7713ba045
Oct 30 21:19:31.465297 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:31.465259    5713 generic.go:356] "Generic (PLEG): container finished" podID="caf52803-d06e-4366-ab90-4e06f15a3ce3" containerID="dbb59b8d8c5d66ed12e8a6cab98ad7f72e48010f04a62672d6d7c41f02d0b9bc" exitCode=0
Oct 30 21:19:31.465467 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:31.465320    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz"
Oct 30 21:19:31.465467 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:31.465346    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz" event={"ID":"caf52803-d06e-4366-ab90-4e06f15a3ce3","Type":"ContainerDied","Data":"dbb59b8d8c5d66ed12e8a6cab98ad7f72e48010f04a62672d6d7c41f02d0b9bc"}
Oct 30 21:19:31.465467 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:31.465431    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz" event={"ID":"caf52803-d06e-4366-ab90-4e06f15a3ce3","Type":"ContainerDied","Data":"0c2796172db0dd424fec7af62d1c4f70e128a77754fefe2447dfd346737963cd"}
Oct 30 21:19:31.465467 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:31.465461    5713 scope.go:122] "RemoveContainer" containerID="dbb59b8d8c5d66ed12e8a6cab98ad7f72e48010f04a62672d6d7c41f02d0b9bc"
Oct 30 21:19:31.467719 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:31.467690    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f0fe13/test-model-f0fe13-kserve-c746dc965-tkz9t" event={"ID":"f4c9a7f4-79ff-4f0b-855a-d0850f5a3115","Type":"ContainerStarted","Data":"e1f7c78a6675631615f2ed193d5d8235024d0436ce9c4a483007adfb56aba349"}
Oct 30 21:19:31.467873 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:31.467732    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f0fe13/test-model-f0fe13-kserve-c746dc965-tkz9t" event={"ID":"f4c9a7f4-79ff-4f0b-855a-d0850f5a3115","Type":"ContainerStarted","Data":"90550b95ba0bda4247fb9a8180460e3c393a543a52598882dfcf74a7713ba045"}
Oct 30 21:19:31.477224 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:31.477206    5713 scope.go:122] "RemoveContainer" containerID="dbb59b8d8c5d66ed12e8a6cab98ad7f72e48010f04a62672d6d7c41f02d0b9bc"
Oct 30 21:19:31.477516 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:19:31.477494    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbb59b8d8c5d66ed12e8a6cab98ad7f72e48010f04a62672d6d7c41f02d0b9bc\": container with ID starting with dbb59b8d8c5d66ed12e8a6cab98ad7f72e48010f04a62672d6d7c41f02d0b9bc not found: ID does not exist" containerID="dbb59b8d8c5d66ed12e8a6cab98ad7f72e48010f04a62672d6d7c41f02d0b9bc"
Oct 30 21:19:31.477616 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:31.477519    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbb59b8d8c5d66ed12e8a6cab98ad7f72e48010f04a62672d6d7c41f02d0b9bc"} err="failed to get container status \"dbb59b8d8c5d66ed12e8a6cab98ad7f72e48010f04a62672d6d7c41f02d0b9bc\": rpc error: code = NotFound desc = could not find container \"dbb59b8d8c5d66ed12e8a6cab98ad7f72e48010f04a62672d6d7c41f02d0b9bc\": container with ID starting with dbb59b8d8c5d66ed12e8a6cab98ad7f72e48010f04a62672d6d7c41f02d0b9bc not found: ID does not exist"
Oct 30 21:19:31.632292 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:31.632181    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-f0fe13/test-model-f0fe13-kserve-c746dc965-tkz9t" podStartSLOduration=1.6321625370000001 podStartE2EDuration="1.632162537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:19:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:31.631993649 +0000 UTC m=+2454.696770050" watchObservedRunningTime="2026-10-30 21:19:31.632162537 +0000 UTC m=+2454.696938935"
Oct 30 21:19:31.745697 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:31.745661    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz"]
Oct 30 21:19:31.749043 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:31.749018    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-1fdaa7-openshift-default-84586db7d5-qvzgz"]
Oct 30 21:19:31.866763 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:31.866724    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-f0fe13/test-model-f0fe13-kserve-c746dc965-tkz9t"
Oct 30 21:19:33.064163 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:33.064122    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="caf52803-d06e-4366-ab90-4e06f15a3ce3" path="/var/lib/kubelet/pods/caf52803-d06e-4366-ab90-4e06f15a3ce3/volumes"
Oct 30 21:19:33.869474 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:33.869441    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f9f277-mfvlh"]
Oct 30 21:19:33.919234 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:33.919198    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f9f277-mfvlh"]
Oct 30 21:19:35.064226 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:35.064193    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e635c82-bd6d-48b9-b19a-4d6f1cb4b2bc" path="/var/lib/kubelet/pods/9e635c82-bd6d-48b9-b19a-4d6f1cb4b2bc/volumes"
Oct 30 21:19:39.272533 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:39.272498    5713 scope.go:122] "RemoveContainer" containerID="e5fa43c9eb690b03c8703384dd6783b631004f0484a1852091ae327a18fd2c1a"
Oct 30 21:19:39.283573 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:39.283553    5713 scope.go:122] "RemoveContainer" containerID="daf7235867bc845f263697e98affeb8e91200330cbecfd0e7edc653b9d5c41ae"
Oct 30 21:19:40.867255 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:40.867222    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-f0fe13/test-model-f0fe13-kserve-c746dc965-tkz9t"
Oct 30 21:19:40.880098 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:40.880075    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-f0fe13/test-model-f0fe13-kserve-c746dc965-tkz9t"
Oct 30 21:19:41.556841 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:41.556809    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-f0fe13/test-model-f0fe13-kserve-c746dc965-tkz9t"
Oct 30 21:19:46.052943 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:46.052910    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86f3e806-5dvf6"]
Oct 30 21:19:46.102838 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:46.102804    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86f3e806-5dvf6"]
Oct 30 21:19:46.105898 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:46.105872    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8d71771-stmqp"]
Oct 30 21:19:46.108938 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:46.108910    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8d71771-stmqp"]
Oct 30 21:19:47.065306 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:47.065267    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92f99a0f-9d0e-43c3-ac42-eae9dc96c81a" path="/var/lib/kubelet/pods/92f99a0f-9d0e-43c3-ac42-eae9dc96c81a/volumes"
Oct 30 21:19:47.065891 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:19:47.065871    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b26e0d7b-ba75-4ed8-a45d-e51ee72fc0b9" path="/var/lib/kubelet/pods/b26e0d7b-ba75-4ed8-a45d-e51ee72fc0b9/volumes"
Oct 30 21:19:54.459336 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:19:54.459294    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:20:16.965626 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:16.965573    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1b889bd-mv4m4"]
Oct 30 21:20:16.966234 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:16.966219    5713 cpu_manager.go:414] "containerMap: removing container" podUID="caf52803-d06e-4366-ab90-4e06f15a3ce3" containerName="istio-proxy"
Oct 30 21:20:16.966282 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:16.966235    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caf52803-d06e-4366-ab90-4e06f15a3ce3" containerName="istio-proxy"
Oct 30 21:20:16.966349 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:16.966339    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="caf52803-d06e-4366-ab90-4e06f15a3ce3" containerName="istio-proxy"
Oct 30 21:20:16.969585 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:16.969562    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1b889bd-mv4m4"
Oct 30 21:20:17.025774 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:17.025741    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1b889bd-mv4m4"]
Oct 30 21:20:17.071164 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:17.071133    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:20:17.071164 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:17.071154    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:20:17.071739 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:17.071716    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c1e7084-8c11-44e2-95fa-e6d8cab13f54-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-d1b889bd-mv4m4\" (UID: \"1c1e7084-8c11-44e2-95fa-e6d8cab13f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1b889bd-mv4m4"
Oct 30 21:20:17.071892 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:17.071869    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7d9z\" (UniqueName: \"kubernetes.io/projected/1c1e7084-8c11-44e2-95fa-e6d8cab13f54-kube-api-access-s7d9z\") pod \"maas-api-revoke-keys-e2e-mt-d1b889bd-mv4m4\" (UID: \"1c1e7084-8c11-44e2-95fa-e6d8cab13f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1b889bd-mv4m4"
Oct 30 21:20:17.071973 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:17.071915    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c1e7084-8c11-44e2-95fa-e6d8cab13f54-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-d1b889bd-mv4m4\" (UID: \"1c1e7084-8c11-44e2-95fa-e6d8cab13f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1b889bd-mv4m4"
Oct 30 21:20:17.120927 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:17.120902    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:20:17.120927 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:17.120906    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:20:17.172880 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:17.172843    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7d9z\" (UniqueName: \"kubernetes.io/projected/1c1e7084-8c11-44e2-95fa-e6d8cab13f54-kube-api-access-s7d9z\") pod \"maas-api-revoke-keys-e2e-mt-d1b889bd-mv4m4\" (UID: \"1c1e7084-8c11-44e2-95fa-e6d8cab13f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1b889bd-mv4m4"
Oct 30 21:20:17.172880 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:17.172881    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c1e7084-8c11-44e2-95fa-e6d8cab13f54-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-d1b889bd-mv4m4\" (UID: \"1c1e7084-8c11-44e2-95fa-e6d8cab13f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1b889bd-mv4m4"
Oct 30 21:20:17.173175 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:17.173042    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c1e7084-8c11-44e2-95fa-e6d8cab13f54-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-d1b889bd-mv4m4\" (UID: \"1c1e7084-8c11-44e2-95fa-e6d8cab13f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1b889bd-mv4m4"
Oct 30 21:20:17.173630 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:17.173600    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c1e7084-8c11-44e2-95fa-e6d8cab13f54-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-d1b889bd-mv4m4\" (UID: \"1c1e7084-8c11-44e2-95fa-e6d8cab13f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1b889bd-mv4m4"
Oct 30 21:20:17.175613 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:17.175592    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c1e7084-8c11-44e2-95fa-e6d8cab13f54-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-d1b889bd-mv4m4\" (UID: \"1c1e7084-8c11-44e2-95fa-e6d8cab13f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1b889bd-mv4m4"
Oct 30 21:20:17.229811 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:17.229731    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7d9z\" (UniqueName: \"kubernetes.io/projected/1c1e7084-8c11-44e2-95fa-e6d8cab13f54-kube-api-access-s7d9z\") pod \"maas-api-revoke-keys-e2e-mt-d1b889bd-mv4m4\" (UID: \"1c1e7084-8c11-44e2-95fa-e6d8cab13f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1b889bd-mv4m4"
Oct 30 21:20:17.284466 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:17.284434    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1b889bd-mv4m4"
Oct 30 21:20:17.689844 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:17.689806    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1b889bd-mv4m4"]
Oct 30 21:20:17.690870 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:20:17.690841    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c1e7084_8c11_44e2_95fa_e6d8cab13f54.slice/crio-7f482f0adca96aa4663a4c958b9c55e4b8cd0cb7d89faf4fdfc4866656cbba17 WatchSource:0}: Error finding container 7f482f0adca96aa4663a4c958b9c55e4b8cd0cb7d89faf4fdfc4866656cbba17: Status 404 returned error can't find the container with id 7f482f0adca96aa4663a4c958b9c55e4b8cd0cb7d89faf4fdfc4866656cbba17
Oct 30 21:20:17.835581 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:17.835541    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1b889bd-mv4m4" event={"ID":"1c1e7084-8c11-44e2-95fa-e6d8cab13f54","Type":"ContainerStarted","Data":"7f482f0adca96aa4663a4c958b9c55e4b8cd0cb7d89faf4fdfc4866656cbba17"}
Oct 30 21:20:18.845003 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:18.844902    5713 generic.go:356] "Generic (PLEG): container finished" podID="1c1e7084-8c11-44e2-95fa-e6d8cab13f54" containerID="3dcd1ac62cd444f42beebbfbb703c50a96bbe8717f8d95b016c5ac63f3843f86" exitCode=0
Oct 30 21:20:18.845003 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:18.844989    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1b889bd-mv4m4" event={"ID":"1c1e7084-8c11-44e2-95fa-e6d8cab13f54","Type":"ContainerDied","Data":"3dcd1ac62cd444f42beebbfbb703c50a96bbe8717f8d95b016c5ac63f3843f86"}
Oct 30 21:20:20.028701 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:20.028671    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1b889bd-mv4m4"
Oct 30 21:20:20.112410 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:20.112310    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c1e7084-8c11-44e2-95fa-e6d8cab13f54-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c1e7084-8c11-44e2-95fa-e6d8cab13f54-maas-api-xks-ca\") pod \"1c1e7084-8c11-44e2-95fa-e6d8cab13f54\" (UID: \"1c1e7084-8c11-44e2-95fa-e6d8cab13f54\") "
Oct 30 21:20:20.112621 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:20.112602    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c1e7084-8c11-44e2-95fa-e6d8cab13f54-kube-api-access-s7d9z\" (UniqueName: \"kubernetes.io/projected/1c1e7084-8c11-44e2-95fa-e6d8cab13f54-kube-api-access-s7d9z\") pod \"1c1e7084-8c11-44e2-95fa-e6d8cab13f54\" (UID: \"1c1e7084-8c11-44e2-95fa-e6d8cab13f54\") "
Oct 30 21:20:20.112698 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:20.112645    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c1e7084-8c11-44e2-95fa-e6d8cab13f54-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c1e7084-8c11-44e2-95fa-e6d8cab13f54-maas-api-openshift-ca\") pod \"1c1e7084-8c11-44e2-95fa-e6d8cab13f54\" (UID: \"1c1e7084-8c11-44e2-95fa-e6d8cab13f54\") "
Oct 30 21:20:20.113043 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:20.113019    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c1e7084-8c11-44e2-95fa-e6d8cab13f54-maas-api-openshift-ca" pod "1c1e7084-8c11-44e2-95fa-e6d8cab13f54" (UID: "1c1e7084-8c11-44e2-95fa-e6d8cab13f54"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:20:20.113154 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:20.113107    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c1e7084-8c11-44e2-95fa-e6d8cab13f54-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:20:20.114527 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:20.114502    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c1e7084-8c11-44e2-95fa-e6d8cab13f54-maas-api-xks-ca" pod "1c1e7084-8c11-44e2-95fa-e6d8cab13f54" (UID: "1c1e7084-8c11-44e2-95fa-e6d8cab13f54"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:20:20.114963 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:20.114943    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c1e7084-8c11-44e2-95fa-e6d8cab13f54-kube-api-access-s7d9z" pod "1c1e7084-8c11-44e2-95fa-e6d8cab13f54" (UID: "1c1e7084-8c11-44e2-95fa-e6d8cab13f54"). InnerVolumeSpecName "kube-api-access-s7d9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:20:20.214019 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:20.213984    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7d9z\" (UniqueName: \"kubernetes.io/projected/1c1e7084-8c11-44e2-95fa-e6d8cab13f54-kube-api-access-s7d9z\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:20:20.214019 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:20.214015    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c1e7084-8c11-44e2-95fa-e6d8cab13f54-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:20:20.862742 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:20.862709    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1b889bd-mv4m4"
Oct 30 21:20:20.862742 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:20.862732    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1b889bd-mv4m4" event={"ID":"1c1e7084-8c11-44e2-95fa-e6d8cab13f54","Type":"ContainerDied","Data":"7f482f0adca96aa4663a4c958b9c55e4b8cd0cb7d89faf4fdfc4866656cbba17"}
Oct 30 21:20:20.862991 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:20.862776    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f482f0adca96aa4663a4c958b9c55e4b8cd0cb7d89faf4fdfc4866656cbba17"
Oct 30 21:20:24.458835 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:20:24.458760    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:20:28.463984 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:28.463940    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d1b889bd-7d887b46cb-kvshd"]
Oct 30 21:20:28.573030 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:28.572998    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d1b889bd-bb457476c-9nz29"]
Oct 30 21:20:33.464998 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:33.464938    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-d1b889bd-7d887b46cb-kvshd" podUID="2a5cc9df-af25-4c17-8105-5bc748364a4d" containerName="payload-processing" containerID="cri-o://d8c62126591999403edc64fb78fae1670c920e90b5f5749528a878f2602dc49b" gracePeriod=25
Oct 30 21:20:33.574064 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:33.574027    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-d1b889bd-bb457476c-9nz29" podUID="aa5ce369-e5ae-4099-b215-20dad86b5c3d" containerName="payload-pre-processing" containerID="cri-o://5da55b990290a85a970e4b2644c5773ad7d24b7a2855c66c6e3792453ccf2071" gracePeriod=25
Oct 30 21:20:38.017080 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:38.017045    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-d1b889bd-7d887b46cb-kvshd" podUID="2a5cc9df-af25-4c17-8105-5bc748364a4d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:20:38.017080 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:38.017065    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-d1b889bd-bb457476c-9nz29" podUID="aa5ce369-e5ae-4099-b215-20dad86b5c3d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:20:39.326127 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:39.326096    5713 scope.go:122] "RemoveContainer" containerID="b890ca3381b72c7b6cf673273b1c0d6dbc918c9dd3b781a080beb6ec7fa41658"
Oct 30 21:20:39.336932 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:39.336908    5713 scope.go:122] "RemoveContainer" containerID="b0fb5920965f2e3c4da209c036db168ff00c88e2d4eee8084db343db1e8a2003"
Oct 30 21:20:39.346444 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:39.346421    5713 scope.go:122] "RemoveContainer" containerID="127e3937b250e69bcfd4c7c03f0ac649d2731f4c7382a2a17153e7993c5c325f"
Oct 30 21:20:39.355908 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:39.355885    5713 scope.go:122] "RemoveContainer" containerID="6605deeeed0b9baa20e27dd93a524617b9639d5e3ee1b7b2a16155e04ae80859"
Oct 30 21:20:39.365662 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:39.365638    5713 scope.go:122] "RemoveContainer" containerID="7d2615e052e0dea06f756a93c1d16a61956e062ec4b1a632bc312e9b4be9fd64"
Oct 30 21:20:45.923137 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:45.923093    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fa33f00-hcbpf"]
Oct 30 21:20:45.924033 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:45.924011    5713 cpu_manager.go:414] "containerMap: removing container" podUID="1c1e7084-8c11-44e2-95fa-e6d8cab13f54" containerName="revoke-keys"
Oct 30 21:20:45.924146 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:45.924035    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c1e7084-8c11-44e2-95fa-e6d8cab13f54" containerName="revoke-keys"
Oct 30 21:20:45.924265 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:45.924253    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c1e7084-8c11-44e2-95fa-e6d8cab13f54" containerName="revoke-keys"
Oct 30 21:20:45.930271 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:45.930246    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa33f00-hcbpf"
Oct 30 21:20:45.982119 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:45.982076    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fa33f00-hcbpf"]
Oct 30 21:20:46.026728 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:46.026689    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:20:46.026936 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:46.026748    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:20:46.026936 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:46.026883    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:20:46.027080 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:46.027058    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:20:46.051087 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:46.051047    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c6b30eb-8d16-4cbb-a7b7-d833f5e5e843-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fa33f00-hcbpf\" (UID: \"0c6b30eb-8d16-4cbb-a7b7-d833f5e5e843\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa33f00-hcbpf"
Oct 30 21:20:46.051354 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:46.051197    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c6b30eb-8d16-4cbb-a7b7-d833f5e5e843-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fa33f00-hcbpf\" (UID: \"0c6b30eb-8d16-4cbb-a7b7-d833f5e5e843\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa33f00-hcbpf"
Oct 30 21:20:46.051354 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:46.051264    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znlkf\" (UniqueName: \"kubernetes.io/projected/0c6b30eb-8d16-4cbb-a7b7-d833f5e5e843-kube-api-access-znlkf\") pod \"maas-api-revoke-sub-7fa33f00-hcbpf\" (UID: \"0c6b30eb-8d16-4cbb-a7b7-d833f5e5e843\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa33f00-hcbpf"
Oct 30 21:20:46.153216 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:46.153176    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c6b30eb-8d16-4cbb-a7b7-d833f5e5e843-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fa33f00-hcbpf\" (UID: \"0c6b30eb-8d16-4cbb-a7b7-d833f5e5e843\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa33f00-hcbpf"
Oct 30 21:20:46.153434 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:46.153232    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znlkf\" (UniqueName: \"kubernetes.io/projected/0c6b30eb-8d16-4cbb-a7b7-d833f5e5e843-kube-api-access-znlkf\") pod \"maas-api-revoke-sub-7fa33f00-hcbpf\" (UID: \"0c6b30eb-8d16-4cbb-a7b7-d833f5e5e843\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa33f00-hcbpf"
Oct 30 21:20:46.153898 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:46.153373    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c6b30eb-8d16-4cbb-a7b7-d833f5e5e843-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fa33f00-hcbpf\" (UID: \"0c6b30eb-8d16-4cbb-a7b7-d833f5e5e843\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa33f00-hcbpf"
Oct 30 21:20:46.154283 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:46.154245    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c6b30eb-8d16-4cbb-a7b7-d833f5e5e843-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fa33f00-hcbpf\" (UID: \"0c6b30eb-8d16-4cbb-a7b7-d833f5e5e843\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa33f00-hcbpf"
Oct 30 21:20:46.165525 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:46.165491    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c6b30eb-8d16-4cbb-a7b7-d833f5e5e843-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fa33f00-hcbpf\" (UID: \"0c6b30eb-8d16-4cbb-a7b7-d833f5e5e843\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa33f00-hcbpf"
Oct 30 21:20:46.211221 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:46.211181    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znlkf\" (UniqueName: \"kubernetes.io/projected/0c6b30eb-8d16-4cbb-a7b7-d833f5e5e843-kube-api-access-znlkf\") pod \"maas-api-revoke-sub-7fa33f00-hcbpf\" (UID: \"0c6b30eb-8d16-4cbb-a7b7-d833f5e5e843\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa33f00-hcbpf"
Oct 30 21:20:46.245112 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:46.245077    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa33f00-hcbpf"
Oct 30 21:20:46.638424 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:46.638393    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fa33f00-hcbpf"]
Oct 30 21:20:46.639516 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:20:46.639481    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c6b30eb_8d16_4cbb_a7b7_d833f5e5e843.slice/crio-8349a46b986c05c312a263ba1eeaa2337e2aba767751f955f7b75ddb03073d9e WatchSource:0}: Error finding container 8349a46b986c05c312a263ba1eeaa2337e2aba767751f955f7b75ddb03073d9e: Status 404 returned error can't find the container with id 8349a46b986c05c312a263ba1eeaa2337e2aba767751f955f7b75ddb03073d9e
Oct 30 21:20:47.079664 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:47.079630    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa33f00-hcbpf" event={"ID":"0c6b30eb-8d16-4cbb-a7b7-d833f5e5e843","Type":"ContainerStarted","Data":"5f3cde77f50f67fc03f49933eb6a2e86ec9be29add90c98e9dda321aaf6951eb"}
Oct 30 21:20:47.080147 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:47.079688    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa33f00-hcbpf" event={"ID":"0c6b30eb-8d16-4cbb-a7b7-d833f5e5e843","Type":"ContainerStarted","Data":"8349a46b986c05c312a263ba1eeaa2337e2aba767751f955f7b75ddb03073d9e"}
Oct 30 21:20:47.193739 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:47.193689    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa33f00-hcbpf" podStartSLOduration=2.193675333 podStartE2EDuration="2.193675333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:20:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:20:47.193254871 +0000 UTC m=+2530.258031270" watchObservedRunningTime="2026-10-30 21:20:47.193675333 +0000 UTC m=+2530.258451730"
Oct 30 21:20:48.017391 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:48.017355    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-d1b889bd-7d887b46cb-kvshd" podUID="2a5cc9df-af25-4c17-8105-5bc748364a4d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:20:48.017574 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:48.017366    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-d1b889bd-bb457476c-9nz29" podUID="aa5ce369-e5ae-4099-b215-20dad86b5c3d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:20:48.089286 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:48.089250    5713 generic.go:356] "Generic (PLEG): container finished" podID="0c6b30eb-8d16-4cbb-a7b7-d833f5e5e843" containerID="5f3cde77f50f67fc03f49933eb6a2e86ec9be29add90c98e9dda321aaf6951eb" exitCode=0
Oct 30 21:20:48.089699 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:48.089325    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa33f00-hcbpf" event={"ID":"0c6b30eb-8d16-4cbb-a7b7-d833f5e5e843","Type":"ContainerDied","Data":"5f3cde77f50f67fc03f49933eb6a2e86ec9be29add90c98e9dda321aaf6951eb"}
Oct 30 21:20:49.255376 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:49.255348    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa33f00-hcbpf"
Oct 30 21:20:49.295144 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:49.295117    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c6b30eb-8d16-4cbb-a7b7-d833f5e5e843-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c6b30eb-8d16-4cbb-a7b7-d833f5e5e843-maas-api-xks-ca\") pod \"0c6b30eb-8d16-4cbb-a7b7-d833f5e5e843\" (UID: \"0c6b30eb-8d16-4cbb-a7b7-d833f5e5e843\") "
Oct 30 21:20:49.295272 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:49.295210    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c6b30eb-8d16-4cbb-a7b7-d833f5e5e843-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c6b30eb-8d16-4cbb-a7b7-d833f5e5e843-maas-api-openshift-ca\") pod \"0c6b30eb-8d16-4cbb-a7b7-d833f5e5e843\" (UID: \"0c6b30eb-8d16-4cbb-a7b7-d833f5e5e843\") "
Oct 30 21:20:49.295314 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:49.295279    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c6b30eb-8d16-4cbb-a7b7-d833f5e5e843-kube-api-access-znlkf\" (UniqueName: \"kubernetes.io/projected/0c6b30eb-8d16-4cbb-a7b7-d833f5e5e843-kube-api-access-znlkf\") pod \"0c6b30eb-8d16-4cbb-a7b7-d833f5e5e843\" (UID: \"0c6b30eb-8d16-4cbb-a7b7-d833f5e5e843\") "
Oct 30 21:20:49.295572 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:49.295547    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c6b30eb-8d16-4cbb-a7b7-d833f5e5e843-maas-api-openshift-ca" pod "0c6b30eb-8d16-4cbb-a7b7-d833f5e5e843" (UID: "0c6b30eb-8d16-4cbb-a7b7-d833f5e5e843"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:20:49.297495 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:49.297473    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c6b30eb-8d16-4cbb-a7b7-d833f5e5e843-kube-api-access-znlkf" pod "0c6b30eb-8d16-4cbb-a7b7-d833f5e5e843" (UID: "0c6b30eb-8d16-4cbb-a7b7-d833f5e5e843"). InnerVolumeSpecName "kube-api-access-znlkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:20:49.297601 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:49.297576    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c6b30eb-8d16-4cbb-a7b7-d833f5e5e843-maas-api-xks-ca" pod "0c6b30eb-8d16-4cbb-a7b7-d833f5e5e843" (UID: "0c6b30eb-8d16-4cbb-a7b7-d833f5e5e843"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:20:49.396664 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:49.396584    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znlkf\" (UniqueName: \"kubernetes.io/projected/0c6b30eb-8d16-4cbb-a7b7-d833f5e5e843-kube-api-access-znlkf\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:20:49.396664 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:49.396620    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c6b30eb-8d16-4cbb-a7b7-d833f5e5e843-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:20:49.396664 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:49.396635    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c6b30eb-8d16-4cbb-a7b7-d833f5e5e843-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:20:50.115270 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:50.115228    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa33f00-hcbpf" event={"ID":"0c6b30eb-8d16-4cbb-a7b7-d833f5e5e843","Type":"ContainerDied","Data":"8349a46b986c05c312a263ba1eeaa2337e2aba767751f955f7b75ddb03073d9e"}
Oct 30 21:20:50.115454 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:50.115308    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8349a46b986c05c312a263ba1eeaa2337e2aba767751f955f7b75ddb03073d9e"
Oct 30 21:20:50.115454 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:50.115250    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa33f00-hcbpf"
Oct 30 21:20:53.745440 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:53.745417    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d1b889bd-7d887b46cb-kvshd"
Oct 30 21:20:53.845220 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:53.845194    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a5cc9df-af25-4c17-8105-5bc748364a4d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a5cc9df-af25-4c17-8105-5bc748364a4d-plugins-config-volume\") pod \"2a5cc9df-af25-4c17-8105-5bc748364a4d\" (UID: \"2a5cc9df-af25-4c17-8105-5bc748364a4d\") "
Oct 30 21:20:53.845400 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:53.845387    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a5cc9df-af25-4c17-8105-5bc748364a4d-kube-api-access-6t664\" (UniqueName: \"kubernetes.io/projected/2a5cc9df-af25-4c17-8105-5bc748364a4d-kube-api-access-6t664\") pod \"2a5cc9df-af25-4c17-8105-5bc748364a4d\" (UID: \"2a5cc9df-af25-4c17-8105-5bc748364a4d\") "
Oct 30 21:20:53.845604 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:53.845581    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a5cc9df-af25-4c17-8105-5bc748364a4d-plugins-config-volume" pod "2a5cc9df-af25-4c17-8105-5bc748364a4d" (UID: "2a5cc9df-af25-4c17-8105-5bc748364a4d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:20:53.845842 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:53.845822    5713 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a5cc9df-af25-4c17-8105-5bc748364a4d-plugins-config-volume\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:20:53.847516 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:53.847496    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a5cc9df-af25-4c17-8105-5bc748364a4d-kube-api-access-6t664" pod "2a5cc9df-af25-4c17-8105-5bc748364a4d" (UID: "2a5cc9df-af25-4c17-8105-5bc748364a4d"). InnerVolumeSpecName "kube-api-access-6t664". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:20:53.853999 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:53.853979    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d1b889bd-bb457476c-9nz29"
Oct 30 21:20:53.946856 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:53.946826    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa5ce369-e5ae-4099-b215-20dad86b5c3d-kube-api-access-zqf9l\" (UniqueName: \"kubernetes.io/projected/aa5ce369-e5ae-4099-b215-20dad86b5c3d-kube-api-access-zqf9l\") pod \"aa5ce369-e5ae-4099-b215-20dad86b5c3d\" (UID: \"aa5ce369-e5ae-4099-b215-20dad86b5c3d\") "
Oct 30 21:20:53.947026 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:53.946909    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa5ce369-e5ae-4099-b215-20dad86b5c3d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa5ce369-e5ae-4099-b215-20dad86b5c3d-plugins-config-volume\") pod \"aa5ce369-e5ae-4099-b215-20dad86b5c3d\" (UID: \"aa5ce369-e5ae-4099-b215-20dad86b5c3d\") "
Oct 30 21:20:53.947293 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:53.947271    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa5ce369-e5ae-4099-b215-20dad86b5c3d-plugins-config-volume" pod "aa5ce369-e5ae-4099-b215-20dad86b5c3d" (UID: "aa5ce369-e5ae-4099-b215-20dad86b5c3d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:20:53.947420 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:53.947401    5713 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa5ce369-e5ae-4099-b215-20dad86b5c3d-plugins-config-volume\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:20:53.947461 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:53.947425    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6t664\" (UniqueName: \"kubernetes.io/projected/2a5cc9df-af25-4c17-8105-5bc748364a4d-kube-api-access-6t664\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:20:53.949187 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:53.949168    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa5ce369-e5ae-4099-b215-20dad86b5c3d-kube-api-access-zqf9l" pod "aa5ce369-e5ae-4099-b215-20dad86b5c3d" (UID: "aa5ce369-e5ae-4099-b215-20dad86b5c3d"). InnerVolumeSpecName "kube-api-access-zqf9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:20:54.048555 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:54.048470    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqf9l\" (UniqueName: \"kubernetes.io/projected/aa5ce369-e5ae-4099-b215-20dad86b5c3d-kube-api-access-zqf9l\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:20:54.153848 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:54.153769    5713 generic.go:356] "Generic (PLEG): container finished" podID="aa5ce369-e5ae-4099-b215-20dad86b5c3d" containerID="5da55b990290a85a970e4b2644c5773ad7d24b7a2855c66c6e3792453ccf2071" exitCode=0
Oct 30 21:20:54.154066 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:54.153861    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d1b889bd-bb457476c-9nz29" event={"ID":"aa5ce369-e5ae-4099-b215-20dad86b5c3d","Type":"ContainerDied","Data":"5da55b990290a85a970e4b2644c5773ad7d24b7a2855c66c6e3792453ccf2071"}
Oct 30 21:20:54.154066 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:54.153879    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d1b889bd-bb457476c-9nz29"
Oct 30 21:20:54.154066 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:54.153969    5713 scope.go:122] "RemoveContainer" containerID="5da55b990290a85a970e4b2644c5773ad7d24b7a2855c66c6e3792453ccf2071"
Oct 30 21:20:54.154259 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:54.153956    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d1b889bd-bb457476c-9nz29" event={"ID":"aa5ce369-e5ae-4099-b215-20dad86b5c3d","Type":"ContainerDied","Data":"47119c473c33e2a9eebee5db49c5dc47abb9652d5bd6e503feb794891c989626"}
Oct 30 21:20:54.155659 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:54.155633    5713 generic.go:356] "Generic (PLEG): container finished" podID="2a5cc9df-af25-4c17-8105-5bc748364a4d" containerID="d8c62126591999403edc64fb78fae1670c920e90b5f5749528a878f2602dc49b" exitCode=0
Oct 30 21:20:54.155769 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:54.155685    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d1b889bd-7d887b46cb-kvshd"
Oct 30 21:20:54.155769 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:54.155703    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d1b889bd-7d887b46cb-kvshd" event={"ID":"2a5cc9df-af25-4c17-8105-5bc748364a4d","Type":"ContainerDied","Data":"d8c62126591999403edc64fb78fae1670c920e90b5f5749528a878f2602dc49b"}
Oct 30 21:20:54.155769 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:54.155755    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d1b889bd-7d887b46cb-kvshd" event={"ID":"2a5cc9df-af25-4c17-8105-5bc748364a4d","Type":"ContainerDied","Data":"cc5ce48f44ac8f18b8c6bf489e67088b1968b3ae48a261748e0bfc82813b24d4"}
Oct 30 21:20:54.165466 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:54.165449    5713 scope.go:122] "RemoveContainer" containerID="5da55b990290a85a970e4b2644c5773ad7d24b7a2855c66c6e3792453ccf2071"
Oct 30 21:20:54.165798 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:20:54.165765    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5da55b990290a85a970e4b2644c5773ad7d24b7a2855c66c6e3792453ccf2071\": container with ID starting with 5da55b990290a85a970e4b2644c5773ad7d24b7a2855c66c6e3792453ccf2071 not found: ID does not exist" containerID="5da55b990290a85a970e4b2644c5773ad7d24b7a2855c66c6e3792453ccf2071"
Oct 30 21:20:54.165904 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:54.165815    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5da55b990290a85a970e4b2644c5773ad7d24b7a2855c66c6e3792453ccf2071"} err="failed to get container status \"5da55b990290a85a970e4b2644c5773ad7d24b7a2855c66c6e3792453ccf2071\": rpc error: code = NotFound desc = could not find container \"5da55b990290a85a970e4b2644c5773ad7d24b7a2855c66c6e3792453ccf2071\": container with ID starting with 5da55b990290a85a970e4b2644c5773ad7d24b7a2855c66c6e3792453ccf2071 not found: ID does not exist"
Oct 30 21:20:54.165904 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:54.165837    5713 scope.go:122] "RemoveContainer" containerID="d8c62126591999403edc64fb78fae1670c920e90b5f5749528a878f2602dc49b"
Oct 30 21:20:54.176272 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:54.176257    5713 scope.go:122] "RemoveContainer" containerID="d8c62126591999403edc64fb78fae1670c920e90b5f5749528a878f2602dc49b"
Oct 30 21:20:54.176536 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:20:54.176516    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8c62126591999403edc64fb78fae1670c920e90b5f5749528a878f2602dc49b\": container with ID starting with d8c62126591999403edc64fb78fae1670c920e90b5f5749528a878f2602dc49b not found: ID does not exist" containerID="d8c62126591999403edc64fb78fae1670c920e90b5f5749528a878f2602dc49b"
Oct 30 21:20:54.176578 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:54.176542    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8c62126591999403edc64fb78fae1670c920e90b5f5749528a878f2602dc49b"} err="failed to get container status \"d8c62126591999403edc64fb78fae1670c920e90b5f5749528a878f2602dc49b\": rpc error: code = NotFound desc = could not find container \"d8c62126591999403edc64fb78fae1670c920e90b5f5749528a878f2602dc49b\": container with ID starting with d8c62126591999403edc64fb78fae1670c920e90b5f5749528a878f2602dc49b not found: ID does not exist"
Oct 30 21:20:54.321508 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:54.321430    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d1b889bd-bb457476c-9nz29"]
Oct 30 21:20:54.324644 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:54.324623    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d1b889bd-bb457476c-9nz29"]
Oct 30 21:20:54.434816 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:54.434766    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d1b889bd-7d887b46cb-kvshd"]
Oct 30 21:20:54.439911 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:54.439888    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d1b889bd-7d887b46cb-kvshd"]
Oct 30 21:20:54.459757 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:20:54.459722    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:20:55.065167 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:55.065136    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a5cc9df-af25-4c17-8105-5bc748364a4d" path="/var/lib/kubelet/pods/2a5cc9df-af25-4c17-8105-5bc748364a4d/volumes"
Oct 30 21:20:55.065542 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:20:55.065521    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa5ce369-e5ae-4099-b215-20dad86b5c3d" path="/var/lib/kubelet/pods/aa5ce369-e5ae-4099-b215-20dad86b5c3d/volumes"
Oct 30 21:21:24.459164 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:21:24.459116    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:21:39.416782 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:21:39.416743    5713 scope.go:122] "RemoveContainer" containerID="835ee41964e483089fa87889a40cf316e41c65f2e500d5547745fb09c7e32334"
Oct 30 21:21:39.427738 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:21:39.427715    5713 scope.go:122] "RemoveContainer" containerID="91d96c9ed2fe3cb7c1a61e7d1eb75aed69a366d4d48b96e67782fbceb96ae245"
Oct 30 21:21:39.437650 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:21:39.437625    5713 scope.go:122] "RemoveContainer" containerID="a783d0caf544c41f0c963fd8f2bc1c6a193c1c627ede560cbd11e44900d02165"
Oct 30 21:21:41.033298 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:21:41.033261    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1b889bd-mv4m4"]
Oct 30 21:21:41.082857 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:21:41.082825    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1b889bd-mv4m4"]
Oct 30 21:21:43.064055 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:21:43.064024    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c1e7084-8c11-44e2-95fa-e6d8cab13f54" path="/var/lib/kubelet/pods/1c1e7084-8c11-44e2-95fa-e6d8cab13f54/volumes"
Oct 30 21:21:54.459719 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:21:54.459674    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:22:10.050546 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:10.050511    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-caf211ae-rd6t4"]
Oct 30 21:22:10.099775 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:10.099744    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-caf211ae-rd6t4"]
Oct 30 21:22:11.064053 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:11.064020    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="241cf10b-84bc-455a-aa52-43cd7da69309" path="/var/lib/kubelet/pods/241cf10b-84bc-455a-aa52-43cd7da69309/volumes"
Oct 30 21:22:14.056213 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:14.056174    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b379ae2e-7gr7m"]
Oct 30 21:22:14.105641 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:14.105604    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b379ae2e-7gr7m"]
Oct 30 21:22:15.048053 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:15.048019    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e78449d-9whnj"]
Oct 30 21:22:15.064749 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:15.064712    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68de3f53-caec-4639-ab62-0c225a536f8e" path="/var/lib/kubelet/pods/68de3f53-caec-4639-ab62-0c225a536f8e/volumes"
Oct 30 21:22:15.097944 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:15.097910    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e78449d-9whnj"]
Oct 30 21:22:17.064714 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:17.064639    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9494ab2-0676-4097-b0e3-e28c7a04cae2" path="/var/lib/kubelet/pods/e9494ab2-0676-4097-b0e3-e28c7a04cae2/volumes"
Oct 30 21:22:23.644681 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:23.644650    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27240488-frzll"]
Oct 30 21:22:23.645309 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:23.645291    5713 cpu_manager.go:414] "containerMap: removing container" podUID="0c6b30eb-8d16-4cbb-a7b7-d833f5e5e843" containerName="revoke-keys"
Oct 30 21:22:23.645355 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:23.645313    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c6b30eb-8d16-4cbb-a7b7-d833f5e5e843" containerName="revoke-keys"
Oct 30 21:22:23.645355 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:23.645345    5713 cpu_manager.go:414] "containerMap: removing container" podUID="aa5ce369-e5ae-4099-b215-20dad86b5c3d" containerName="payload-pre-processing"
Oct 30 21:22:23.645355 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:23.645351    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa5ce369-e5ae-4099-b215-20dad86b5c3d" containerName="payload-pre-processing"
Oct 30 21:22:23.645461 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:23.645363    5713 cpu_manager.go:414] "containerMap: removing container" podUID="2a5cc9df-af25-4c17-8105-5bc748364a4d" containerName="payload-processing"
Oct 30 21:22:23.645461 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:23.645367    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a5cc9df-af25-4c17-8105-5bc748364a4d" containerName="payload-processing"
Oct 30 21:22:23.645535 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:23.645464    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c6b30eb-8d16-4cbb-a7b7-d833f5e5e843" containerName="revoke-keys"
Oct 30 21:22:23.645535 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:23.645480    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa5ce369-e5ae-4099-b215-20dad86b5c3d" containerName="payload-pre-processing"
Oct 30 21:22:23.645535 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:23.645487    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a5cc9df-af25-4c17-8105-5bc748364a4d" containerName="payload-processing"
Oct 30 21:22:23.650204 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:23.650188    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27240488-frzll"
Oct 30 21:22:23.705770 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:23.705738    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27240488-frzll"]
Oct 30 21:22:23.751214 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:23.751182    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:22:23.751214 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:23.751206    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:22:23.751428 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:23.751283    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:22:23.752186 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:23.752165    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:22:23.808529 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:23.808498    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dtdw\" (UniqueName: \"kubernetes.io/projected/ea3af8c8-8a0e-4e42-8d3e-40d59316fb49-kube-api-access-2dtdw\") pod \"maas-api-revoke-sub-27240488-frzll\" (UID: \"ea3af8c8-8a0e-4e42-8d3e-40d59316fb49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27240488-frzll"
Oct 30 21:22:23.808668 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:23.808611    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea3af8c8-8a0e-4e42-8d3e-40d59316fb49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27240488-frzll\" (UID: \"ea3af8c8-8a0e-4e42-8d3e-40d59316fb49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27240488-frzll"
Oct 30 21:22:23.808743 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:23.808723    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea3af8c8-8a0e-4e42-8d3e-40d59316fb49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27240488-frzll\" (UID: \"ea3af8c8-8a0e-4e42-8d3e-40d59316fb49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27240488-frzll"
Oct 30 21:22:23.910186 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:23.910107    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea3af8c8-8a0e-4e42-8d3e-40d59316fb49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27240488-frzll\" (UID: \"ea3af8c8-8a0e-4e42-8d3e-40d59316fb49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27240488-frzll"
Oct 30 21:22:23.910339 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:23.910221    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dtdw\" (UniqueName: \"kubernetes.io/projected/ea3af8c8-8a0e-4e42-8d3e-40d59316fb49-kube-api-access-2dtdw\") pod \"maas-api-revoke-sub-27240488-frzll\" (UID: \"ea3af8c8-8a0e-4e42-8d3e-40d59316fb49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27240488-frzll"
Oct 30 21:22:23.910339 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:23.910274    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea3af8c8-8a0e-4e42-8d3e-40d59316fb49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27240488-frzll\" (UID: \"ea3af8c8-8a0e-4e42-8d3e-40d59316fb49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27240488-frzll"
Oct 30 21:22:23.910833 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:23.910775    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea3af8c8-8a0e-4e42-8d3e-40d59316fb49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27240488-frzll\" (UID: \"ea3af8c8-8a0e-4e42-8d3e-40d59316fb49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27240488-frzll"
Oct 30 21:22:23.912851 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:23.912830    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea3af8c8-8a0e-4e42-8d3e-40d59316fb49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27240488-frzll\" (UID: \"ea3af8c8-8a0e-4e42-8d3e-40d59316fb49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27240488-frzll"
Oct 30 21:22:23.966962 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:23.966929    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dtdw\" (UniqueName: \"kubernetes.io/projected/ea3af8c8-8a0e-4e42-8d3e-40d59316fb49-kube-api-access-2dtdw\") pod \"maas-api-revoke-sub-27240488-frzll\" (UID: \"ea3af8c8-8a0e-4e42-8d3e-40d59316fb49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27240488-frzll"
Oct 30 21:22:24.263977 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:24.263942    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27240488-frzll"
Oct 30 21:22:24.459410 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:22:24.459375    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:22:24.646800 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:24.646760    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27240488-frzll"]
Oct 30 21:22:24.647862 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:22:24.647833    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea3af8c8_8a0e_4e42_8d3e_40d59316fb49.slice/crio-c204f6968595073351ed1a48c887d088228743d21f23c37f75f18daeccfbf0ac WatchSource:0}: Error finding container c204f6968595073351ed1a48c887d088228743d21f23c37f75f18daeccfbf0ac: Status 404 returned error can't find the container with id c204f6968595073351ed1a48c887d088228743d21f23c37f75f18daeccfbf0ac
Oct 30 21:22:24.856766 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:24.856731    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27240488-frzll" event={"ID":"ea3af8c8-8a0e-4e42-8d3e-40d59316fb49","Type":"ContainerStarted","Data":"c204f6968595073351ed1a48c887d088228743d21f23c37f75f18daeccfbf0ac"}
Oct 30 21:22:25.866010 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:25.865924    5713 generic.go:356] "Generic (PLEG): container finished" podID="ea3af8c8-8a0e-4e42-8d3e-40d59316fb49" containerID="af280fe0f0ee44f1469d87fc0f1c8b2dd9a4550c37fb390f91c65a16a926207a" exitCode=0
Oct 30 21:22:25.866387 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:25.866014    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27240488-frzll" event={"ID":"ea3af8c8-8a0e-4e42-8d3e-40d59316fb49","Type":"ContainerDied","Data":"af280fe0f0ee44f1469d87fc0f1c8b2dd9a4550c37fb390f91c65a16a926207a"}
Oct 30 21:22:27.032266 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:27.032241    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27240488-frzll"
Oct 30 21:22:27.048364 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:27.048335    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea3af8c8-8a0e-4e42-8d3e-40d59316fb49-kube-api-access-2dtdw\" (UniqueName: \"kubernetes.io/projected/ea3af8c8-8a0e-4e42-8d3e-40d59316fb49-kube-api-access-2dtdw\") pod \"ea3af8c8-8a0e-4e42-8d3e-40d59316fb49\" (UID: \"ea3af8c8-8a0e-4e42-8d3e-40d59316fb49\") "
Oct 30 21:22:27.048534 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:27.048375    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea3af8c8-8a0e-4e42-8d3e-40d59316fb49-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea3af8c8-8a0e-4e42-8d3e-40d59316fb49-maas-api-openshift-ca\") pod \"ea3af8c8-8a0e-4e42-8d3e-40d59316fb49\" (UID: \"ea3af8c8-8a0e-4e42-8d3e-40d59316fb49\") "
Oct 30 21:22:27.048606 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:27.048540    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea3af8c8-8a0e-4e42-8d3e-40d59316fb49-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea3af8c8-8a0e-4e42-8d3e-40d59316fb49-maas-api-xks-ca\") pod \"ea3af8c8-8a0e-4e42-8d3e-40d59316fb49\" (UID: \"ea3af8c8-8a0e-4e42-8d3e-40d59316fb49\") "
Oct 30 21:22:27.048773 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:27.048748    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea3af8c8-8a0e-4e42-8d3e-40d59316fb49-maas-api-openshift-ca" pod "ea3af8c8-8a0e-4e42-8d3e-40d59316fb49" (UID: "ea3af8c8-8a0e-4e42-8d3e-40d59316fb49"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:22:27.049088 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:27.049068    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea3af8c8-8a0e-4e42-8d3e-40d59316fb49-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:22:27.050673 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:27.050651    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea3af8c8-8a0e-4e42-8d3e-40d59316fb49-kube-api-access-2dtdw" pod "ea3af8c8-8a0e-4e42-8d3e-40d59316fb49" (UID: "ea3af8c8-8a0e-4e42-8d3e-40d59316fb49"). InnerVolumeSpecName "kube-api-access-2dtdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:27.050752 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:27.050706    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea3af8c8-8a0e-4e42-8d3e-40d59316fb49-maas-api-xks-ca" pod "ea3af8c8-8a0e-4e42-8d3e-40d59316fb49" (UID: "ea3af8c8-8a0e-4e42-8d3e-40d59316fb49"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:22:27.150931 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:27.150850    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea3af8c8-8a0e-4e42-8d3e-40d59316fb49-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:22:27.151161 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:27.151133    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dtdw\" (UniqueName: \"kubernetes.io/projected/ea3af8c8-8a0e-4e42-8d3e-40d59316fb49-kube-api-access-2dtdw\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:22:27.881378 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:27.881345    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27240488-frzll"
Oct 30 21:22:27.881585 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:27.881397    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27240488-frzll" event={"ID":"ea3af8c8-8a0e-4e42-8d3e-40d59316fb49","Type":"ContainerDied","Data":"c204f6968595073351ed1a48c887d088228743d21f23c37f75f18daeccfbf0ac"}
Oct 30 21:22:27.881585 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:27.881489    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c204f6968595073351ed1a48c887d088228743d21f23c37f75f18daeccfbf0ac"
Oct 30 21:22:35.592743 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:35.592706    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-2f9eff/test-model-2f9eff-kserve-f8c48b795-svkpt"]
Oct 30 21:22:36.588003 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:36.587960    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-114defe4-vktnw"]
Oct 30 21:22:36.588616 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:36.588595    5713 cpu_manager.go:414] "containerMap: removing container" podUID="ea3af8c8-8a0e-4e42-8d3e-40d59316fb49" containerName="revoke-keys"
Oct 30 21:22:36.588725 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:36.588618    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea3af8c8-8a0e-4e42-8d3e-40d59316fb49" containerName="revoke-keys"
Oct 30 21:22:36.588820 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:36.588806    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea3af8c8-8a0e-4e42-8d3e-40d59316fb49" containerName="revoke-keys"
Oct 30 21:22:36.592294 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:36.592260    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-114defe4-vktnw"
Oct 30 21:22:36.644940 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:36.644904    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:22:36.676690 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:36.676652    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfqfk\" (UniqueName: \"kubernetes.io/projected/84380376-2a47-47d8-a71e-a7fa6355f0c3-kube-api-access-tfqfk\") pod \"maas-api-revoke-sub-114defe4-vktnw\" (UID: \"84380376-2a47-47d8-a71e-a7fa6355f0c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-114defe4-vktnw"
Oct 30 21:22:36.676898 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:36.676837    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84380376-2a47-47d8-a71e-a7fa6355f0c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-114defe4-vktnw\" (UID: \"84380376-2a47-47d8-a71e-a7fa6355f0c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-114defe4-vktnw"
Oct 30 21:22:36.676898 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:36.676886    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84380376-2a47-47d8-a71e-a7fa6355f0c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-114defe4-vktnw\" (UID: \"84380376-2a47-47d8-a71e-a7fa6355f0c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-114defe4-vktnw"
Oct 30 21:22:36.694462 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:36.694432    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:22:36.694462 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:36.694452    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:22:36.694632 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:36.694513    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:22:36.697900 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:36.697880    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-114defe4-vktnw"]
Oct 30 21:22:36.778274 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:36.778240    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84380376-2a47-47d8-a71e-a7fa6355f0c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-114defe4-vktnw\" (UID: \"84380376-2a47-47d8-a71e-a7fa6355f0c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-114defe4-vktnw"
Oct 30 21:22:36.778457 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:36.778327    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84380376-2a47-47d8-a71e-a7fa6355f0c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-114defe4-vktnw\" (UID: \"84380376-2a47-47d8-a71e-a7fa6355f0c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-114defe4-vktnw"
Oct 30 21:22:36.778457 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:36.778431    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfqfk\" (UniqueName: \"kubernetes.io/projected/84380376-2a47-47d8-a71e-a7fa6355f0c3-kube-api-access-tfqfk\") pod \"maas-api-revoke-sub-114defe4-vktnw\" (UID: \"84380376-2a47-47d8-a71e-a7fa6355f0c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-114defe4-vktnw"
Oct 30 21:22:36.778973 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:36.778953    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84380376-2a47-47d8-a71e-a7fa6355f0c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-114defe4-vktnw\" (UID: \"84380376-2a47-47d8-a71e-a7fa6355f0c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-114defe4-vktnw"
Oct 30 21:22:36.781068 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:36.781040    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84380376-2a47-47d8-a71e-a7fa6355f0c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-114defe4-vktnw\" (UID: \"84380376-2a47-47d8-a71e-a7fa6355f0c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-114defe4-vktnw"
Oct 30 21:22:36.836464 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:36.836431    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfqfk\" (UniqueName: \"kubernetes.io/projected/84380376-2a47-47d8-a71e-a7fa6355f0c3-kube-api-access-tfqfk\") pod \"maas-api-revoke-sub-114defe4-vktnw\" (UID: \"84380376-2a47-47d8-a71e-a7fa6355f0c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-114defe4-vktnw"
Oct 30 21:22:36.905784 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:36.905707    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-114defe4-vktnw"
Oct 30 21:22:37.285749 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:37.285715    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-114defe4-vktnw"]
Oct 30 21:22:37.285917 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:22:37.285750    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84380376_2a47_47d8_a71e_a7fa6355f0c3.slice/crio-dbd144471dbeb0c5572a58b321ce3b34999f37337122870b66c7e181d1147806 WatchSource:0}: Error finding container dbd144471dbeb0c5572a58b321ce3b34999f37337122870b66c7e181d1147806: Status 404 returned error can't find the container with id dbd144471dbeb0c5572a58b321ce3b34999f37337122870b66c7e181d1147806
Oct 30 21:22:37.954034 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:37.953997    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-114defe4-vktnw" event={"ID":"84380376-2a47-47d8-a71e-a7fa6355f0c3","Type":"ContainerStarted","Data":"5f554b6a489a5bb6a34123459bbc03e4e83cc5b3e916373e5badcdf8af87fe8e"}
Oct 30 21:22:37.954439 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:37.954056    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-114defe4-vktnw" event={"ID":"84380376-2a47-47d8-a71e-a7fa6355f0c3","Type":"ContainerStarted","Data":"dbd144471dbeb0c5572a58b321ce3b34999f37337122870b66c7e181d1147806"}
Oct 30 21:22:38.071609 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:38.071544    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-114defe4-vktnw" podStartSLOduration=2.071525603 podStartE2EDuration="2.071525603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:38.070766126 +0000 UTC m=+2641.135542525" watchObservedRunningTime="2026-10-30 21:22:38.071525603 +0000 UTC m=+2641.136302002"
Oct 30 21:22:38.963077 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:38.963037    5713 generic.go:356] "Generic (PLEG): container finished" podID="84380376-2a47-47d8-a71e-a7fa6355f0c3" containerID="5f554b6a489a5bb6a34123459bbc03e4e83cc5b3e916373e5badcdf8af87fe8e" exitCode=0
Oct 30 21:22:38.963496 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:38.963074    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-114defe4-vktnw" event={"ID":"84380376-2a47-47d8-a71e-a7fa6355f0c3","Type":"ContainerDied","Data":"5f554b6a489a5bb6a34123459bbc03e4e83cc5b3e916373e5badcdf8af87fe8e"}
Oct 30 21:22:39.487404 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:39.487369    5713 scope.go:122] "RemoveContainer" containerID="3fcd580fdec568aaf4816fe6d37463cb9eb543152672db9bd2deec13e1d11073"
Oct 30 21:22:39.498907 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:39.498886    5713 scope.go:122] "RemoveContainer" containerID="88dd3ed8d9891d89f0acc3ecf3065dc73774069844d401cbbbea4280b1e6247d"
Oct 30 21:22:40.140939 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:40.140913    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-114defe4-vktnw"
Oct 30 21:22:40.220756 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:40.220673    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84380376-2a47-47d8-a71e-a7fa6355f0c3-kube-api-access-tfqfk\" (UniqueName: \"kubernetes.io/projected/84380376-2a47-47d8-a71e-a7fa6355f0c3-kube-api-access-tfqfk\") pod \"84380376-2a47-47d8-a71e-a7fa6355f0c3\" (UID: \"84380376-2a47-47d8-a71e-a7fa6355f0c3\") "
Oct 30 21:22:40.220920 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:40.220763    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84380376-2a47-47d8-a71e-a7fa6355f0c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84380376-2a47-47d8-a71e-a7fa6355f0c3-maas-api-openshift-ca\") pod \"84380376-2a47-47d8-a71e-a7fa6355f0c3\" (UID: \"84380376-2a47-47d8-a71e-a7fa6355f0c3\") "
Oct 30 21:22:40.220991 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:40.220973    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84380376-2a47-47d8-a71e-a7fa6355f0c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84380376-2a47-47d8-a71e-a7fa6355f0c3-maas-api-xks-ca\") pod \"84380376-2a47-47d8-a71e-a7fa6355f0c3\" (UID: \"84380376-2a47-47d8-a71e-a7fa6355f0c3\") "
Oct 30 21:22:40.221137 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:40.221113    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84380376-2a47-47d8-a71e-a7fa6355f0c3-maas-api-openshift-ca" pod "84380376-2a47-47d8-a71e-a7fa6355f0c3" (UID: "84380376-2a47-47d8-a71e-a7fa6355f0c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:22:40.221421 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:40.221397    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84380376-2a47-47d8-a71e-a7fa6355f0c3-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:22:40.223140 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:40.223118    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84380376-2a47-47d8-a71e-a7fa6355f0c3-kube-api-access-tfqfk" pod "84380376-2a47-47d8-a71e-a7fa6355f0c3" (UID: "84380376-2a47-47d8-a71e-a7fa6355f0c3"). InnerVolumeSpecName "kube-api-access-tfqfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:40.223217 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:40.223167    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84380376-2a47-47d8-a71e-a7fa6355f0c3-maas-api-xks-ca" pod "84380376-2a47-47d8-a71e-a7fa6355f0c3" (UID: "84380376-2a47-47d8-a71e-a7fa6355f0c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:22:40.323120 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:40.323087    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfqfk\" (UniqueName: \"kubernetes.io/projected/84380376-2a47-47d8-a71e-a7fa6355f0c3-kube-api-access-tfqfk\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:22:40.323120 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:40.323116    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84380376-2a47-47d8-a71e-a7fa6355f0c3-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:22:40.980454 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:40.980423    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-114defe4-vktnw"
Oct 30 21:22:40.980656 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:40.980467    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-114defe4-vktnw" event={"ID":"84380376-2a47-47d8-a71e-a7fa6355f0c3","Type":"ContainerDied","Data":"dbd144471dbeb0c5572a58b321ce3b34999f37337122870b66c7e181d1147806"}
Oct 30 21:22:40.980656 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:40.980554    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dbd144471dbeb0c5572a58b321ce3b34999f37337122870b66c7e181d1147806"
Oct 30 21:22:50.613655 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:50.613614    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-2f9eff/test-model-2f9eff-kserve-f8c48b795-svkpt" podUID="4c2ba144-eb1e-40c2-92db-0e5181b50515" containerName="main" containerID="cri-o://7611df1deaf8818615087d9c6ae3735318191db071c182ce15f4049668478a90" gracePeriod=45
Oct 30 21:22:50.976462 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:50.976433    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2f9eff/test-model-2f9eff-kserve-f8c48b795-svkpt"
Oct 30 21:22:50.993527 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:50.993496    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c2ba144-eb1e-40c2-92db-0e5181b50515-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c2ba144-eb1e-40c2-92db-0e5181b50515-dshm\") pod \"4c2ba144-eb1e-40c2-92db-0e5181b50515\" (UID: \"4c2ba144-eb1e-40c2-92db-0e5181b50515\") "
Oct 30 21:22:50.993731 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:50.993545    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c2ba144-eb1e-40c2-92db-0e5181b50515-kube-api-access-hk76p\" (UniqueName: \"kubernetes.io/projected/4c2ba144-eb1e-40c2-92db-0e5181b50515-kube-api-access-hk76p\") pod \"4c2ba144-eb1e-40c2-92db-0e5181b50515\" (UID: \"4c2ba144-eb1e-40c2-92db-0e5181b50515\") "
Oct 30 21:22:50.993731 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:50.993579    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c2ba144-eb1e-40c2-92db-0e5181b50515-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c2ba144-eb1e-40c2-92db-0e5181b50515-tmp-dir\") pod \"4c2ba144-eb1e-40c2-92db-0e5181b50515\" (UID: \"4c2ba144-eb1e-40c2-92db-0e5181b50515\") "
Oct 30 21:22:50.993731 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:50.993633    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c2ba144-eb1e-40c2-92db-0e5181b50515-tls-certs\" (UniqueName: \"kubernetes.io/secret/4c2ba144-eb1e-40c2-92db-0e5181b50515-tls-certs\") pod \"4c2ba144-eb1e-40c2-92db-0e5181b50515\" (UID: \"4c2ba144-eb1e-40c2-92db-0e5181b50515\") "
Oct 30 21:22:50.993923 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:50.993860    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c2ba144-eb1e-40c2-92db-0e5181b50515-home\" (UniqueName: \"kubernetes.io/empty-dir/4c2ba144-eb1e-40c2-92db-0e5181b50515-home\") pod \"4c2ba144-eb1e-40c2-92db-0e5181b50515\" (UID: \"4c2ba144-eb1e-40c2-92db-0e5181b50515\") "
Oct 30 21:22:50.993923 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:50.993898    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c2ba144-eb1e-40c2-92db-0e5181b50515-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c2ba144-eb1e-40c2-92db-0e5181b50515-model-cache\") pod \"4c2ba144-eb1e-40c2-92db-0e5181b50515\" (UID: \"4c2ba144-eb1e-40c2-92db-0e5181b50515\") "
Oct 30 21:22:50.993923 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:50.993902    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c2ba144-eb1e-40c2-92db-0e5181b50515-tmp-dir" pod "4c2ba144-eb1e-40c2-92db-0e5181b50515" (UID: "4c2ba144-eb1e-40c2-92db-0e5181b50515"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:50.994131 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:50.994109    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c2ba144-eb1e-40c2-92db-0e5181b50515-home" pod "4c2ba144-eb1e-40c2-92db-0e5181b50515" (UID: "4c2ba144-eb1e-40c2-92db-0e5181b50515"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:50.994234 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:50.994210    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c2ba144-eb1e-40c2-92db-0e5181b50515-model-cache" pod "4c2ba144-eb1e-40c2-92db-0e5181b50515" (UID: "4c2ba144-eb1e-40c2-92db-0e5181b50515"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:50.994892 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:50.994872    5713 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c2ba144-eb1e-40c2-92db-0e5181b50515-home\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:22:50.994970 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:50.994899    5713 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c2ba144-eb1e-40c2-92db-0e5181b50515-model-cache\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:22:50.994970 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:50.994915    5713 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c2ba144-eb1e-40c2-92db-0e5181b50515-tmp-dir\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:22:50.996180 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:50.996154    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c2ba144-eb1e-40c2-92db-0e5181b50515-tls-certs" pod "4c2ba144-eb1e-40c2-92db-0e5181b50515" (UID: "4c2ba144-eb1e-40c2-92db-0e5181b50515"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:22:50.996374 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:50.996359    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c2ba144-eb1e-40c2-92db-0e5181b50515-kube-api-access-hk76p" pod "4c2ba144-eb1e-40c2-92db-0e5181b50515" (UID: "4c2ba144-eb1e-40c2-92db-0e5181b50515"). InnerVolumeSpecName "kube-api-access-hk76p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:50.996374 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:50.996353    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c2ba144-eb1e-40c2-92db-0e5181b50515-dshm" pod "4c2ba144-eb1e-40c2-92db-0e5181b50515" (UID: "4c2ba144-eb1e-40c2-92db-0e5181b50515"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:51.076295 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:51.076258    5713 generic.go:356] "Generic (PLEG): container finished" podID="4c2ba144-eb1e-40c2-92db-0e5181b50515" containerID="7611df1deaf8818615087d9c6ae3735318191db071c182ce15f4049668478a90" exitCode=0
Oct 30 21:22:51.076475 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:51.076316    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2f9eff/test-model-2f9eff-kserve-f8c48b795-svkpt"
Oct 30 21:22:51.076475 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:51.076343    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2f9eff/test-model-2f9eff-kserve-f8c48b795-svkpt" event={"ID":"4c2ba144-eb1e-40c2-92db-0e5181b50515","Type":"ContainerDied","Data":"7611df1deaf8818615087d9c6ae3735318191db071c182ce15f4049668478a90"}
Oct 30 21:22:51.076475 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:51.076387    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2f9eff/test-model-2f9eff-kserve-f8c48b795-svkpt" event={"ID":"4c2ba144-eb1e-40c2-92db-0e5181b50515","Type":"ContainerDied","Data":"94835c401eb4e87a48fb5ba0dd29c8e7d89e81f614657377a92070779874b0c0"}
Oct 30 21:22:51.076475 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:51.076403    5713 scope.go:122] "RemoveContainer" containerID="7611df1deaf8818615087d9c6ae3735318191db071c182ce15f4049668478a90"
Oct 30 21:22:51.088329 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:51.088290    5713 scope.go:122] "RemoveContainer" containerID="7611df1deaf8818615087d9c6ae3735318191db071c182ce15f4049668478a90"
Oct 30 21:22:51.088662 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:22:51.088633    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7611df1deaf8818615087d9c6ae3735318191db071c182ce15f4049668478a90\": container with ID starting with 7611df1deaf8818615087d9c6ae3735318191db071c182ce15f4049668478a90 not found: ID does not exist" containerID="7611df1deaf8818615087d9c6ae3735318191db071c182ce15f4049668478a90"
Oct 30 21:22:51.088739 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:51.088672    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7611df1deaf8818615087d9c6ae3735318191db071c182ce15f4049668478a90"} err="failed to get container status \"7611df1deaf8818615087d9c6ae3735318191db071c182ce15f4049668478a90\": rpc error: code = NotFound desc = could not find container \"7611df1deaf8818615087d9c6ae3735318191db071c182ce15f4049668478a90\": container with ID starting with 7611df1deaf8818615087d9c6ae3735318191db071c182ce15f4049668478a90 not found: ID does not exist"
Oct 30 21:22:51.096420 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:51.096387    5713 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c2ba144-eb1e-40c2-92db-0e5181b50515-dshm\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:22:51.096420 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:51.096423    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hk76p\" (UniqueName: \"kubernetes.io/projected/4c2ba144-eb1e-40c2-92db-0e5181b50515-kube-api-access-hk76p\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:22:51.096595 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:51.096438    5713 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c2ba144-eb1e-40c2-92db-0e5181b50515-tls-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:22:51.220161 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:51.220124    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-2f9eff/test-model-2f9eff-kserve-f8c48b795-svkpt"]
Oct 30 21:22:51.223483 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:51.223462    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-2f9eff/test-model-2f9eff-kserve-f8c48b795-svkpt"]
Oct 30 21:22:53.064544 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:53.064510    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c2ba144-eb1e-40c2-92db-0e5181b50515" path="/var/lib/kubelet/pods/4c2ba144-eb1e-40c2-92db-0e5181b50515/volumes"
Oct 30 21:22:53.860966 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:53.860936    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7"]
Oct 30 21:22:53.861562 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:53.861545    5713 cpu_manager.go:414] "containerMap: removing container" podUID="84380376-2a47-47d8-a71e-a7fa6355f0c3" containerName="revoke-keys"
Oct 30 21:22:53.861612 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:53.861565    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84380376-2a47-47d8-a71e-a7fa6355f0c3" containerName="revoke-keys"
Oct 30 21:22:53.861612 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:53.861593    5713 cpu_manager.go:414] "containerMap: removing container" podUID="4c2ba144-eb1e-40c2-92db-0e5181b50515" containerName="main"
Oct 30 21:22:53.861612 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:53.861598    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c2ba144-eb1e-40c2-92db-0e5181b50515" containerName="main"
Oct 30 21:22:53.861719 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:53.861709    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="84380376-2a47-47d8-a71e-a7fa6355f0c3" containerName="revoke-keys"
Oct 30 21:22:53.861760 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:53.861722    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c2ba144-eb1e-40c2-92db-0e5181b50515" containerName="main"
Oct 30 21:22:53.866855 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:53.866838    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7"
Oct 30 21:22:53.931085 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:53.931050    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e54eff2-416c-4a97-8ce1-2ea30c7e982a-istiod-ca-cert\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7\" (UID: \"7e54eff2-416c-4a97-8ce1-2ea30c7e982a\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7"
Oct 30 21:22:53.931234 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:53.931168    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e54eff2-416c-4a97-8ce1-2ea30c7e982a-istio-podinfo\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7\" (UID: \"7e54eff2-416c-4a97-8ce1-2ea30c7e982a\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7"
Oct 30 21:22:53.931234 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:53.931204    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e54eff2-416c-4a97-8ce1-2ea30c7e982a-istio-envoy\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7\" (UID: \"7e54eff2-416c-4a97-8ce1-2ea30c7e982a\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7"
Oct 30 21:22:53.931338 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:53.931320    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjs7z\" (UniqueName: \"kubernetes.io/projected/7e54eff2-416c-4a97-8ce1-2ea30c7e982a-kube-api-access-wjs7z\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7\" (UID: \"7e54eff2-416c-4a97-8ce1-2ea30c7e982a\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7"
Oct 30 21:22:53.931386 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:53.931361    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e54eff2-416c-4a97-8ce1-2ea30c7e982a-workload-socket\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7\" (UID: \"7e54eff2-416c-4a97-8ce1-2ea30c7e982a\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7"
Oct 30 21:22:53.931450 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:53.931434    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e54eff2-416c-4a97-8ce1-2ea30c7e982a-workload-certs\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7\" (UID: \"7e54eff2-416c-4a97-8ce1-2ea30c7e982a\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7"
Oct 30 21:22:53.931510 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:53.931498    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e54eff2-416c-4a97-8ce1-2ea30c7e982a-istio-token\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7\" (UID: \"7e54eff2-416c-4a97-8ce1-2ea30c7e982a\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7"
Oct 30 21:22:53.931558 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:53.931531    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e54eff2-416c-4a97-8ce1-2ea30c7e982a-credential-socket\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7\" (UID: \"7e54eff2-416c-4a97-8ce1-2ea30c7e982a\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7"
Oct 30 21:22:53.931601 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:53.931564    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e54eff2-416c-4a97-8ce1-2ea30c7e982a-istio-data\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7\" (UID: \"7e54eff2-416c-4a97-8ce1-2ea30c7e982a\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7"
Oct 30 21:22:54.032840 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:54.032772    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e54eff2-416c-4a97-8ce1-2ea30c7e982a-istio-podinfo\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7\" (UID: \"7e54eff2-416c-4a97-8ce1-2ea30c7e982a\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7"
Oct 30 21:22:54.033046 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:54.032865    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e54eff2-416c-4a97-8ce1-2ea30c7e982a-istio-envoy\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7\" (UID: \"7e54eff2-416c-4a97-8ce1-2ea30c7e982a\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7"
Oct 30 21:22:54.033046 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:54.032937    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjs7z\" (UniqueName: \"kubernetes.io/projected/7e54eff2-416c-4a97-8ce1-2ea30c7e982a-kube-api-access-wjs7z\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7\" (UID: \"7e54eff2-416c-4a97-8ce1-2ea30c7e982a\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7"
Oct 30 21:22:54.033046 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:54.032975    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e54eff2-416c-4a97-8ce1-2ea30c7e982a-workload-socket\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7\" (UID: \"7e54eff2-416c-4a97-8ce1-2ea30c7e982a\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7"
Oct 30 21:22:54.033046 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:54.033007    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e54eff2-416c-4a97-8ce1-2ea30c7e982a-workload-certs\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7\" (UID: \"7e54eff2-416c-4a97-8ce1-2ea30c7e982a\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7"
Oct 30 21:22:54.033306 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:54.033176    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e54eff2-416c-4a97-8ce1-2ea30c7e982a-istio-token\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7\" (UID: \"7e54eff2-416c-4a97-8ce1-2ea30c7e982a\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7"
Oct 30 21:22:54.033306 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:54.033223    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e54eff2-416c-4a97-8ce1-2ea30c7e982a-credential-socket\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7\" (UID: \"7e54eff2-416c-4a97-8ce1-2ea30c7e982a\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7"
Oct 30 21:22:54.033306 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:54.033289    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e54eff2-416c-4a97-8ce1-2ea30c7e982a-istio-data\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7\" (UID: \"7e54eff2-416c-4a97-8ce1-2ea30c7e982a\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7"
Oct 30 21:22:54.033480 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:54.033358    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e54eff2-416c-4a97-8ce1-2ea30c7e982a-istiod-ca-cert\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7\" (UID: \"7e54eff2-416c-4a97-8ce1-2ea30c7e982a\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7"
Oct 30 21:22:54.033480 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:54.033449    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e54eff2-416c-4a97-8ce1-2ea30c7e982a-workload-certs\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7\" (UID: \"7e54eff2-416c-4a97-8ce1-2ea30c7e982a\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7"
Oct 30 21:22:54.033726 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:54.033704    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e54eff2-416c-4a97-8ce1-2ea30c7e982a-workload-socket\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7\" (UID: \"7e54eff2-416c-4a97-8ce1-2ea30c7e982a\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7"
Oct 30 21:22:54.034128 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:54.034077    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e54eff2-416c-4a97-8ce1-2ea30c7e982a-istio-data\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7\" (UID: \"7e54eff2-416c-4a97-8ce1-2ea30c7e982a\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7"
Oct 30 21:22:54.034262 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:54.034207    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e54eff2-416c-4a97-8ce1-2ea30c7e982a-credential-socket\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7\" (UID: \"7e54eff2-416c-4a97-8ce1-2ea30c7e982a\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7"
Oct 30 21:22:54.034421 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:54.034397    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e54eff2-416c-4a97-8ce1-2ea30c7e982a-istiod-ca-cert\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7\" (UID: \"7e54eff2-416c-4a97-8ce1-2ea30c7e982a\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7"
Oct 30 21:22:54.035699 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:54.035678    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e54eff2-416c-4a97-8ce1-2ea30c7e982a-istio-podinfo\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7\" (UID: \"7e54eff2-416c-4a97-8ce1-2ea30c7e982a\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7"
Oct 30 21:22:54.035825 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:54.035807    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e54eff2-416c-4a97-8ce1-2ea30c7e982a-istio-envoy\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7\" (UID: \"7e54eff2-416c-4a97-8ce1-2ea30c7e982a\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7"
Oct 30 21:22:54.069700 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:54.069624    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7"]
Oct 30 21:22:54.091126 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:54.091094    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjs7z\" (UniqueName: \"kubernetes.io/projected/7e54eff2-416c-4a97-8ce1-2ea30c7e982a-kube-api-access-wjs7z\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7\" (UID: \"7e54eff2-416c-4a97-8ce1-2ea30c7e982a\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7"
Oct 30 21:22:54.091126 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:54.091118    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e54eff2-416c-4a97-8ce1-2ea30c7e982a-istio-token\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7\" (UID: \"7e54eff2-416c-4a97-8ce1-2ea30c7e982a\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7"
Oct 30 21:22:54.181075 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:54.181047    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7"
Oct 30 21:22:54.461352 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:22:54.461269    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:22:54.623804 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:54.623758    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7"]
Oct 30 21:22:54.624735 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:22:54.624708    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e54eff2_416c_4a97_8ce1_2ea30c7e982a.slice/crio-5733e2f6025d8c019885cbf46efceb972a845510653f6f577f166c2e7da3161f WatchSource:0}: Error finding container 5733e2f6025d8c019885cbf46efceb972a845510653f6f577f166c2e7da3161f: Status 404 returned error can't find the container with id 5733e2f6025d8c019885cbf46efceb972a845510653f6f577f166c2e7da3161f
Oct 30 21:22:54.627282 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:54.627252    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:22:54.627360 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:54.627317    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:22:54.627404 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:54.627358    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:22:55.112941 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:55.112897    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7" event={"ID":"7e54eff2-416c-4a97-8ce1-2ea30c7e982a","Type":"ContainerStarted","Data":"bfb64ae3b71ab72444b59294b30251a88f71ba2e1d44dd5ab46ab655db6c6e9c"}
Oct 30 21:22:55.113402 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:55.112964    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7" event={"ID":"7e54eff2-416c-4a97-8ce1-2ea30c7e982a","Type":"ContainerStarted","Data":"5733e2f6025d8c019885cbf46efceb972a845510653f6f577f166c2e7da3161f"}
Oct 30 21:22:55.181942 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:55.181911    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7"
Oct 30 21:22:55.183423 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:55.183392    5713 patch_prober.go:29] interesting pod/e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body=
Oct 30 21:22:55.183583 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:55.183453    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7" podUID="7e54eff2-416c-4a97-8ce1-2ea30c7e982a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused"
Oct 30 21:22:55.289347 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:55.289294    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7" podStartSLOduration=2.289280636 podStartE2EDuration="2.289280636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:55.279256945 +0000 UTC m=+2658.344033345" watchObservedRunningTime="2026-10-30 21:22:55.289280636 +0000 UTC m=+2658.354057033"
Oct 30 21:22:56.182238 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:56.182206    5713 patch_prober.go:29] interesting pod/e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body=
Oct 30 21:22:56.182624 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:56.182257    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7" podUID="7e54eff2-416c-4a97-8ce1-2ea30c7e982a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused"
Oct 30 21:22:57.181549 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:57.181511    5713 patch_prober.go:29] interesting pod/e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body=
Oct 30 21:22:57.181740 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:57.181576    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7" podUID="7e54eff2-416c-4a97-8ce1-2ea30c7e982a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused"
Oct 30 21:22:58.182299 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:58.182272    5713 patch_prober.go:29] interesting pod/e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body=
Oct 30 21:22:58.182693 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:58.182317    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7" podUID="7e54eff2-416c-4a97-8ce1-2ea30c7e982a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused"
Oct 30 21:22:59.181967 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:59.181933    5713 patch_prober.go:29] interesting pod/e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body=
Oct 30 21:22:59.181967 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:22:59.182001    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7" podUID="7e54eff2-416c-4a97-8ce1-2ea30c7e982a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused"
Oct 30 21:23:00.182761 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:00.182721    5713 patch_prober.go:29] interesting pod/e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body=
Oct 30 21:23:00.183354 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:00.183320    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7" podUID="7e54eff2-416c-4a97-8ce1-2ea30c7e982a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused"
Oct 30 21:23:00.283616 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:00.283586    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-08df98/test-model-08df98-kserve-5d5c94f6c7-4cqkf"]
Oct 30 21:23:00.852623 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:00.852586    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f9eff-ncvzk"]
Oct 30 21:23:00.857831 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:00.857810    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f9eff-ncvzk"
Oct 30 21:23:00.910348 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:00.910320    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:23:00.913808 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:00.913764    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f9eff-ncvzk"]
Oct 30 21:23:00.936246 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:00.936210    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/588ccf9a-db41-45b7-a2d7-be18dfde096e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2f9eff-ncvzk\" (UID: \"588ccf9a-db41-45b7-a2d7-be18dfde096e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f9eff-ncvzk"
Oct 30 21:23:00.936407 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:00.936345    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87977\" (UniqueName: \"kubernetes.io/projected/588ccf9a-db41-45b7-a2d7-be18dfde096e-kube-api-access-87977\") pod \"maas-api-revoke-keys-e2e-inf-a-2f9eff-ncvzk\" (UID: \"588ccf9a-db41-45b7-a2d7-be18dfde096e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f9eff-ncvzk"
Oct 30 21:23:00.936508 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:00.936480    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/588ccf9a-db41-45b7-a2d7-be18dfde096e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2f9eff-ncvzk\" (UID: \"588ccf9a-db41-45b7-a2d7-be18dfde096e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f9eff-ncvzk"
Oct 30 21:23:00.958713 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:00.958685    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:23:00.958878 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:00.958771    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:23:00.958878 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:00.958827    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:23:01.038037 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:01.037989    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/588ccf9a-db41-45b7-a2d7-be18dfde096e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2f9eff-ncvzk\" (UID: \"588ccf9a-db41-45b7-a2d7-be18dfde096e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f9eff-ncvzk"
Oct 30 21:23:01.038245 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:01.038090    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87977\" (UniqueName: \"kubernetes.io/projected/588ccf9a-db41-45b7-a2d7-be18dfde096e-kube-api-access-87977\") pod \"maas-api-revoke-keys-e2e-inf-a-2f9eff-ncvzk\" (UID: \"588ccf9a-db41-45b7-a2d7-be18dfde096e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f9eff-ncvzk"
Oct 30 21:23:01.038245 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:01.038160    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/588ccf9a-db41-45b7-a2d7-be18dfde096e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2f9eff-ncvzk\" (UID: \"588ccf9a-db41-45b7-a2d7-be18dfde096e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f9eff-ncvzk"
Oct 30 21:23:01.038809 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:01.038766    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/588ccf9a-db41-45b7-a2d7-be18dfde096e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2f9eff-ncvzk\" (UID: \"588ccf9a-db41-45b7-a2d7-be18dfde096e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f9eff-ncvzk"
Oct 30 21:23:01.040939 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:01.040919    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/588ccf9a-db41-45b7-a2d7-be18dfde096e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2f9eff-ncvzk\" (UID: \"588ccf9a-db41-45b7-a2d7-be18dfde096e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f9eff-ncvzk"
Oct 30 21:23:01.096071 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:01.096038    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87977\" (UniqueName: \"kubernetes.io/projected/588ccf9a-db41-45b7-a2d7-be18dfde096e-kube-api-access-87977\") pod \"maas-api-revoke-keys-e2e-inf-a-2f9eff-ncvzk\" (UID: \"588ccf9a-db41-45b7-a2d7-be18dfde096e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f9eff-ncvzk"
Oct 30 21:23:01.173000 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:01.172922    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f9eff-ncvzk"
Oct 30 21:23:01.186988 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:01.186968    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7"
Oct 30 21:23:01.187531 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:01.187468    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7"
Oct 30 21:23:01.188207 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:01.188185    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7"
Oct 30 21:23:01.554949 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:01.554907    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f9eff-ncvzk"]
Oct 30 21:23:01.558760 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:23:01.558731    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod588ccf9a_db41_45b7_a2d7_be18dfde096e.slice/crio-fb2ea1972d3686bf257dbe2c1e4f4bd449b2a78f4d31fd9ed5fdb929b2f7d989 WatchSource:0}: Error finding container fb2ea1972d3686bf257dbe2c1e4f4bd449b2a78f4d31fd9ed5fdb929b2f7d989: Status 404 returned error can't find the container with id fb2ea1972d3686bf257dbe2c1e4f4bd449b2a78f4d31fd9ed5fdb929b2f7d989
Oct 30 21:23:02.201029 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:02.200993    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f9eff-ncvzk" event={"ID":"588ccf9a-db41-45b7-a2d7-be18dfde096e","Type":"ContainerStarted","Data":"681b52304e085676252963dd556f4a0bc5918e190d80add968e58d0c87236de9"}
Oct 30 21:23:02.201466 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:02.201055    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f9eff-ncvzk" event={"ID":"588ccf9a-db41-45b7-a2d7-be18dfde096e","Type":"ContainerStarted","Data":"fb2ea1972d3686bf257dbe2c1e4f4bd449b2a78f4d31fd9ed5fdb929b2f7d989"}
Oct 30 21:23:02.316603 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:02.316549    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f9eff-ncvzk" podStartSLOduration=2.316535071 podStartE2EDuration="2.316535071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:02.313298548 +0000 UTC m=+2665.378074951" watchObservedRunningTime="2026-10-30 21:23:02.316535071 +0000 UTC m=+2665.381311468"
Oct 30 21:23:03.209367 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:03.209330    5713 generic.go:356] "Generic (PLEG): container finished" podID="588ccf9a-db41-45b7-a2d7-be18dfde096e" containerID="681b52304e085676252963dd556f4a0bc5918e190d80add968e58d0c87236de9" exitCode=0
Oct 30 21:23:03.209840 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:03.209410    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f9eff-ncvzk" event={"ID":"588ccf9a-db41-45b7-a2d7-be18dfde096e","Type":"ContainerDied","Data":"681b52304e085676252963dd556f4a0bc5918e190d80add968e58d0c87236de9"}
Oct 30 21:23:04.375569 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:04.375543    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f9eff-ncvzk"
Oct 30 21:23:04.483783 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:04.483699    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/588ccf9a-db41-45b7-a2d7-be18dfde096e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/588ccf9a-db41-45b7-a2d7-be18dfde096e-maas-api-openshift-ca\") pod \"588ccf9a-db41-45b7-a2d7-be18dfde096e\" (UID: \"588ccf9a-db41-45b7-a2d7-be18dfde096e\") "
Oct 30 21:23:04.483981 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:04.483855    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/588ccf9a-db41-45b7-a2d7-be18dfde096e-kube-api-access-87977\" (UniqueName: \"kubernetes.io/projected/588ccf9a-db41-45b7-a2d7-be18dfde096e-kube-api-access-87977\") pod \"588ccf9a-db41-45b7-a2d7-be18dfde096e\" (UID: \"588ccf9a-db41-45b7-a2d7-be18dfde096e\") "
Oct 30 21:23:04.483981 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:04.483920    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/588ccf9a-db41-45b7-a2d7-be18dfde096e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/588ccf9a-db41-45b7-a2d7-be18dfde096e-maas-api-xks-ca\") pod \"588ccf9a-db41-45b7-a2d7-be18dfde096e\" (UID: \"588ccf9a-db41-45b7-a2d7-be18dfde096e\") "
Oct 30 21:23:04.484186 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:04.484159    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/588ccf9a-db41-45b7-a2d7-be18dfde096e-maas-api-openshift-ca" pod "588ccf9a-db41-45b7-a2d7-be18dfde096e" (UID: "588ccf9a-db41-45b7-a2d7-be18dfde096e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:04.484533 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:04.484515    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/588ccf9a-db41-45b7-a2d7-be18dfde096e-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:23:04.486244 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:04.486212    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/588ccf9a-db41-45b7-a2d7-be18dfde096e-maas-api-xks-ca" pod "588ccf9a-db41-45b7-a2d7-be18dfde096e" (UID: "588ccf9a-db41-45b7-a2d7-be18dfde096e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:23:04.486378 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:04.486295    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/588ccf9a-db41-45b7-a2d7-be18dfde096e-kube-api-access-87977" pod "588ccf9a-db41-45b7-a2d7-be18dfde096e" (UID: "588ccf9a-db41-45b7-a2d7-be18dfde096e"). InnerVolumeSpecName "kube-api-access-87977". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:04.585622 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:04.585576    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87977\" (UniqueName: \"kubernetes.io/projected/588ccf9a-db41-45b7-a2d7-be18dfde096e-kube-api-access-87977\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:23:04.585622 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:04.585616    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/588ccf9a-db41-45b7-a2d7-be18dfde096e-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:23:05.226528 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:05.226496    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f9eff-ncvzk" event={"ID":"588ccf9a-db41-45b7-a2d7-be18dfde096e","Type":"ContainerDied","Data":"fb2ea1972d3686bf257dbe2c1e4f4bd449b2a78f4d31fd9ed5fdb929b2f7d989"}
Oct 30 21:23:05.226709 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:05.226546    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb2ea1972d3686bf257dbe2c1e4f4bd449b2a78f4d31fd9ed5fdb929b2f7d989"
Oct 30 21:23:05.226709 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:05.226506    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f9eff-ncvzk"
Oct 30 21:23:08.044420 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:08.044387    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-103ce74c-hcszl"]
Oct 30 21:23:08.094202 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:08.094170    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-103ce74c-hcszl"]
Oct 30 21:23:08.985919 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:08.985888    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq"]
Oct 30 21:23:08.986461 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:08.986447    5713 cpu_manager.go:414] "containerMap: removing container" podUID="588ccf9a-db41-45b7-a2d7-be18dfde096e" containerName="revoke-keys"
Oct 30 21:23:08.986508 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:08.986463    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="588ccf9a-db41-45b7-a2d7-be18dfde096e" containerName="revoke-keys"
Oct 30 21:23:08.986600 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:08.986591    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="588ccf9a-db41-45b7-a2d7-be18dfde096e" containerName="revoke-keys"
Oct 30 21:23:08.989976 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:08.989955    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq"
Oct 30 21:23:09.052359 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:09.052324    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a34d3155-xkmp5"]
Oct 30 21:23:09.064900 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:09.064863    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ddb1527-fa02-4407-a861-5e850c65f7ae" path="/var/lib/kubelet/pods/2ddb1527-fa02-4407-a861-5e850c65f7ae/volumes"
Oct 30 21:23:09.097298 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:09.097260    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a34d3155-xkmp5"]
Oct 30 21:23:09.141991 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:09.141955    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21edc9eb-88e3-495f-870c-925863b09ce1-istio-token\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq\" (UID: \"21edc9eb-88e3-495f-870c-925863b09ce1\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq"
Oct 30 21:23:09.142180 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:09.142007    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21edc9eb-88e3-495f-870c-925863b09ce1-workload-certs\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq\" (UID: \"21edc9eb-88e3-495f-870c-925863b09ce1\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq"
Oct 30 21:23:09.142180 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:09.142030    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21edc9eb-88e3-495f-870c-925863b09ce1-istio-envoy\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq\" (UID: \"21edc9eb-88e3-495f-870c-925863b09ce1\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq"
Oct 30 21:23:09.142295 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:09.142172    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21edc9eb-88e3-495f-870c-925863b09ce1-workload-socket\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq\" (UID: \"21edc9eb-88e3-495f-870c-925863b09ce1\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq"
Oct 30 21:23:09.142295 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:09.142266    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21edc9eb-88e3-495f-870c-925863b09ce1-istio-data\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq\" (UID: \"21edc9eb-88e3-495f-870c-925863b09ce1\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq"
Oct 30 21:23:09.142392 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:09.142299    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21edc9eb-88e3-495f-870c-925863b09ce1-credential-socket\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq\" (UID: \"21edc9eb-88e3-495f-870c-925863b09ce1\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq"
Oct 30 21:23:09.142392 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:09.142340    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21edc9eb-88e3-495f-870c-925863b09ce1-istiod-ca-cert\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq\" (UID: \"21edc9eb-88e3-495f-870c-925863b09ce1\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq"
Oct 30 21:23:09.142506 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:09.142412    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21edc9eb-88e3-495f-870c-925863b09ce1-istio-podinfo\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq\" (UID: \"21edc9eb-88e3-495f-870c-925863b09ce1\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq"
Oct 30 21:23:09.142565 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:09.142496    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqvqb\" (UniqueName: \"kubernetes.io/projected/21edc9eb-88e3-495f-870c-925863b09ce1-kube-api-access-vqvqb\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq\" (UID: \"21edc9eb-88e3-495f-870c-925863b09ce1\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq"
Oct 30 21:23:09.191622 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:09.191580    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq"]
Oct 30 21:23:09.243827 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:09.243695    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21edc9eb-88e3-495f-870c-925863b09ce1-istio-podinfo\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq\" (UID: \"21edc9eb-88e3-495f-870c-925863b09ce1\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq"
Oct 30 21:23:09.243827 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:09.243750    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqvqb\" (UniqueName: \"kubernetes.io/projected/21edc9eb-88e3-495f-870c-925863b09ce1-kube-api-access-vqvqb\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq\" (UID: \"21edc9eb-88e3-495f-870c-925863b09ce1\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq"
Oct 30 21:23:09.244096 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:09.243875    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21edc9eb-88e3-495f-870c-925863b09ce1-istio-token\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq\" (UID: \"21edc9eb-88e3-495f-870c-925863b09ce1\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq"
Oct 30 21:23:09.244096 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:09.243910    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21edc9eb-88e3-495f-870c-925863b09ce1-workload-certs\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq\" (UID: \"21edc9eb-88e3-495f-870c-925863b09ce1\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq"
Oct 30 21:23:09.244096 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:09.243928    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21edc9eb-88e3-495f-870c-925863b09ce1-istio-envoy\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq\" (UID: \"21edc9eb-88e3-495f-870c-925863b09ce1\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq"
Oct 30 21:23:09.244096 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:09.243983    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21edc9eb-88e3-495f-870c-925863b09ce1-workload-socket\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq\" (UID: \"21edc9eb-88e3-495f-870c-925863b09ce1\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq"
Oct 30 21:23:09.244096 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:09.244082    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21edc9eb-88e3-495f-870c-925863b09ce1-istio-data\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq\" (UID: \"21edc9eb-88e3-495f-870c-925863b09ce1\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq"
Oct 30 21:23:09.244392 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:09.244109    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21edc9eb-88e3-495f-870c-925863b09ce1-credential-socket\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq\" (UID: \"21edc9eb-88e3-495f-870c-925863b09ce1\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq"
Oct 30 21:23:09.244392 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:09.244153    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21edc9eb-88e3-495f-870c-925863b09ce1-istiod-ca-cert\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq\" (UID: \"21edc9eb-88e3-495f-870c-925863b09ce1\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq"
Oct 30 21:23:09.244392 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:09.244337    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21edc9eb-88e3-495f-870c-925863b09ce1-workload-certs\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq\" (UID: \"21edc9eb-88e3-495f-870c-925863b09ce1\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq"
Oct 30 21:23:09.244560 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:09.244429    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21edc9eb-88e3-495f-870c-925863b09ce1-workload-socket\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq\" (UID: \"21edc9eb-88e3-495f-870c-925863b09ce1\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq"
Oct 30 21:23:09.244560 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:09.244481    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21edc9eb-88e3-495f-870c-925863b09ce1-istio-data\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq\" (UID: \"21edc9eb-88e3-495f-870c-925863b09ce1\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq"
Oct 30 21:23:09.244735 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:09.244595    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21edc9eb-88e3-495f-870c-925863b09ce1-credential-socket\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq\" (UID: \"21edc9eb-88e3-495f-870c-925863b09ce1\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq"
Oct 30 21:23:09.245117 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:09.245094    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21edc9eb-88e3-495f-870c-925863b09ce1-istiod-ca-cert\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq\" (UID: \"21edc9eb-88e3-495f-870c-925863b09ce1\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq"
Oct 30 21:23:09.246412 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:09.246387    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21edc9eb-88e3-495f-870c-925863b09ce1-istio-envoy\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq\" (UID: \"21edc9eb-88e3-495f-870c-925863b09ce1\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq"
Oct 30 21:23:09.246527 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:09.246507    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21edc9eb-88e3-495f-870c-925863b09ce1-istio-podinfo\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq\" (UID: \"21edc9eb-88e3-495f-870c-925863b09ce1\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq"
Oct 30 21:23:09.301575 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:09.301539    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21edc9eb-88e3-495f-870c-925863b09ce1-istio-token\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq\" (UID: \"21edc9eb-88e3-495f-870c-925863b09ce1\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq"
Oct 30 21:23:09.301734 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:09.301607    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqvqb\" (UniqueName: \"kubernetes.io/projected/21edc9eb-88e3-495f-870c-925863b09ce1-kube-api-access-vqvqb\") pod \"e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq\" (UID: \"21edc9eb-88e3-495f-870c-925863b09ce1\") " pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq"
Oct 30 21:23:09.304445 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:09.304413    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq"
Oct 30 21:23:09.762881 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:09.762854    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq"]
Oct 30 21:23:09.763025 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:23:09.762995    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21edc9eb_88e3_495f_870c_925863b09ce1.slice/crio-6f231fd8564319bc84e7683bde5056a73d2c540d0a1dcd6744804176fec6ced1 WatchSource:0}: Error finding container 6f231fd8564319bc84e7683bde5056a73d2c540d0a1dcd6744804176fec6ced1: Status 404 returned error can't find the container with id 6f231fd8564319bc84e7683bde5056a73d2c540d0a1dcd6744804176fec6ced1
Oct 30 21:23:09.765603 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:09.765565    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:23:09.765730 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:09.765648    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:23:09.765730 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:09.765696    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:23:10.265635 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:10.265586    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq" event={"ID":"21edc9eb-88e3-495f-870c-925863b09ce1","Type":"ContainerStarted","Data":"29e2fc6a89bc0ae81bbdde2d775e4af2599b92161def2347fffd75f647cc22a7"}
Oct 30 21:23:10.266305 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:10.265652    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq" event={"ID":"21edc9eb-88e3-495f-870c-925863b09ce1","Type":"ContainerStarted","Data":"6f231fd8564319bc84e7683bde5056a73d2c540d0a1dcd6744804176fec6ced1"}
Oct 30 21:23:10.305249 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:10.305200    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq"
Oct 30 21:23:10.308994 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:10.308946    5713 patch_prober.go:29] interesting pod/e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body=
Oct 30 21:23:10.309171 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:10.309020    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq" podUID="21edc9eb-88e3-495f-870c-925863b09ce1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused"
Oct 30 21:23:10.384322 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:10.384260    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq" podStartSLOduration=2.384240278 podStartE2EDuration="2.384240278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:10.382577097 +0000 UTC m=+2673.447353489" watchObservedRunningTime="2026-10-30 21:23:10.384240278 +0000 UTC m=+2673.449016676"
Oct 30 21:23:11.068083 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:11.068041    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ca575a1-4c51-4967-9798-fc18d5a7752d" path="/var/lib/kubelet/pods/4ca575a1-4c51-4967-9798-fc18d5a7752d/volumes"
Oct 30 21:23:11.305813 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:11.304981    5713 patch_prober.go:29] interesting pod/e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body=
Oct 30 21:23:11.305813 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:11.305086    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq" podUID="21edc9eb-88e3-495f-870c-925863b09ce1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused"
Oct 30 21:23:12.305081 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:12.305051    5713 patch_prober.go:29] interesting pod/e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body=
Oct 30 21:23:12.305250 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:12.305100    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq" podUID="21edc9eb-88e3-495f-870c-925863b09ce1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused"
Oct 30 21:23:12.409488 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:12.409449    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2f9eff-689ff796b8-sf94t"]
Oct 30 21:23:12.519981 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:12.519946    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2f9eff-7796bfd547-fkj5q"]
Oct 30 21:23:13.305164 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:13.305116    5713 patch_prober.go:29] interesting pod/e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body=
Oct 30 21:23:13.305360 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:13.305182    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq" podUID="21edc9eb-88e3-495f-870c-925863b09ce1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused"
Oct 30 21:23:14.305009 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:14.304970    5713 patch_prober.go:29] interesting pod/e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body=
Oct 30 21:23:14.305512 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:14.305049    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq" podUID="21edc9eb-88e3-495f-870c-925863b09ce1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused"
Oct 30 21:23:15.304988 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:15.304938    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-08df98/test-model-08df98-kserve-5d5c94f6c7-4cqkf" podUID="f8861d60-e55d-41cc-9d3e-114ce28bbe01" containerName="main" containerID="cri-o://1e2fde808bb29c728bbae00d0f2e78fab595ba8b8df38afcff2de9e27d236583" gracePeriod=45
Oct 30 21:23:15.305188 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:15.304961    5713 patch_prober.go:29] interesting pod/e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body=
Oct 30 21:23:15.305188 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:15.305093    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq" podUID="21edc9eb-88e3-495f-870c-925863b09ce1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused"
Oct 30 21:23:15.459426 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:15.459376    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-08df98/test-model-08df98-kserve-5d5c94f6c7-4cqkf" podUID="f8861d60-e55d-41cc-9d3e-114ce28bbe01" containerName="main" probeResult="failure" output="Get \"https://10.134.0.217:8000/ready\": dial tcp 10.134.0.217:8000: connect: connection refused"
Oct 30 21:23:15.626088 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:15.626061    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-08df98/test-model-08df98-kserve-5d5c94f6c7-4cqkf"
Oct 30 21:23:15.757401 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:15.757364    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8861d60-e55d-41cc-9d3e-114ce28bbe01-tls-certs\" (UniqueName: \"kubernetes.io/secret/f8861d60-e55d-41cc-9d3e-114ce28bbe01-tls-certs\") pod \"f8861d60-e55d-41cc-9d3e-114ce28bbe01\" (UID: \"f8861d60-e55d-41cc-9d3e-114ce28bbe01\") "
Oct 30 21:23:15.758920 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:15.758150    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8861d60-e55d-41cc-9d3e-114ce28bbe01-kube-api-access-xqnrb\" (UniqueName: \"kubernetes.io/projected/f8861d60-e55d-41cc-9d3e-114ce28bbe01-kube-api-access-xqnrb\") pod \"f8861d60-e55d-41cc-9d3e-114ce28bbe01\" (UID: \"f8861d60-e55d-41cc-9d3e-114ce28bbe01\") "
Oct 30 21:23:15.758920 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:15.758246    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8861d60-e55d-41cc-9d3e-114ce28bbe01-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8861d60-e55d-41cc-9d3e-114ce28bbe01-dshm\") pod \"f8861d60-e55d-41cc-9d3e-114ce28bbe01\" (UID: \"f8861d60-e55d-41cc-9d3e-114ce28bbe01\") "
Oct 30 21:23:15.758920 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:15.758298    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8861d60-e55d-41cc-9d3e-114ce28bbe01-home\" (UniqueName: \"kubernetes.io/empty-dir/f8861d60-e55d-41cc-9d3e-114ce28bbe01-home\") pod \"f8861d60-e55d-41cc-9d3e-114ce28bbe01\" (UID: \"f8861d60-e55d-41cc-9d3e-114ce28bbe01\") "
Oct 30 21:23:15.758920 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:15.758330    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8861d60-e55d-41cc-9d3e-114ce28bbe01-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8861d60-e55d-41cc-9d3e-114ce28bbe01-model-cache\") pod \"f8861d60-e55d-41cc-9d3e-114ce28bbe01\" (UID: \"f8861d60-e55d-41cc-9d3e-114ce28bbe01\") "
Oct 30 21:23:15.758920 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:15.758381    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8861d60-e55d-41cc-9d3e-114ce28bbe01-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8861d60-e55d-41cc-9d3e-114ce28bbe01-tmp-dir\") pod \"f8861d60-e55d-41cc-9d3e-114ce28bbe01\" (UID: \"f8861d60-e55d-41cc-9d3e-114ce28bbe01\") "
Oct 30 21:23:15.759390 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:15.759360    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8861d60-e55d-41cc-9d3e-114ce28bbe01-tmp-dir" pod "f8861d60-e55d-41cc-9d3e-114ce28bbe01" (UID: "f8861d60-e55d-41cc-9d3e-114ce28bbe01"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:15.759606 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:15.759581    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8861d60-e55d-41cc-9d3e-114ce28bbe01-model-cache" pod "f8861d60-e55d-41cc-9d3e-114ce28bbe01" (UID: "f8861d60-e55d-41cc-9d3e-114ce28bbe01"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:15.759689 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:15.759385    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8861d60-e55d-41cc-9d3e-114ce28bbe01-home" pod "f8861d60-e55d-41cc-9d3e-114ce28bbe01" (UID: "f8861d60-e55d-41cc-9d3e-114ce28bbe01"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:15.761666 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:15.761606    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8861d60-e55d-41cc-9d3e-114ce28bbe01-kube-api-access-xqnrb" pod "f8861d60-e55d-41cc-9d3e-114ce28bbe01" (UID: "f8861d60-e55d-41cc-9d3e-114ce28bbe01"). InnerVolumeSpecName "kube-api-access-xqnrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:15.761962 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:15.761939    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8861d60-e55d-41cc-9d3e-114ce28bbe01-tls-certs" pod "f8861d60-e55d-41cc-9d3e-114ce28bbe01" (UID: "f8861d60-e55d-41cc-9d3e-114ce28bbe01"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:23:15.762047 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:15.761996    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8861d60-e55d-41cc-9d3e-114ce28bbe01-dshm" pod "f8861d60-e55d-41cc-9d3e-114ce28bbe01" (UID: "f8861d60-e55d-41cc-9d3e-114ce28bbe01"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:15.860057 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:15.859961    5713 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8861d60-e55d-41cc-9d3e-114ce28bbe01-tls-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:23:15.860057 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:15.860004    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqnrb\" (UniqueName: \"kubernetes.io/projected/f8861d60-e55d-41cc-9d3e-114ce28bbe01-kube-api-access-xqnrb\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:23:15.860057 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:15.860015    5713 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8861d60-e55d-41cc-9d3e-114ce28bbe01-dshm\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:23:15.860057 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:15.860024    5713 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8861d60-e55d-41cc-9d3e-114ce28bbe01-home\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:23:15.860057 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:15.860032    5713 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8861d60-e55d-41cc-9d3e-114ce28bbe01-model-cache\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:23:15.860057 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:15.860059    5713 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8861d60-e55d-41cc-9d3e-114ce28bbe01-tmp-dir\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:23:15.988640 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:15.988602    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c69b6088-mtrx4"]
Oct 30 21:23:15.989260 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:15.989240    5713 cpu_manager.go:414] "containerMap: removing container" podUID="f8861d60-e55d-41cc-9d3e-114ce28bbe01" containerName="main"
Oct 30 21:23:15.989384 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:15.989261    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8861d60-e55d-41cc-9d3e-114ce28bbe01" containerName="main"
Oct 30 21:23:15.989472 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:15.989457    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8861d60-e55d-41cc-9d3e-114ce28bbe01" containerName="main"
Oct 30 21:23:15.992947 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:15.992926    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69b6088-mtrx4"
Oct 30 21:23:16.046119 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:16.046074    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:23:16.046119 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:16.046076    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:23:16.046392 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:16.046162    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:23:16.046392 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:16.046264    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:23:16.049681 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:16.049658    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c69b6088-mtrx4"]
Oct 30 21:23:16.061749 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:16.061722    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4ecc41c-33fd-4558-bf79-e4ef840c0aa1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c69b6088-mtrx4\" (UID: \"b4ecc41c-33fd-4558-bf79-e4ef840c0aa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69b6088-mtrx4"
Oct 30 21:23:16.061945 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:16.061928    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4ecc41c-33fd-4558-bf79-e4ef840c0aa1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c69b6088-mtrx4\" (UID: \"b4ecc41c-33fd-4558-bf79-e4ef840c0aa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69b6088-mtrx4"
Oct 30 21:23:16.062034 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:16.061987    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh66r\" (UniqueName: \"kubernetes.io/projected/b4ecc41c-33fd-4558-bf79-e4ef840c0aa1-kube-api-access-gh66r\") pod \"maas-api-revoke-sub-c69b6088-mtrx4\" (UID: \"b4ecc41c-33fd-4558-bf79-e4ef840c0aa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69b6088-mtrx4"
Oct 30 21:23:16.163718 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:16.162945    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4ecc41c-33fd-4558-bf79-e4ef840c0aa1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c69b6088-mtrx4\" (UID: \"b4ecc41c-33fd-4558-bf79-e4ef840c0aa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69b6088-mtrx4"
Oct 30 21:23:16.163718 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:16.163056    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh66r\" (UniqueName: \"kubernetes.io/projected/b4ecc41c-33fd-4558-bf79-e4ef840c0aa1-kube-api-access-gh66r\") pod \"maas-api-revoke-sub-c69b6088-mtrx4\" (UID: \"b4ecc41c-33fd-4558-bf79-e4ef840c0aa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69b6088-mtrx4"
Oct 30 21:23:16.163718 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:16.163120    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4ecc41c-33fd-4558-bf79-e4ef840c0aa1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c69b6088-mtrx4\" (UID: \"b4ecc41c-33fd-4558-bf79-e4ef840c0aa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69b6088-mtrx4"
Oct 30 21:23:16.164180 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:16.164089    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4ecc41c-33fd-4558-bf79-e4ef840c0aa1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c69b6088-mtrx4\" (UID: \"b4ecc41c-33fd-4558-bf79-e4ef840c0aa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69b6088-mtrx4"
Oct 30 21:23:16.165999 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:16.165969    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4ecc41c-33fd-4558-bf79-e4ef840c0aa1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c69b6088-mtrx4\" (UID: \"b4ecc41c-33fd-4558-bf79-e4ef840c0aa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69b6088-mtrx4"
Oct 30 21:23:16.219739 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:16.219705    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh66r\" (UniqueName: \"kubernetes.io/projected/b4ecc41c-33fd-4558-bf79-e4ef840c0aa1-kube-api-access-gh66r\") pod \"maas-api-revoke-sub-c69b6088-mtrx4\" (UID: \"b4ecc41c-33fd-4558-bf79-e4ef840c0aa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69b6088-mtrx4"
Oct 30 21:23:16.307882 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:16.307850    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69b6088-mtrx4"
Oct 30 21:23:16.336491 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:16.336452    5713 generic.go:356] "Generic (PLEG): container finished" podID="f8861d60-e55d-41cc-9d3e-114ce28bbe01" containerID="1e2fde808bb29c728bbae00d0f2e78fab595ba8b8df38afcff2de9e27d236583" exitCode=0
Oct 30 21:23:16.336652 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:16.336514    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-08df98/test-model-08df98-kserve-5d5c94f6c7-4cqkf"
Oct 30 21:23:16.336652 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:16.336518    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-08df98/test-model-08df98-kserve-5d5c94f6c7-4cqkf" event={"ID":"f8861d60-e55d-41cc-9d3e-114ce28bbe01","Type":"ContainerDied","Data":"1e2fde808bb29c728bbae00d0f2e78fab595ba8b8df38afcff2de9e27d236583"}
Oct 30 21:23:16.336652 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:16.336566    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-08df98/test-model-08df98-kserve-5d5c94f6c7-4cqkf" event={"ID":"f8861d60-e55d-41cc-9d3e-114ce28bbe01","Type":"ContainerDied","Data":"ab998f1583aa120cee557f86a8e3fff0827f10c9326ddea4bea599d9900ed4c1"}
Oct 30 21:23:16.336652 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:16.336589    5713 scope.go:122] "RemoveContainer" containerID="1e2fde808bb29c728bbae00d0f2e78fab595ba8b8df38afcff2de9e27d236583"
Oct 30 21:23:16.351487 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:16.351450    5713 scope.go:122] "RemoveContainer" containerID="1e2fde808bb29c728bbae00d0f2e78fab595ba8b8df38afcff2de9e27d236583"
Oct 30 21:23:16.351861 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:23:16.351840    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e2fde808bb29c728bbae00d0f2e78fab595ba8b8df38afcff2de9e27d236583\": container with ID starting with 1e2fde808bb29c728bbae00d0f2e78fab595ba8b8df38afcff2de9e27d236583 not found: ID does not exist" containerID="1e2fde808bb29c728bbae00d0f2e78fab595ba8b8df38afcff2de9e27d236583"
Oct 30 21:23:16.351943 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:16.351871    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e2fde808bb29c728bbae00d0f2e78fab595ba8b8df38afcff2de9e27d236583"} err="failed to get container status \"1e2fde808bb29c728bbae00d0f2e78fab595ba8b8df38afcff2de9e27d236583\": rpc error: code = NotFound desc = could not find container \"1e2fde808bb29c728bbae00d0f2e78fab595ba8b8df38afcff2de9e27d236583\": container with ID starting with 1e2fde808bb29c728bbae00d0f2e78fab595ba8b8df38afcff2de9e27d236583 not found: ID does not exist"
Oct 30 21:23:16.373435 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:16.373405    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq"
Oct 30 21:23:16.373978 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:16.373919    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq"
Oct 30 21:23:16.376746 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:16.376726    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq"
Oct 30 21:23:16.623424 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:16.623391    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-08df98/test-model-08df98-kserve-5d5c94f6c7-4cqkf"]
Oct 30 21:23:16.628547 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:16.628520    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-08df98/test-model-08df98-kserve-5d5c94f6c7-4cqkf"]
Oct 30 21:23:16.696990 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:16.696963    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c69b6088-mtrx4"]
Oct 30 21:23:16.698642 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:23:16.698617    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4ecc41c_33fd_4558_bf79_e4ef840c0aa1.slice/crio-36fe5ad74baae687d0a39404713cb5dee35a0296106eb786b43b8e0105b029e7 WatchSource:0}: Error finding container 36fe5ad74baae687d0a39404713cb5dee35a0296106eb786b43b8e0105b029e7: Status 404 returned error can't find the container with id 36fe5ad74baae687d0a39404713cb5dee35a0296106eb786b43b8e0105b029e7
Oct 30 21:23:17.063905 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:17.063874    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8861d60-e55d-41cc-9d3e-114ce28bbe01" path="/var/lib/kubelet/pods/f8861d60-e55d-41cc-9d3e-114ce28bbe01/volumes"
Oct 30 21:23:17.345668 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:17.345581    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69b6088-mtrx4" event={"ID":"b4ecc41c-33fd-4558-bf79-e4ef840c0aa1","Type":"ContainerStarted","Data":"b2d822dd8135f76a5d3a34e1e43818b80a0a83e4c3fe5afb78dc9af238c877ed"}
Oct 30 21:23:17.345668 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:17.345643    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69b6088-mtrx4" event={"ID":"b4ecc41c-33fd-4558-bf79-e4ef840c0aa1","Type":"ContainerStarted","Data":"36fe5ad74baae687d0a39404713cb5dee35a0296106eb786b43b8e0105b029e7"}
Oct 30 21:23:17.410152 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:17.410110    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-2f9eff-689ff796b8-sf94t" podUID="102176e5-2bda-4e24-b86a-0d50fba75ac3" containerName="payload-processing" containerID="cri-o://14f252049ea00e1ae4dae8092ecc9b3f8a97b53841993edef17577a08d6ee1b2" gracePeriod=25
Oct 30 21:23:17.464576 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:17.464519    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69b6088-mtrx4" podStartSLOduration=2.464505432 podStartE2EDuration="2.464505432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:17.461226332 +0000 UTC m=+2680.526002731" watchObservedRunningTime="2026-10-30 21:23:17.464505432 +0000 UTC m=+2680.529281830"
Oct 30 21:23:17.520475 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:17.520424    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f9eff-7796bfd547-fkj5q" podUID="51a91785-6750-4b1a-8174-4598816ea82d" containerName="payload-pre-processing" containerID="cri-o://0ca83bbf41038832964465ad63185b5410f84cabaf8611b8c8678222565da7f7" gracePeriod=25
Oct 30 21:23:17.672433 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:17.672406    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2f9eff-689ff796b8-sf94t"
Oct 30 21:23:17.679713 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:17.679686    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/102176e5-2bda-4e24-b86a-0d50fba75ac3-kube-api-access-zd5qq\" (UniqueName: \"kubernetes.io/projected/102176e5-2bda-4e24-b86a-0d50fba75ac3-kube-api-access-zd5qq\") pod \"102176e5-2bda-4e24-b86a-0d50fba75ac3\" (UID: \"102176e5-2bda-4e24-b86a-0d50fba75ac3\") "
Oct 30 21:23:17.679925 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:17.679761    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/102176e5-2bda-4e24-b86a-0d50fba75ac3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/102176e5-2bda-4e24-b86a-0d50fba75ac3-plugins-config-volume\") pod \"102176e5-2bda-4e24-b86a-0d50fba75ac3\" (UID: \"102176e5-2bda-4e24-b86a-0d50fba75ac3\") "
Oct 30 21:23:17.680185 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:17.680161    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/102176e5-2bda-4e24-b86a-0d50fba75ac3-plugins-config-volume" pod "102176e5-2bda-4e24-b86a-0d50fba75ac3" (UID: "102176e5-2bda-4e24-b86a-0d50fba75ac3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:17.680357 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:17.680340    5713 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/102176e5-2bda-4e24-b86a-0d50fba75ac3-plugins-config-volume\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:23:17.682037 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:17.682012    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/102176e5-2bda-4e24-b86a-0d50fba75ac3-kube-api-access-zd5qq" pod "102176e5-2bda-4e24-b86a-0d50fba75ac3" (UID: "102176e5-2bda-4e24-b86a-0d50fba75ac3"). InnerVolumeSpecName "kube-api-access-zd5qq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:17.781592 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:17.781545    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zd5qq\" (UniqueName: \"kubernetes.io/projected/102176e5-2bda-4e24-b86a-0d50fba75ac3-kube-api-access-zd5qq\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:23:18.355691 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:18.355582    5713 generic.go:356] "Generic (PLEG): container finished" podID="b4ecc41c-33fd-4558-bf79-e4ef840c0aa1" containerID="b2d822dd8135f76a5d3a34e1e43818b80a0a83e4c3fe5afb78dc9af238c877ed" exitCode=0
Oct 30 21:23:18.355691 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:18.355681    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69b6088-mtrx4" event={"ID":"b4ecc41c-33fd-4558-bf79-e4ef840c0aa1","Type":"ContainerDied","Data":"b2d822dd8135f76a5d3a34e1e43818b80a0a83e4c3fe5afb78dc9af238c877ed"}
Oct 30 21:23:18.357434 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:18.357405    5713 generic.go:356] "Generic (PLEG): container finished" podID="102176e5-2bda-4e24-b86a-0d50fba75ac3" containerID="14f252049ea00e1ae4dae8092ecc9b3f8a97b53841993edef17577a08d6ee1b2" exitCode=0
Oct 30 21:23:18.357604 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:18.357450    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2f9eff-689ff796b8-sf94t"
Oct 30 21:23:18.357604 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:18.357488    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2f9eff-689ff796b8-sf94t" event={"ID":"102176e5-2bda-4e24-b86a-0d50fba75ac3","Type":"ContainerDied","Data":"14f252049ea00e1ae4dae8092ecc9b3f8a97b53841993edef17577a08d6ee1b2"}
Oct 30 21:23:18.357604 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:18.357534    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2f9eff-689ff796b8-sf94t" event={"ID":"102176e5-2bda-4e24-b86a-0d50fba75ac3","Type":"ContainerDied","Data":"2283a57982a9589e62767c12d0dd42433c9cf89231e50ab0a898c2e5e2965d34"}
Oct 30 21:23:18.357604 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:18.357562    5713 scope.go:122] "RemoveContainer" containerID="14f252049ea00e1ae4dae8092ecc9b3f8a97b53841993edef17577a08d6ee1b2"
Oct 30 21:23:18.368416 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:18.368388    5713 scope.go:122] "RemoveContainer" containerID="14f252049ea00e1ae4dae8092ecc9b3f8a97b53841993edef17577a08d6ee1b2"
Oct 30 21:23:18.368710 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:23:18.368689    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14f252049ea00e1ae4dae8092ecc9b3f8a97b53841993edef17577a08d6ee1b2\": container with ID starting with 14f252049ea00e1ae4dae8092ecc9b3f8a97b53841993edef17577a08d6ee1b2 not found: ID does not exist" containerID="14f252049ea00e1ae4dae8092ecc9b3f8a97b53841993edef17577a08d6ee1b2"
Oct 30 21:23:18.368762 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:18.368719    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14f252049ea00e1ae4dae8092ecc9b3f8a97b53841993edef17577a08d6ee1b2"} err="failed to get container status \"14f252049ea00e1ae4dae8092ecc9b3f8a97b53841993edef17577a08d6ee1b2\": rpc error: code = NotFound desc = could not find container \"14f252049ea00e1ae4dae8092ecc9b3f8a97b53841993edef17577a08d6ee1b2\": container with ID starting with 14f252049ea00e1ae4dae8092ecc9b3f8a97b53841993edef17577a08d6ee1b2 not found: ID does not exist"
Oct 30 21:23:18.580849 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:18.580808    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2f9eff-689ff796b8-sf94t"]
Oct 30 21:23:18.584242 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:18.584211    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2f9eff-689ff796b8-sf94t"]
Oct 30 21:23:19.063487 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:19.063458    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="102176e5-2bda-4e24-b86a-0d50fba75ac3" path="/var/lib/kubelet/pods/102176e5-2bda-4e24-b86a-0d50fba75ac3/volumes"
Oct 30 21:23:19.518003 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:19.517976    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69b6088-mtrx4"
Oct 30 21:23:19.605835 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:19.605805    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4ecc41c-33fd-4558-bf79-e4ef840c0aa1-kube-api-access-gh66r\" (UniqueName: \"kubernetes.io/projected/b4ecc41c-33fd-4558-bf79-e4ef840c0aa1-kube-api-access-gh66r\") pod \"b4ecc41c-33fd-4558-bf79-e4ef840c0aa1\" (UID: \"b4ecc41c-33fd-4558-bf79-e4ef840c0aa1\") "
Oct 30 21:23:19.606045 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:19.605974    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4ecc41c-33fd-4558-bf79-e4ef840c0aa1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4ecc41c-33fd-4558-bf79-e4ef840c0aa1-maas-api-openshift-ca\") pod \"b4ecc41c-33fd-4558-bf79-e4ef840c0aa1\" (UID: \"b4ecc41c-33fd-4558-bf79-e4ef840c0aa1\") "
Oct 30 21:23:19.606133 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:19.606119    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4ecc41c-33fd-4558-bf79-e4ef840c0aa1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4ecc41c-33fd-4558-bf79-e4ef840c0aa1-maas-api-xks-ca\") pod \"b4ecc41c-33fd-4558-bf79-e4ef840c0aa1\" (UID: \"b4ecc41c-33fd-4558-bf79-e4ef840c0aa1\") "
Oct 30 21:23:19.606337 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:19.606313    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4ecc41c-33fd-4558-bf79-e4ef840c0aa1-maas-api-openshift-ca" pod "b4ecc41c-33fd-4558-bf79-e4ef840c0aa1" (UID: "b4ecc41c-33fd-4558-bf79-e4ef840c0aa1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:19.606615 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:19.606598    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4ecc41c-33fd-4558-bf79-e4ef840c0aa1-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:23:19.608115 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:19.608088    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4ecc41c-33fd-4558-bf79-e4ef840c0aa1-kube-api-access-gh66r" pod "b4ecc41c-33fd-4558-bf79-e4ef840c0aa1" (UID: "b4ecc41c-33fd-4558-bf79-e4ef840c0aa1"). InnerVolumeSpecName "kube-api-access-gh66r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:19.608384 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:19.608363    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4ecc41c-33fd-4558-bf79-e4ef840c0aa1-maas-api-xks-ca" pod "b4ecc41c-33fd-4558-bf79-e4ef840c0aa1" (UID: "b4ecc41c-33fd-4558-bf79-e4ef840c0aa1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:23:19.708545 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:19.708484    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4ecc41c-33fd-4558-bf79-e4ef840c0aa1-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:23:19.708545 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:19.708538    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gh66r\" (UniqueName: \"kubernetes.io/projected/b4ecc41c-33fd-4558-bf79-e4ef840c0aa1-kube-api-access-gh66r\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:23:20.374776 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:20.374744    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69b6088-mtrx4"
Oct 30 21:23:20.374776 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:20.374769    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69b6088-mtrx4" event={"ID":"b4ecc41c-33fd-4558-bf79-e4ef840c0aa1","Type":"ContainerDied","Data":"36fe5ad74baae687d0a39404713cb5dee35a0296106eb786b43b8e0105b029e7"}
Oct 30 21:23:20.375040 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:20.374837    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36fe5ad74baae687d0a39404713cb5dee35a0296106eb786b43b8e0105b029e7"
Oct 30 21:23:22.158319 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:22.158279    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f9eff-7796bfd547-fkj5q" podUID="51a91785-6750-4b1a-8174-4598816ea82d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:23:24.460034 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:23:24.459986    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:23:30.355626 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:30.355592    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f9eff-ncvzk"]
Oct 30 21:23:30.405093 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:30.405064    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f9eff-ncvzk"]
Oct 30 21:23:31.063136 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:31.063099    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="588ccf9a-db41-45b7-a2d7-be18dfde096e" path="/var/lib/kubelet/pods/588ccf9a-db41-45b7-a2d7-be18dfde096e/volumes"
Oct 30 21:23:32.158370 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:32.158331    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f9eff-7796bfd547-fkj5q" podUID="51a91785-6750-4b1a-8174-4598816ea82d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:23:36.055184 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:36.055144    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac25069b-rb2gz"]
Oct 30 21:23:36.105272 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:36.105240    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f4cd36b-555c2"]
Oct 30 21:23:36.108150 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:36.108132    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac25069b-rb2gz"]
Oct 30 21:23:36.110578 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:36.110556    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f4cd36b-555c2"]
Oct 30 21:23:36.894266 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:36.894219    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1b889bd-gqxc5"]
Oct 30 21:23:36.895325 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:36.895301    5713 cpu_manager.go:414] "containerMap: removing container" podUID="102176e5-2bda-4e24-b86a-0d50fba75ac3" containerName="payload-processing"
Oct 30 21:23:36.895325 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:36.895324    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="102176e5-2bda-4e24-b86a-0d50fba75ac3" containerName="payload-processing"
Oct 30 21:23:36.895493 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:36.895355    5713 cpu_manager.go:414] "containerMap: removing container" podUID="b4ecc41c-33fd-4558-bf79-e4ef840c0aa1" containerName="revoke-keys"
Oct 30 21:23:36.895493 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:36.895361    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4ecc41c-33fd-4558-bf79-e4ef840c0aa1" containerName="revoke-keys"
Oct 30 21:23:36.895493 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:36.895467    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4ecc41c-33fd-4558-bf79-e4ef840c0aa1" containerName="revoke-keys"
Oct 30 21:23:36.895493 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:36.895482    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="102176e5-2bda-4e24-b86a-0d50fba75ac3" containerName="payload-processing"
Oct 30 21:23:36.899206 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:36.899184    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1b889bd-gqxc5"
Oct 30 21:23:36.960142 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:36.960103    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1b889bd-gqxc5"]
Oct 30 21:23:36.970225 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:36.970183    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:23:36.970580 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:36.970560    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:23:37.003773 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:37.003746    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:23:37.003978 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:37.003912    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:23:37.040169 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:37.040140    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9eaeca5-7abf-4af7-90e3-b190c2db7fbe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-d1b889bd-gqxc5\" (UID: \"e9eaeca5-7abf-4af7-90e3-b190c2db7fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1b889bd-gqxc5"
Oct 30 21:23:37.040322 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:37.040178    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnp6g\" (UniqueName: \"kubernetes.io/projected/e9eaeca5-7abf-4af7-90e3-b190c2db7fbe-kube-api-access-fnp6g\") pod \"maas-api-revoke-keys-e2e-mt-d1b889bd-gqxc5\" (UID: \"e9eaeca5-7abf-4af7-90e3-b190c2db7fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1b889bd-gqxc5"
Oct 30 21:23:37.040421 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:37.040400    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9eaeca5-7abf-4af7-90e3-b190c2db7fbe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-d1b889bd-gqxc5\" (UID: \"e9eaeca5-7abf-4af7-90e3-b190c2db7fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1b889bd-gqxc5"
Oct 30 21:23:37.065341 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:37.065313    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40901aeb-4a8b-4e5b-9756-6594247fc30e" path="/var/lib/kubelet/pods/40901aeb-4a8b-4e5b-9756-6594247fc30e/volumes"
Oct 30 21:23:37.065725 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:37.065711    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51fb32f8-14a4-4d70-9cea-051fcb96f989" path="/var/lib/kubelet/pods/51fb32f8-14a4-4d70-9cea-051fcb96f989/volumes"
Oct 30 21:23:37.141645 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:37.141611    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9eaeca5-7abf-4af7-90e3-b190c2db7fbe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-d1b889bd-gqxc5\" (UID: \"e9eaeca5-7abf-4af7-90e3-b190c2db7fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1b889bd-gqxc5"
Oct 30 21:23:37.141997 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:37.141968    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9eaeca5-7abf-4af7-90e3-b190c2db7fbe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-d1b889bd-gqxc5\" (UID: \"e9eaeca5-7abf-4af7-90e3-b190c2db7fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1b889bd-gqxc5"
Oct 30 21:23:37.142144 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:37.142013    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnp6g\" (UniqueName: \"kubernetes.io/projected/e9eaeca5-7abf-4af7-90e3-b190c2db7fbe-kube-api-access-fnp6g\") pod \"maas-api-revoke-keys-e2e-mt-d1b889bd-gqxc5\" (UID: \"e9eaeca5-7abf-4af7-90e3-b190c2db7fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1b889bd-gqxc5"
Oct 30 21:23:37.171771 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:37.171697    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g"]
Oct 30 21:23:37.172000 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:37.171962    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g" podUID="94f9b9e7-7097-43d9-870e-98b67f32d0e9" containerName="istio-proxy" containerID="cri-o://bdedc3f83a88ecf8efa43b98fdbc64d2cbfe0d670af59a0a8d403e3b35ef9d0e" gracePeriod=30
Oct 30 21:23:37.193279 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:37.193235    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:23:37.193442 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:37.193415    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:23:37.203063 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:37.203033    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9eaeca5-7abf-4af7-90e3-b190c2db7fbe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-d1b889bd-gqxc5\" (UID: \"e9eaeca5-7abf-4af7-90e3-b190c2db7fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1b889bd-gqxc5"
Oct 30 21:23:37.204483 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:37.204463    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9eaeca5-7abf-4af7-90e3-b190c2db7fbe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-d1b889bd-gqxc5\" (UID: \"e9eaeca5-7abf-4af7-90e3-b190c2db7fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1b889bd-gqxc5"
Oct 30 21:23:37.247596 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:37.247578    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:23:37.259875 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:37.259854    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnp6g\" (UniqueName: \"kubernetes.io/projected/e9eaeca5-7abf-4af7-90e3-b190c2db7fbe-kube-api-access-fnp6g\") pod \"maas-api-revoke-keys-e2e-mt-d1b889bd-gqxc5\" (UID: \"e9eaeca5-7abf-4af7-90e3-b190c2db7fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1b889bd-gqxc5"
Oct 30 21:23:37.564285 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:37.564261    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:23:37.573619 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:37.573600    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1b889bd-gqxc5"
Oct 30 21:23:37.707966 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:37.707944    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d4597073-8ff0-4a4e-933c-c64e934fe977/prometheus/0.log"
Oct 30 21:23:37.711981 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:37.711958    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d4597073-8ff0-4a4e-933c-c64e934fe977/prometheus/0.log"
Oct 30 21:23:37.724312 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:37.724296    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r7lpf_764461d5-e0b2-4856-a97e-0c5d3d34a497/ovn-acl-logging/0.log"
Oct 30 21:23:37.729816 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:37.729782    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r7lpf_764461d5-e0b2-4856-a97e-0c5d3d34a497/ovn-acl-logging/0.log"
Oct 30 21:23:37.830083 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:37.830017    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f9eff-7796bfd547-fkj5q"
Oct 30 21:23:37.952530 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:37.952496    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51a91785-6750-4b1a-8174-4598816ea82d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51a91785-6750-4b1a-8174-4598816ea82d-plugins-config-volume\") pod \"51a91785-6750-4b1a-8174-4598816ea82d\" (UID: \"51a91785-6750-4b1a-8174-4598816ea82d\") "
Oct 30 21:23:37.952783 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:37.952763    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51a91785-6750-4b1a-8174-4598816ea82d-kube-api-access-ttrv8\" (UniqueName: \"kubernetes.io/projected/51a91785-6750-4b1a-8174-4598816ea82d-kube-api-access-ttrv8\") pod \"51a91785-6750-4b1a-8174-4598816ea82d\" (UID: \"51a91785-6750-4b1a-8174-4598816ea82d\") "
Oct 30 21:23:37.952915 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:37.952894    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51a91785-6750-4b1a-8174-4598816ea82d-plugins-config-volume" pod "51a91785-6750-4b1a-8174-4598816ea82d" (UID: "51a91785-6750-4b1a-8174-4598816ea82d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:37.953301 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:37.953280    5713 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51a91785-6750-4b1a-8174-4598816ea82d-plugins-config-volume\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:23:37.954420 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:37.954397    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1b889bd-gqxc5"]
Oct 30 21:23:37.955239 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:37.955214    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51a91785-6750-4b1a-8174-4598816ea82d-kube-api-access-ttrv8" pod "51a91785-6750-4b1a-8174-4598816ea82d" (UID: "51a91785-6750-4b1a-8174-4598816ea82d"). InnerVolumeSpecName "kube-api-access-ttrv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:37.999206 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:23:37.999175    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9eaeca5_7abf_4af7_90e3_b190c2db7fbe.slice/crio-80791e0c27dcc555b190cee333fd5705970fc7aaea453cc47d6315276dc4ccfc WatchSource:0}: Error finding container 80791e0c27dcc555b190cee333fd5705970fc7aaea453cc47d6315276dc4ccfc: Status 404 returned error can't find the container with id 80791e0c27dcc555b190cee333fd5705970fc7aaea453cc47d6315276dc4ccfc
Oct 30 21:23:38.006333 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:38.006306    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-08df98-969r5"]
Oct 30 21:23:38.006972 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:38.006898    5713 cpu_manager.go:414] "containerMap: removing container" podUID="51a91785-6750-4b1a-8174-4598816ea82d" containerName="payload-pre-processing"
Oct 30 21:23:38.006972 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:38.006914    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51a91785-6750-4b1a-8174-4598816ea82d" containerName="payload-pre-processing"
Oct 30 21:23:38.006972 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:38.007008    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="51a91785-6750-4b1a-8174-4598816ea82d" containerName="payload-pre-processing"
Oct 30 21:23:38.011377 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:38.011359    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-08df98-969r5"
Oct 30 21:23:38.054112 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:38.054083    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttrv8\" (UniqueName: \"kubernetes.io/projected/51a91785-6750-4b1a-8174-4598816ea82d-kube-api-access-ttrv8\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:23:38.067429 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:38.067344    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-08df98-969r5"]
Oct 30 21:23:38.155102 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:38.155071    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e8c5c3f-1ca9-4434-a0cb-c45f0e249fbb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-08df98-969r5\" (UID: \"3e8c5c3f-1ca9-4434-a0cb-c45f0e249fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-08df98-969r5"
Oct 30 21:23:38.155305 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:38.155288    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrrvn\" (UniqueName: \"kubernetes.io/projected/3e8c5c3f-1ca9-4434-a0cb-c45f0e249fbb-kube-api-access-rrrvn\") pod \"maas-api-revoke-keys-e2e-inf-b-08df98-969r5\" (UID: \"3e8c5c3f-1ca9-4434-a0cb-c45f0e249fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-08df98-969r5"
Oct 30 21:23:38.155386 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:38.155347    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e8c5c3f-1ca9-4434-a0cb-c45f0e249fbb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-08df98-969r5\" (UID: \"3e8c5c3f-1ca9-4434-a0cb-c45f0e249fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-08df98-969r5"
Oct 30 21:23:38.257098 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:38.257073    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrrvn\" (UniqueName: \"kubernetes.io/projected/3e8c5c3f-1ca9-4434-a0cb-c45f0e249fbb-kube-api-access-rrrvn\") pod \"maas-api-revoke-keys-e2e-inf-b-08df98-969r5\" (UID: \"3e8c5c3f-1ca9-4434-a0cb-c45f0e249fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-08df98-969r5"
Oct 30 21:23:38.257215 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:38.257122    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e8c5c3f-1ca9-4434-a0cb-c45f0e249fbb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-08df98-969r5\" (UID: \"3e8c5c3f-1ca9-4434-a0cb-c45f0e249fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-08df98-969r5"
Oct 30 21:23:38.257215 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:38.257177    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e8c5c3f-1ca9-4434-a0cb-c45f0e249fbb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-08df98-969r5\" (UID: \"3e8c5c3f-1ca9-4434-a0cb-c45f0e249fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-08df98-969r5"
Oct 30 21:23:38.257753 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:38.257735    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e8c5c3f-1ca9-4434-a0cb-c45f0e249fbb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-08df98-969r5\" (UID: \"3e8c5c3f-1ca9-4434-a0cb-c45f0e249fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-08df98-969r5"
Oct 30 21:23:38.259440 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:38.259423    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e8c5c3f-1ca9-4434-a0cb-c45f0e249fbb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-08df98-969r5\" (UID: \"3e8c5c3f-1ca9-4434-a0cb-c45f0e249fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-08df98-969r5"
Oct 30 21:23:38.316139 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:38.316112    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrrvn\" (UniqueName: \"kubernetes.io/projected/3e8c5c3f-1ca9-4434-a0cb-c45f0e249fbb-kube-api-access-rrrvn\") pod \"maas-api-revoke-keys-e2e-inf-b-08df98-969r5\" (UID: \"3e8c5c3f-1ca9-4434-a0cb-c45f0e249fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-08df98-969r5"
Oct 30 21:23:38.327997 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:38.327934    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-08df98-969r5"
Oct 30 21:23:38.510422 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:38.510383    5713 generic.go:356] "Generic (PLEG): container finished" podID="51a91785-6750-4b1a-8174-4598816ea82d" containerID="0ca83bbf41038832964465ad63185b5410f84cabaf8611b8c8678222565da7f7" exitCode=0
Oct 30 21:23:38.510757 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:38.510453    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f9eff-7796bfd547-fkj5q"
Oct 30 21:23:38.510757 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:38.510507    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f9eff-7796bfd547-fkj5q" event={"ID":"51a91785-6750-4b1a-8174-4598816ea82d","Type":"ContainerDied","Data":"0ca83bbf41038832964465ad63185b5410f84cabaf8611b8c8678222565da7f7"}
Oct 30 21:23:38.510757 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:38.510573    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f9eff-7796bfd547-fkj5q" event={"ID":"51a91785-6750-4b1a-8174-4598816ea82d","Type":"ContainerDied","Data":"0d88c2b32b38dbab0703e640b63828d0f8034ba0ce357586eddcbf5a92ae0305"}
Oct 30 21:23:38.510757 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:38.510611    5713 scope.go:122] "RemoveContainer" containerID="0ca83bbf41038832964465ad63185b5410f84cabaf8611b8c8678222565da7f7"
Oct 30 21:23:38.513212 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:38.513187    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1b889bd-gqxc5" event={"ID":"e9eaeca5-7abf-4af7-90e3-b190c2db7fbe","Type":"ContainerStarted","Data":"b7e8085ea675ba9ae1c1c8a2853a092e79dbc7c1fb7161a5e387e12d092eacfe"}
Oct 30 21:23:38.513350 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:38.513228    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1b889bd-gqxc5" event={"ID":"e9eaeca5-7abf-4af7-90e3-b190c2db7fbe","Type":"ContainerStarted","Data":"80791e0c27dcc555b190cee333fd5705970fc7aaea453cc47d6315276dc4ccfc"}
Oct 30 21:23:38.522490 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:38.522472    5713 scope.go:122] "RemoveContainer" containerID="0ca83bbf41038832964465ad63185b5410f84cabaf8611b8c8678222565da7f7"
Oct 30 21:23:38.522746 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:23:38.522730    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ca83bbf41038832964465ad63185b5410f84cabaf8611b8c8678222565da7f7\": container with ID starting with 0ca83bbf41038832964465ad63185b5410f84cabaf8611b8c8678222565da7f7 not found: ID does not exist" containerID="0ca83bbf41038832964465ad63185b5410f84cabaf8611b8c8678222565da7f7"
Oct 30 21:23:38.522821 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:38.522754    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ca83bbf41038832964465ad63185b5410f84cabaf8611b8c8678222565da7f7"} err="failed to get container status \"0ca83bbf41038832964465ad63185b5410f84cabaf8611b8c8678222565da7f7\": rpc error: code = NotFound desc = could not find container \"0ca83bbf41038832964465ad63185b5410f84cabaf8611b8c8678222565da7f7\": container with ID starting with 0ca83bbf41038832964465ad63185b5410f84cabaf8611b8c8678222565da7f7 not found: ID does not exist"
Oct 30 21:23:38.683926 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:38.680138    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2f9eff-7796bfd547-fkj5q"]
Oct 30 21:23:38.687521 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:38.687496    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2f9eff-7796bfd547-fkj5q"]
Oct 30 21:23:38.705697 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:38.705671    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-08df98-969r5"]
Oct 30 21:23:38.707019 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:23:38.706996    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e8c5c3f_1ca9_4434_a0cb_c45f0e249fbb.slice/crio-ebee417326cfb7c2b7cf65e7cdaf6c98171884742fea0a8b6c7dfc1bd36f2779 WatchSource:0}: Error finding container ebee417326cfb7c2b7cf65e7cdaf6c98171884742fea0a8b6c7dfc1bd36f2779: Status 404 returned error can't find the container with id ebee417326cfb7c2b7cf65e7cdaf6c98171884742fea0a8b6c7dfc1bd36f2779
Oct 30 21:23:38.799615 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:38.799568    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1b889bd-gqxc5" podStartSLOduration=2.799554394 podStartE2EDuration="2.799554394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:38.797400835 +0000 UTC m=+2701.862177248" watchObservedRunningTime="2026-10-30 21:23:38.799554394 +0000 UTC m=+2701.864330819"
Oct 30 21:23:39.064404 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:39.064380    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51a91785-6750-4b1a-8174-4598816ea82d" path="/var/lib/kubelet/pods/51a91785-6750-4b1a-8174-4598816ea82d/volumes"
Oct 30 21:23:39.523433 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:39.523354    5713 generic.go:356] "Generic (PLEG): container finished" podID="e9eaeca5-7abf-4af7-90e3-b190c2db7fbe" containerID="b7e8085ea675ba9ae1c1c8a2853a092e79dbc7c1fb7161a5e387e12d092eacfe" exitCode=0
Oct 30 21:23:39.523878 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:39.523430    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1b889bd-gqxc5" event={"ID":"e9eaeca5-7abf-4af7-90e3-b190c2db7fbe","Type":"ContainerDied","Data":"b7e8085ea675ba9ae1c1c8a2853a092e79dbc7c1fb7161a5e387e12d092eacfe"}
Oct 30 21:23:39.525181 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:39.525147    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-08df98-969r5" event={"ID":"3e8c5c3f-1ca9-4434-a0cb-c45f0e249fbb","Type":"ContainerStarted","Data":"acdbaf0edc8e390fdccf693dc68f058d775574a71919631bee029e0bcd341603"}
Oct 30 21:23:39.525309 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:39.525202    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-08df98-969r5" event={"ID":"3e8c5c3f-1ca9-4434-a0cb-c45f0e249fbb","Type":"ContainerStarted","Data":"ebee417326cfb7c2b7cf65e7cdaf6c98171884742fea0a8b6c7dfc1bd36f2779"}
Oct 30 21:23:39.534185 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:39.534165    5713 scope.go:122] "RemoveContainer" containerID="2e1cf5cdc07bb868a125c9bedbd39f925396229d24ff6995c4bf38f70f88913d"
Oct 30 21:23:39.547009 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:39.546985    5713 scope.go:122] "RemoveContainer" containerID="d9b2b969fb1bce450c7bcb55efe7f7f26c3a8691ca5cc084880d196e62eec877"
Oct 30 21:23:39.559977 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:39.559950    5713 scope.go:122] "RemoveContainer" containerID="006654326e72547b4c6660562faa7365d04c9b61c23f23549f15dbb89b2d1211"
Oct 30 21:23:39.568771 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:39.568737    5713 scope.go:122] "RemoveContainer" containerID="1b96c773bdfa48925cff24fbd4767064c77bfcc230379e2d4b92f70e5d81fa76"
Oct 30 21:23:39.752713 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:39.752643    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-08df98-969r5" podStartSLOduration=2.752623442 podStartE2EDuration="2.752623442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:39.749329081 +0000 UTC m=+2702.814105484" watchObservedRunningTime="2026-10-30 21:23:39.752623442 +0000 UTC m=+2702.817399842"
Oct 30 21:23:40.539573 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:40.539526    5713 generic.go:356] "Generic (PLEG): container finished" podID="3e8c5c3f-1ca9-4434-a0cb-c45f0e249fbb" containerID="acdbaf0edc8e390fdccf693dc68f058d775574a71919631bee029e0bcd341603" exitCode=0
Oct 30 21:23:40.540136 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:40.539606    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-08df98-969r5" event={"ID":"3e8c5c3f-1ca9-4434-a0cb-c45f0e249fbb","Type":"ContainerDied","Data":"acdbaf0edc8e390fdccf693dc68f058d775574a71919631bee029e0bcd341603"}
Oct 30 21:23:40.731208 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:40.731184    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1b889bd-gqxc5"
Oct 30 21:23:40.790582 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:40.790504    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9eaeca5-7abf-4af7-90e3-b190c2db7fbe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9eaeca5-7abf-4af7-90e3-b190c2db7fbe-maas-api-openshift-ca\") pod \"e9eaeca5-7abf-4af7-90e3-b190c2db7fbe\" (UID: \"e9eaeca5-7abf-4af7-90e3-b190c2db7fbe\") "
Oct 30 21:23:40.790582 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:40.790572    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9eaeca5-7abf-4af7-90e3-b190c2db7fbe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9eaeca5-7abf-4af7-90e3-b190c2db7fbe-maas-api-xks-ca\") pod \"e9eaeca5-7abf-4af7-90e3-b190c2db7fbe\" (UID: \"e9eaeca5-7abf-4af7-90e3-b190c2db7fbe\") "
Oct 30 21:23:40.790778 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:40.790600    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9eaeca5-7abf-4af7-90e3-b190c2db7fbe-kube-api-access-fnp6g\" (UniqueName: \"kubernetes.io/projected/e9eaeca5-7abf-4af7-90e3-b190c2db7fbe-kube-api-access-fnp6g\") pod \"e9eaeca5-7abf-4af7-90e3-b190c2db7fbe\" (UID: \"e9eaeca5-7abf-4af7-90e3-b190c2db7fbe\") "
Oct 30 21:23:40.790895 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:40.790874    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9eaeca5-7abf-4af7-90e3-b190c2db7fbe-maas-api-openshift-ca" pod "e9eaeca5-7abf-4af7-90e3-b190c2db7fbe" (UID: "e9eaeca5-7abf-4af7-90e3-b190c2db7fbe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:40.791364 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:40.791305    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9eaeca5-7abf-4af7-90e3-b190c2db7fbe-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:23:40.792884 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:40.792865    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9eaeca5-7abf-4af7-90e3-b190c2db7fbe-maas-api-xks-ca" pod "e9eaeca5-7abf-4af7-90e3-b190c2db7fbe" (UID: "e9eaeca5-7abf-4af7-90e3-b190c2db7fbe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:23:40.793024 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:40.793007    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9eaeca5-7abf-4af7-90e3-b190c2db7fbe-kube-api-access-fnp6g" pod "e9eaeca5-7abf-4af7-90e3-b190c2db7fbe" (UID: "e9eaeca5-7abf-4af7-90e3-b190c2db7fbe"). InnerVolumeSpecName "kube-api-access-fnp6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:40.892646 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:40.892612    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9eaeca5-7abf-4af7-90e3-b190c2db7fbe-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:23:40.892646 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:40.892642    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnp6g\" (UniqueName: \"kubernetes.io/projected/e9eaeca5-7abf-4af7-90e3-b190c2db7fbe-kube-api-access-fnp6g\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:23:41.548375 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:41.548345    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1b889bd-gqxc5"
Oct 30 21:23:41.548375 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:41.548366    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1b889bd-gqxc5" event={"ID":"e9eaeca5-7abf-4af7-90e3-b190c2db7fbe","Type":"ContainerDied","Data":"80791e0c27dcc555b190cee333fd5705970fc7aaea453cc47d6315276dc4ccfc"}
Oct 30 21:23:41.548925 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:41.548412    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80791e0c27dcc555b190cee333fd5705970fc7aaea453cc47d6315276dc4ccfc"
Oct 30 21:23:41.698482 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:41.698452    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-08df98-969r5"
Oct 30 21:23:41.802869 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:41.802784    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e8c5c3f-1ca9-4434-a0cb-c45f0e249fbb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e8c5c3f-1ca9-4434-a0cb-c45f0e249fbb-maas-api-xks-ca\") pod \"3e8c5c3f-1ca9-4434-a0cb-c45f0e249fbb\" (UID: \"3e8c5c3f-1ca9-4434-a0cb-c45f0e249fbb\") "
Oct 30 21:23:41.802869 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:41.802841    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e8c5c3f-1ca9-4434-a0cb-c45f0e249fbb-kube-api-access-rrrvn\" (UniqueName: \"kubernetes.io/projected/3e8c5c3f-1ca9-4434-a0cb-c45f0e249fbb-kube-api-access-rrrvn\") pod \"3e8c5c3f-1ca9-4434-a0cb-c45f0e249fbb\" (UID: \"3e8c5c3f-1ca9-4434-a0cb-c45f0e249fbb\") "
Oct 30 21:23:41.803116 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:41.802966    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e8c5c3f-1ca9-4434-a0cb-c45f0e249fbb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e8c5c3f-1ca9-4434-a0cb-c45f0e249fbb-maas-api-openshift-ca\") pod \"3e8c5c3f-1ca9-4434-a0cb-c45f0e249fbb\" (UID: \"3e8c5c3f-1ca9-4434-a0cb-c45f0e249fbb\") "
Oct 30 21:23:41.803358 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:41.803334    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e8c5c3f-1ca9-4434-a0cb-c45f0e249fbb-maas-api-openshift-ca" pod "3e8c5c3f-1ca9-4434-a0cb-c45f0e249fbb" (UID: "3e8c5c3f-1ca9-4434-a0cb-c45f0e249fbb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:41.803516 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:41.803497    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e8c5c3f-1ca9-4434-a0cb-c45f0e249fbb-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:23:41.804972 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:41.804947    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e8c5c3f-1ca9-4434-a0cb-c45f0e249fbb-maas-api-xks-ca" pod "3e8c5c3f-1ca9-4434-a0cb-c45f0e249fbb" (UID: "3e8c5c3f-1ca9-4434-a0cb-c45f0e249fbb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:23:41.805173 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:41.805150    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e8c5c3f-1ca9-4434-a0cb-c45f0e249fbb-kube-api-access-rrrvn" pod "3e8c5c3f-1ca9-4434-a0cb-c45f0e249fbb" (UID: "3e8c5c3f-1ca9-4434-a0cb-c45f0e249fbb"). InnerVolumeSpecName "kube-api-access-rrrvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:41.904600 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:41.904564    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e8c5c3f-1ca9-4434-a0cb-c45f0e249fbb-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:23:41.904600 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:41.904597    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrrvn\" (UniqueName: \"kubernetes.io/projected/3e8c5c3f-1ca9-4434-a0cb-c45f0e249fbb-kube-api-access-rrrvn\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:23:42.432965 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:42.432939    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g"
Oct 30 21:23:42.511685 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:42.511601    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94f9b9e7-7097-43d9-870e-98b67f32d0e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f9b9e7-7097-43d9-870e-98b67f32d0e9-credential-socket\") pod \"94f9b9e7-7097-43d9-870e-98b67f32d0e9\" (UID: \"94f9b9e7-7097-43d9-870e-98b67f32d0e9\") "
Oct 30 21:23:42.511685 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:42.511649    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94f9b9e7-7097-43d9-870e-98b67f32d0e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94f9b9e7-7097-43d9-870e-98b67f32d0e9-istio-envoy\") pod \"94f9b9e7-7097-43d9-870e-98b67f32d0e9\" (UID: \"94f9b9e7-7097-43d9-870e-98b67f32d0e9\") "
Oct 30 21:23:42.511685 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:42.511680    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94f9b9e7-7097-43d9-870e-98b67f32d0e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f9b9e7-7097-43d9-870e-98b67f32d0e9-workload-socket\") pod \"94f9b9e7-7097-43d9-870e-98b67f32d0e9\" (UID: \"94f9b9e7-7097-43d9-870e-98b67f32d0e9\") "
Oct 30 21:23:42.512016 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:42.511715    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94f9b9e7-7097-43d9-870e-98b67f32d0e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94f9b9e7-7097-43d9-870e-98b67f32d0e9-istiod-ca-cert\") pod \"94f9b9e7-7097-43d9-870e-98b67f32d0e9\" (UID: \"94f9b9e7-7097-43d9-870e-98b67f32d0e9\") "
Oct 30 21:23:42.512016 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:42.511740    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94f9b9e7-7097-43d9-870e-98b67f32d0e9-kube-api-access-8sdmw\" (UniqueName: \"kubernetes.io/projected/94f9b9e7-7097-43d9-870e-98b67f32d0e9-kube-api-access-8sdmw\") pod \"94f9b9e7-7097-43d9-870e-98b67f32d0e9\" (UID: \"94f9b9e7-7097-43d9-870e-98b67f32d0e9\") "
Oct 30 21:23:42.512016 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:42.511763    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94f9b9e7-7097-43d9-870e-98b67f32d0e9-istio-token\" (UniqueName: \"kubernetes.io/projected/94f9b9e7-7097-43d9-870e-98b67f32d0e9-istio-token\") pod \"94f9b9e7-7097-43d9-870e-98b67f32d0e9\" (UID: \"94f9b9e7-7097-43d9-870e-98b67f32d0e9\") "
Oct 30 21:23:42.512016 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:42.511782    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94f9b9e7-7097-43d9-870e-98b67f32d0e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94f9b9e7-7097-43d9-870e-98b67f32d0e9-istio-data\") pod \"94f9b9e7-7097-43d9-870e-98b67f32d0e9\" (UID: \"94f9b9e7-7097-43d9-870e-98b67f32d0e9\") "
Oct 30 21:23:42.512016 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:42.511941    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94f9b9e7-7097-43d9-870e-98b67f32d0e9-credential-socket" pod "94f9b9e7-7097-43d9-870e-98b67f32d0e9" (UID: "94f9b9e7-7097-43d9-870e-98b67f32d0e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:42.512016 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:42.512009    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94f9b9e7-7097-43d9-870e-98b67f32d0e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94f9b9e7-7097-43d9-870e-98b67f32d0e9-workload-certs\") pod \"94f9b9e7-7097-43d9-870e-98b67f32d0e9\" (UID: \"94f9b9e7-7097-43d9-870e-98b67f32d0e9\") "
Oct 30 21:23:42.512353 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:42.512042    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/94f9b9e7-7097-43d9-870e-98b67f32d0e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94f9b9e7-7097-43d9-870e-98b67f32d0e9-istio-podinfo\") pod \"94f9b9e7-7097-43d9-870e-98b67f32d0e9\" (UID: \"94f9b9e7-7097-43d9-870e-98b67f32d0e9\") "
Oct 30 21:23:42.512353 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:42.512073    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94f9b9e7-7097-43d9-870e-98b67f32d0e9-workload-socket" pod "94f9b9e7-7097-43d9-870e-98b67f32d0e9" (UID: "94f9b9e7-7097-43d9-870e-98b67f32d0e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:42.512353 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:42.512219    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94f9b9e7-7097-43d9-870e-98b67f32d0e9-istio-data" pod "94f9b9e7-7097-43d9-870e-98b67f32d0e9" (UID: "94f9b9e7-7097-43d9-870e-98b67f32d0e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:42.512353 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:42.512257    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94f9b9e7-7097-43d9-870e-98b67f32d0e9-istiod-ca-cert" pod "94f9b9e7-7097-43d9-870e-98b67f32d0e9" (UID: "94f9b9e7-7097-43d9-870e-98b67f32d0e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:42.512353 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:42.512319    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94f9b9e7-7097-43d9-870e-98b67f32d0e9-workload-certs" pod "94f9b9e7-7097-43d9-870e-98b67f32d0e9" (UID: "94f9b9e7-7097-43d9-870e-98b67f32d0e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:42.512886 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:42.512860    5713 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94f9b9e7-7097-43d9-870e-98b67f32d0e9-workload-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:23:42.513017 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:42.512890    5713 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f9b9e7-7097-43d9-870e-98b67f32d0e9-credential-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:23:42.513017 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:42.512907    5713 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f9b9e7-7097-43d9-870e-98b67f32d0e9-workload-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:23:42.513017 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:42.512920    5713 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94f9b9e7-7097-43d9-870e-98b67f32d0e9-istiod-ca-cert\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:23:42.513017 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:42.512938    5713 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94f9b9e7-7097-43d9-870e-98b67f32d0e9-istio-data\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:23:42.514257 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:42.514234    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94f9b9e7-7097-43d9-870e-98b67f32d0e9-istio-envoy" pod "94f9b9e7-7097-43d9-870e-98b67f32d0e9" (UID: "94f9b9e7-7097-43d9-870e-98b67f32d0e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:42.514257 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:42.514246    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94f9b9e7-7097-43d9-870e-98b67f32d0e9-kube-api-access-8sdmw" pod "94f9b9e7-7097-43d9-870e-98b67f32d0e9" (UID: "94f9b9e7-7097-43d9-870e-98b67f32d0e9"). InnerVolumeSpecName "kube-api-access-8sdmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:42.514431 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:42.514273    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94f9b9e7-7097-43d9-870e-98b67f32d0e9-istio-token" pod "94f9b9e7-7097-43d9-870e-98b67f32d0e9" (UID: "94f9b9e7-7097-43d9-870e-98b67f32d0e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:42.514551 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:42.514534    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/94f9b9e7-7097-43d9-870e-98b67f32d0e9-istio-podinfo" pod "94f9b9e7-7097-43d9-870e-98b67f32d0e9" (UID: "94f9b9e7-7097-43d9-870e-98b67f32d0e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:23:42.565894 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:42.565847    5713 generic.go:356] "Generic (PLEG): container finished" podID="94f9b9e7-7097-43d9-870e-98b67f32d0e9" containerID="bdedc3f83a88ecf8efa43b98fdbc64d2cbfe0d670af59a0a8d403e3b35ef9d0e" exitCode=0
Oct 30 21:23:42.566366 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:42.566042    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g" event={"ID":"94f9b9e7-7097-43d9-870e-98b67f32d0e9","Type":"ContainerDied","Data":"bdedc3f83a88ecf8efa43b98fdbc64d2cbfe0d670af59a0a8d403e3b35ef9d0e"}
Oct 30 21:23:42.566366 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:42.566129    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g" event={"ID":"94f9b9e7-7097-43d9-870e-98b67f32d0e9","Type":"ContainerDied","Data":"fc4f92a1df47f58de375cfca934ce992b8812af965c71c70e7a883fe5f199d25"}
Oct 30 21:23:42.566366 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:42.566170    5713 scope.go:122] "RemoveContainer" containerID="bdedc3f83a88ecf8efa43b98fdbc64d2cbfe0d670af59a0a8d403e3b35ef9d0e"
Oct 30 21:23:42.566527 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:42.566371    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g"
Oct 30 21:23:42.572506 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:42.572465    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-08df98-969r5" event={"ID":"3e8c5c3f-1ca9-4434-a0cb-c45f0e249fbb","Type":"ContainerDied","Data":"ebee417326cfb7c2b7cf65e7cdaf6c98171884742fea0a8b6c7dfc1bd36f2779"}
Oct 30 21:23:42.572653 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:42.572514    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ebee417326cfb7c2b7cf65e7cdaf6c98171884742fea0a8b6c7dfc1bd36f2779"
Oct 30 21:23:42.572653 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:42.572565    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-08df98-969r5"
Oct 30 21:23:42.585205 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:42.585049    5713 scope.go:122] "RemoveContainer" containerID="b38d889798e8ba005656c437fe5daa846aacd07bb171168a11444965343e6eab"
Oct 30 21:23:42.596350 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:42.596324    5713 scope.go:122] "RemoveContainer" containerID="bdedc3f83a88ecf8efa43b98fdbc64d2cbfe0d670af59a0a8d403e3b35ef9d0e"
Oct 30 21:23:42.596708 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:23:42.596675    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdedc3f83a88ecf8efa43b98fdbc64d2cbfe0d670af59a0a8d403e3b35ef9d0e\": container with ID starting with bdedc3f83a88ecf8efa43b98fdbc64d2cbfe0d670af59a0a8d403e3b35ef9d0e not found: ID does not exist" containerID="bdedc3f83a88ecf8efa43b98fdbc64d2cbfe0d670af59a0a8d403e3b35ef9d0e"
Oct 30 21:23:42.596812 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:42.596716    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdedc3f83a88ecf8efa43b98fdbc64d2cbfe0d670af59a0a8d403e3b35ef9d0e"} err="failed to get container status \"bdedc3f83a88ecf8efa43b98fdbc64d2cbfe0d670af59a0a8d403e3b35ef9d0e\": rpc error: code = NotFound desc = could not find container \"bdedc3f83a88ecf8efa43b98fdbc64d2cbfe0d670af59a0a8d403e3b35ef9d0e\": container with ID starting with bdedc3f83a88ecf8efa43b98fdbc64d2cbfe0d670af59a0a8d403e3b35ef9d0e not found: ID does not exist"
Oct 30 21:23:42.596812 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:42.596736    5713 scope.go:122] "RemoveContainer" containerID="b38d889798e8ba005656c437fe5daa846aacd07bb171168a11444965343e6eab"
Oct 30 21:23:42.597091 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:23:42.597060    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b38d889798e8ba005656c437fe5daa846aacd07bb171168a11444965343e6eab\": container with ID starting with b38d889798e8ba005656c437fe5daa846aacd07bb171168a11444965343e6eab not found: ID does not exist" containerID="b38d889798e8ba005656c437fe5daa846aacd07bb171168a11444965343e6eab"
Oct 30 21:23:42.597163 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:42.597094    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b38d889798e8ba005656c437fe5daa846aacd07bb171168a11444965343e6eab"} err="failed to get container status \"b38d889798e8ba005656c437fe5daa846aacd07bb171168a11444965343e6eab\": rpc error: code = NotFound desc = could not find container \"b38d889798e8ba005656c437fe5daa846aacd07bb171168a11444965343e6eab\": container with ID starting with b38d889798e8ba005656c437fe5daa846aacd07bb171168a11444965343e6eab not found: ID does not exist"
Oct 30 21:23:42.614025 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:42.613997    5713 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94f9b9e7-7097-43d9-870e-98b67f32d0e9-istio-podinfo\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:23:42.614025 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:42.614021    5713 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94f9b9e7-7097-43d9-870e-98b67f32d0e9-istio-envoy\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:23:42.614168 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:42.614031    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sdmw\" (UniqueName: \"kubernetes.io/projected/94f9b9e7-7097-43d9-870e-98b67f32d0e9-kube-api-access-8sdmw\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:23:42.614168 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:42.614041    5713 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94f9b9e7-7097-43d9-870e-98b67f32d0e9-istio-token\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:23:42.761980 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:42.761779    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g"]
Oct 30 21:23:42.767805 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:42.767754    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-2f9eff-openshift-default-76f4d54d8c-ms57g"]
Oct 30 21:23:43.063339 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:43.063262    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94f9b9e7-7097-43d9-870e-98b67f32d0e9" path="/var/lib/kubelet/pods/94f9b9e7-7097-43d9-870e-98b67f32d0e9/volumes"
Oct 30 21:23:47.902645 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:47.902562    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f0fe13/test-model-f0fe13-kserve-c746dc965-tkz9t"]
Oct 30 21:23:48.908435 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:48.908402    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e547c79b-m84wh"]
Oct 30 21:23:48.908956 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:48.908941    5713 cpu_manager.go:414] "containerMap: removing container" podUID="94f9b9e7-7097-43d9-870e-98b67f32d0e9" containerName="istio-proxy"
Oct 30 21:23:48.908956 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:48.908957    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94f9b9e7-7097-43d9-870e-98b67f32d0e9" containerName="istio-proxy"
Oct 30 21:23:48.909057 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:48.908982    5713 cpu_manager.go:414] "containerMap: removing container" podUID="94f9b9e7-7097-43d9-870e-98b67f32d0e9" containerName="istio-proxy"
Oct 30 21:23:48.909057 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:48.908988    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94f9b9e7-7097-43d9-870e-98b67f32d0e9" containerName="istio-proxy"
Oct 30 21:23:48.909057 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:48.909003    5713 cpu_manager.go:414] "containerMap: removing container" podUID="e9eaeca5-7abf-4af7-90e3-b190c2db7fbe" containerName="revoke-keys"
Oct 30 21:23:48.909057 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:48.909009    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9eaeca5-7abf-4af7-90e3-b190c2db7fbe" containerName="revoke-keys"
Oct 30 21:23:48.909057 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:48.909024    5713 cpu_manager.go:414] "containerMap: removing container" podUID="3e8c5c3f-1ca9-4434-a0cb-c45f0e249fbb" containerName="revoke-keys"
Oct 30 21:23:48.909057 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:48.909029    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e8c5c3f-1ca9-4434-a0cb-c45f0e249fbb" containerName="revoke-keys"
Oct 30 21:23:48.909246 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:48.909107    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="94f9b9e7-7097-43d9-870e-98b67f32d0e9" containerName="istio-proxy"
Oct 30 21:23:48.909246 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:48.909118    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9eaeca5-7abf-4af7-90e3-b190c2db7fbe" containerName="revoke-keys"
Oct 30 21:23:48.909246 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:48.909128    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e8c5c3f-1ca9-4434-a0cb-c45f0e249fbb" containerName="revoke-keys"
Oct 30 21:23:48.909246 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:48.909136    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="94f9b9e7-7097-43d9-870e-98b67f32d0e9" containerName="istio-proxy"
Oct 30 21:23:48.913862 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:48.913845    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e547c79b-m84wh"
Oct 30 21:23:48.969489 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:48.969461    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e547c79b-m84wh"]
Oct 30 21:23:48.996396 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:48.996361    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x62r2\" (UniqueName: \"kubernetes.io/projected/afaab19a-aaae-4a24-9e43-0a7e4594641d-kube-api-access-x62r2\") pod \"maas-api-revoke-sub-e547c79b-m84wh\" (UID: \"afaab19a-aaae-4a24-9e43-0a7e4594641d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e547c79b-m84wh"
Oct 30 21:23:48.996561 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:48.996432    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afaab19a-aaae-4a24-9e43-0a7e4594641d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e547c79b-m84wh\" (UID: \"afaab19a-aaae-4a24-9e43-0a7e4594641d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e547c79b-m84wh"
Oct 30 21:23:48.996678 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:48.996660    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afaab19a-aaae-4a24-9e43-0a7e4594641d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e547c79b-m84wh\" (UID: \"afaab19a-aaae-4a24-9e43-0a7e4594641d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e547c79b-m84wh"
Oct 30 21:23:49.015448 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:49.015417    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:23:49.015591 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:49.015452    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:23:49.015591 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:49.015471    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:23:49.015731 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:49.015621    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:23:49.097983 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:49.097887    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afaab19a-aaae-4a24-9e43-0a7e4594641d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e547c79b-m84wh\" (UID: \"afaab19a-aaae-4a24-9e43-0a7e4594641d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e547c79b-m84wh"
Oct 30 21:23:49.098174 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:49.098000    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x62r2\" (UniqueName: \"kubernetes.io/projected/afaab19a-aaae-4a24-9e43-0a7e4594641d-kube-api-access-x62r2\") pod \"maas-api-revoke-sub-e547c79b-m84wh\" (UID: \"afaab19a-aaae-4a24-9e43-0a7e4594641d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e547c79b-m84wh"
Oct 30 21:23:49.098174 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:49.098094    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afaab19a-aaae-4a24-9e43-0a7e4594641d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e547c79b-m84wh\" (UID: \"afaab19a-aaae-4a24-9e43-0a7e4594641d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e547c79b-m84wh"
Oct 30 21:23:49.098643 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:49.098607    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afaab19a-aaae-4a24-9e43-0a7e4594641d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e547c79b-m84wh\" (UID: \"afaab19a-aaae-4a24-9e43-0a7e4594641d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e547c79b-m84wh"
Oct 30 21:23:49.100685 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:49.100656    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afaab19a-aaae-4a24-9e43-0a7e4594641d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e547c79b-m84wh\" (UID: \"afaab19a-aaae-4a24-9e43-0a7e4594641d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e547c79b-m84wh"
Oct 30 21:23:49.159072 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:49.159003    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x62r2\" (UniqueName: \"kubernetes.io/projected/afaab19a-aaae-4a24-9e43-0a7e4594641d-kube-api-access-x62r2\") pod \"maas-api-revoke-sub-e547c79b-m84wh\" (UID: \"afaab19a-aaae-4a24-9e43-0a7e4594641d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e547c79b-m84wh"
Oct 30 21:23:49.228109 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:49.228084    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e547c79b-m84wh"
Oct 30 21:23:49.605007 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:49.604980    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e547c79b-m84wh"]
Oct 30 21:23:49.606192 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:23:49.606165    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafaab19a_aaae_4a24_9e43_0a7e4594641d.slice/crio-689db030a0102e57af9e5501b0d26debaf74563241dabef42e121d1b7f89320d WatchSource:0}: Error finding container 689db030a0102e57af9e5501b0d26debaf74563241dabef42e121d1b7f89320d: Status 404 returned error can't find the container with id 689db030a0102e57af9e5501b0d26debaf74563241dabef42e121d1b7f89320d
Oct 30 21:23:49.634608 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:49.634581    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e547c79b-m84wh" event={"ID":"afaab19a-aaae-4a24-9e43-0a7e4594641d","Type":"ContainerStarted","Data":"689db030a0102e57af9e5501b0d26debaf74563241dabef42e121d1b7f89320d"}
Oct 30 21:23:50.642943 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:50.642910    5713 generic.go:356] "Generic (PLEG): container finished" podID="afaab19a-aaae-4a24-9e43-0a7e4594641d" containerID="567a08a3a0e5764238904d74e0842d7b2ee4dedea92b43847711e0675bc39e31" exitCode=0
Oct 30 21:23:50.643349 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:50.642992    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e547c79b-m84wh" event={"ID":"afaab19a-aaae-4a24-9e43-0a7e4594641d","Type":"ContainerDied","Data":"567a08a3a0e5764238904d74e0842d7b2ee4dedea92b43847711e0675bc39e31"}
Oct 30 21:23:50.749201 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:50.749158    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-08df98-57c8c9bdbd-2nsdc"]
Oct 30 21:23:50.849832 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:50.849731    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-08df98-699dbdd999-t8wvt"]
Oct 30 21:23:51.787417 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:51.787394    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e547c79b-m84wh"
Oct 30 21:23:51.934295 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:51.934262    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afaab19a-aaae-4a24-9e43-0a7e4594641d-kube-api-access-x62r2\" (UniqueName: \"kubernetes.io/projected/afaab19a-aaae-4a24-9e43-0a7e4594641d-kube-api-access-x62r2\") pod \"afaab19a-aaae-4a24-9e43-0a7e4594641d\" (UID: \"afaab19a-aaae-4a24-9e43-0a7e4594641d\") "
Oct 30 21:23:51.934458 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:51.934446    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afaab19a-aaae-4a24-9e43-0a7e4594641d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afaab19a-aaae-4a24-9e43-0a7e4594641d-maas-api-xks-ca\") pod \"afaab19a-aaae-4a24-9e43-0a7e4594641d\" (UID: \"afaab19a-aaae-4a24-9e43-0a7e4594641d\") "
Oct 30 21:23:51.934521 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:51.934503    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afaab19a-aaae-4a24-9e43-0a7e4594641d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afaab19a-aaae-4a24-9e43-0a7e4594641d-maas-api-openshift-ca\") pod \"afaab19a-aaae-4a24-9e43-0a7e4594641d\" (UID: \"afaab19a-aaae-4a24-9e43-0a7e4594641d\") "
Oct 30 21:23:51.934929 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:51.934900    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afaab19a-aaae-4a24-9e43-0a7e4594641d-maas-api-openshift-ca" pod "afaab19a-aaae-4a24-9e43-0a7e4594641d" (UID: "afaab19a-aaae-4a24-9e43-0a7e4594641d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:51.935059 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:51.935027    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afaab19a-aaae-4a24-9e43-0a7e4594641d-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:23:51.937101 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:51.937074    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afaab19a-aaae-4a24-9e43-0a7e4594641d-kube-api-access-x62r2" pod "afaab19a-aaae-4a24-9e43-0a7e4594641d" (UID: "afaab19a-aaae-4a24-9e43-0a7e4594641d"). InnerVolumeSpecName "kube-api-access-x62r2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:51.937918 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:51.937893    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afaab19a-aaae-4a24-9e43-0a7e4594641d-maas-api-xks-ca" pod "afaab19a-aaae-4a24-9e43-0a7e4594641d" (UID: "afaab19a-aaae-4a24-9e43-0a7e4594641d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:23:52.036383 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:52.036350    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x62r2\" (UniqueName: \"kubernetes.io/projected/afaab19a-aaae-4a24-9e43-0a7e4594641d-kube-api-access-x62r2\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:23:52.036383 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:52.036380    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afaab19a-aaae-4a24-9e43-0a7e4594641d-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:23:52.658883 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:52.658852    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e547c79b-m84wh"
Oct 30 21:23:52.659077 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:52.658879    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e547c79b-m84wh" event={"ID":"afaab19a-aaae-4a24-9e43-0a7e4594641d","Type":"ContainerDied","Data":"689db030a0102e57af9e5501b0d26debaf74563241dabef42e121d1b7f89320d"}
Oct 30 21:23:52.659077 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:52.658928    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="689db030a0102e57af9e5501b0d26debaf74563241dabef42e121d1b7f89320d"
Oct 30 21:23:53.388915 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:53.388871    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-vmz5r"]
Oct 30 21:23:53.389530 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:53.389185    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-vmz5r" podUID="94c10451-978e-44bb-8fa5-864dcd6f6d93" containerName="istio-proxy" containerID="cri-o://44a3d1cade193a7daf80aed75493a7b2326af8ef29549b67503878efb8dd6cc3" gracePeriod=30
Oct 30 21:23:53.440463 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:53.440422    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5hlrx"]
Oct 30 21:23:53.440842 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:53.440811    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5hlrx" podUID="454f552c-5306-44d0-8d5d-df9555872183" containerName="istio-proxy" containerID="cri-o://953db0957b097a6cfee8cd7bee85597a6d1982031c8480cc7ce33894bf1d89cd" gracePeriod=30
Oct 30 21:23:53.756413 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:53.756377    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-5hlrx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:23:53.756610 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:53.756444    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5hlrx" podUID="454f552c-5306-44d0-8d5d-df9555872183" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:23:54.459129 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:23:54.459093    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:23:55.749703 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:55.749660    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-08df98-57c8c9bdbd-2nsdc" podUID="3b0becd4-be23-402c-8870-a49d40b04f90" containerName="payload-processing" containerID="cri-o://8eb1414e7fd9205b027baafcf20ec8f8cffbbe46552c04a25e2ab3747b6d8a15" gracePeriod=25
Oct 30 21:23:55.850350 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:55.850301    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-08df98-699dbdd999-t8wvt" podUID="aec20547-eca3-464b-9c2d-974ff1673459" containerName="payload-pre-processing" containerID="cri-o://a175d0a592d30634c21dc1848654e188d7c1d47fdf5585e6b389fed68d368dc5" gracePeriod=25
Oct 30 21:23:56.018541 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:56.018478    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-08df98-57c8c9bdbd-2nsdc"
Oct 30 21:23:56.197663 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:56.197631    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b0becd4-be23-402c-8870-a49d40b04f90-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b0becd4-be23-402c-8870-a49d40b04f90-plugins-config-volume\") pod \"3b0becd4-be23-402c-8870-a49d40b04f90\" (UID: \"3b0becd4-be23-402c-8870-a49d40b04f90\") "
Oct 30 21:23:56.197854 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:56.197843    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b0becd4-be23-402c-8870-a49d40b04f90-kube-api-access-wgsw4\" (UniqueName: \"kubernetes.io/projected/3b0becd4-be23-402c-8870-a49d40b04f90-kube-api-access-wgsw4\") pod \"3b0becd4-be23-402c-8870-a49d40b04f90\" (UID: \"3b0becd4-be23-402c-8870-a49d40b04f90\") "
Oct 30 21:23:56.198097 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:56.198067    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b0becd4-be23-402c-8870-a49d40b04f90-plugins-config-volume" pod "3b0becd4-be23-402c-8870-a49d40b04f90" (UID: "3b0becd4-be23-402c-8870-a49d40b04f90"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:56.198271 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:56.198251    5713 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b0becd4-be23-402c-8870-a49d40b04f90-plugins-config-volume\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:23:56.200184 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:56.200158    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b0becd4-be23-402c-8870-a49d40b04f90-kube-api-access-wgsw4" pod "3b0becd4-be23-402c-8870-a49d40b04f90" (UID: "3b0becd4-be23-402c-8870-a49d40b04f90"). InnerVolumeSpecName "kube-api-access-wgsw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:56.299964 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:56.299884    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgsw4\" (UniqueName: \"kubernetes.io/projected/3b0becd4-be23-402c-8870-a49d40b04f90-kube-api-access-wgsw4\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:23:56.395048 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:56.395008    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-08df98-699dbdd999-t8wvt" podUID="aec20547-eca3-464b-9c2d-974ff1673459" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:23:56.687906 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:56.687871    5713 generic.go:356] "Generic (PLEG): container finished" podID="3b0becd4-be23-402c-8870-a49d40b04f90" containerID="8eb1414e7fd9205b027baafcf20ec8f8cffbbe46552c04a25e2ab3747b6d8a15" exitCode=0
Oct 30 21:23:56.688127 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:56.687920    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-08df98-57c8c9bdbd-2nsdc"
Oct 30 21:23:56.688127 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:56.687955    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-08df98-57c8c9bdbd-2nsdc" event={"ID":"3b0becd4-be23-402c-8870-a49d40b04f90","Type":"ContainerDied","Data":"8eb1414e7fd9205b027baafcf20ec8f8cffbbe46552c04a25e2ab3747b6d8a15"}
Oct 30 21:23:56.688127 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:56.688016    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-08df98-57c8c9bdbd-2nsdc" event={"ID":"3b0becd4-be23-402c-8870-a49d40b04f90","Type":"ContainerDied","Data":"8e2c45a305b6ff0e52b724bfbc0098c254215a30e1704d60d25513bea0072f06"}
Oct 30 21:23:56.688127 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:56.688039    5713 scope.go:122] "RemoveContainer" containerID="8eb1414e7fd9205b027baafcf20ec8f8cffbbe46552c04a25e2ab3747b6d8a15"
Oct 30 21:23:56.700684 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:56.700663    5713 scope.go:122] "RemoveContainer" containerID="8eb1414e7fd9205b027baafcf20ec8f8cffbbe46552c04a25e2ab3747b6d8a15"
Oct 30 21:23:56.701024 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:23:56.701003    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8eb1414e7fd9205b027baafcf20ec8f8cffbbe46552c04a25e2ab3747b6d8a15\": container with ID starting with 8eb1414e7fd9205b027baafcf20ec8f8cffbbe46552c04a25e2ab3747b6d8a15 not found: ID does not exist" containerID="8eb1414e7fd9205b027baafcf20ec8f8cffbbe46552c04a25e2ab3747b6d8a15"
Oct 30 21:23:56.701107 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:56.701034    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8eb1414e7fd9205b027baafcf20ec8f8cffbbe46552c04a25e2ab3747b6d8a15"} err="failed to get container status \"8eb1414e7fd9205b027baafcf20ec8f8cffbbe46552c04a25e2ab3747b6d8a15\": rpc error: code = NotFound desc = could not find container \"8eb1414e7fd9205b027baafcf20ec8f8cffbbe46552c04a25e2ab3747b6d8a15\": container with ID starting with 8eb1414e7fd9205b027baafcf20ec8f8cffbbe46552c04a25e2ab3747b6d8a15 not found: ID does not exist"
Oct 30 21:23:56.851989 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:56.851959    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-08df98-57c8c9bdbd-2nsdc"]
Oct 30 21:23:56.857247 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:56.857217    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-08df98-57c8c9bdbd-2nsdc"]
Oct 30 21:23:57.064545 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:57.064465    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b0becd4-be23-402c-8870-a49d40b04f90" path="/var/lib/kubelet/pods/3b0becd4-be23-402c-8870-a49d40b04f90/volumes"
Oct 30 21:23:58.707853 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:58.707671    5713 generic.go:356] "Generic (PLEG): container finished" podID="94c10451-978e-44bb-8fa5-864dcd6f6d93" containerID="44a3d1cade193a7daf80aed75493a7b2326af8ef29549b67503878efb8dd6cc3" exitCode=0
Oct 30 21:23:58.707853 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:58.707741    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-vmz5r" event={"ID":"94c10451-978e-44bb-8fa5-864dcd6f6d93","Type":"ContainerDied","Data":"44a3d1cade193a7daf80aed75493a7b2326af8ef29549b67503878efb8dd6cc3"}
Oct 30 21:23:58.710163 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:58.710142    5713 generic.go:356] "Generic (PLEG): container finished" podID="454f552c-5306-44d0-8d5d-df9555872183" containerID="953db0957b097a6cfee8cd7bee85597a6d1982031c8480cc7ce33894bf1d89cd" exitCode=0
Oct 30 21:23:58.710279 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:58.710219    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5hlrx" event={"ID":"454f552c-5306-44d0-8d5d-df9555872183","Type":"ContainerDied","Data":"953db0957b097a6cfee8cd7bee85597a6d1982031c8480cc7ce33894bf1d89cd"}
Oct 30 21:23:58.742189 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:58.742159    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-vmz5r"
Oct 30 21:23:58.745173 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:58.745156    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5hlrx"
Oct 30 21:23:58.935513 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:58.935481    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/454f552c-5306-44d0-8d5d-df9555872183-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/454f552c-5306-44d0-8d5d-df9555872183-istio-podinfo\") pod \"454f552c-5306-44d0-8d5d-df9555872183\" (UID: \"454f552c-5306-44d0-8d5d-df9555872183\") "
Oct 30 21:23:58.935716 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:58.935541    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94c10451-978e-44bb-8fa5-864dcd6f6d93-istio-token\" (UniqueName: \"kubernetes.io/projected/94c10451-978e-44bb-8fa5-864dcd6f6d93-istio-token\") pod \"94c10451-978e-44bb-8fa5-864dcd6f6d93\" (UID: \"94c10451-978e-44bb-8fa5-864dcd6f6d93\") "
Oct 30 21:23:58.935716 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:58.935574    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94c10451-978e-44bb-8fa5-864dcd6f6d93-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c10451-978e-44bb-8fa5-864dcd6f6d93-credential-socket\") pod \"94c10451-978e-44bb-8fa5-864dcd6f6d93\" (UID: \"94c10451-978e-44bb-8fa5-864dcd6f6d93\") "
Oct 30 21:23:58.935716 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:58.935621    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/454f552c-5306-44d0-8d5d-df9555872183-kube-api-access-8s8pb\" (UniqueName: \"kubernetes.io/projected/454f552c-5306-44d0-8d5d-df9555872183-kube-api-access-8s8pb\") pod \"454f552c-5306-44d0-8d5d-df9555872183\" (UID: \"454f552c-5306-44d0-8d5d-df9555872183\") "
Oct 30 21:23:58.935716 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:58.935669    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/454f552c-5306-44d0-8d5d-df9555872183-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/454f552c-5306-44d0-8d5d-df9555872183-istiod-ca-cert\") pod \"454f552c-5306-44d0-8d5d-df9555872183\" (UID: \"454f552c-5306-44d0-8d5d-df9555872183\") "
Oct 30 21:23:58.935984 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:58.935720    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/454f552c-5306-44d0-8d5d-df9555872183-istio-token\" (UniqueName: \"kubernetes.io/projected/454f552c-5306-44d0-8d5d-df9555872183-istio-token\") pod \"454f552c-5306-44d0-8d5d-df9555872183\" (UID: \"454f552c-5306-44d0-8d5d-df9555872183\") "
Oct 30 21:23:58.935984 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:58.935757    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/454f552c-5306-44d0-8d5d-df9555872183-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/454f552c-5306-44d0-8d5d-df9555872183-workload-socket\") pod \"454f552c-5306-44d0-8d5d-df9555872183\" (UID: \"454f552c-5306-44d0-8d5d-df9555872183\") "
Oct 30 21:23:58.935984 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:58.935840    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/454f552c-5306-44d0-8d5d-df9555872183-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/454f552c-5306-44d0-8d5d-df9555872183-istio-envoy\") pod \"454f552c-5306-44d0-8d5d-df9555872183\" (UID: \"454f552c-5306-44d0-8d5d-df9555872183\") "
Oct 30 21:23:58.935984 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:58.935897    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/94c10451-978e-44bb-8fa5-864dcd6f6d93-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94c10451-978e-44bb-8fa5-864dcd6f6d93-istio-podinfo\") pod \"94c10451-978e-44bb-8fa5-864dcd6f6d93\" (UID: \"94c10451-978e-44bb-8fa5-864dcd6f6d93\") "
Oct 30 21:23:58.935984 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:58.935917    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/454f552c-5306-44d0-8d5d-df9555872183-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/454f552c-5306-44d0-8d5d-df9555872183-credential-socket\") pod \"454f552c-5306-44d0-8d5d-df9555872183\" (UID: \"454f552c-5306-44d0-8d5d-df9555872183\") "
Oct 30 21:23:58.935984 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:58.935963    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94c10451-978e-44bb-8fa5-864dcd6f6d93-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94c10451-978e-44bb-8fa5-864dcd6f6d93-istio-data\") pod \"94c10451-978e-44bb-8fa5-864dcd6f6d93\" (UID: \"94c10451-978e-44bb-8fa5-864dcd6f6d93\") "
Oct 30 21:23:58.936298 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:58.936023    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/454f552c-5306-44d0-8d5d-df9555872183-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/454f552c-5306-44d0-8d5d-df9555872183-workload-certs\") pod \"454f552c-5306-44d0-8d5d-df9555872183\" (UID: \"454f552c-5306-44d0-8d5d-df9555872183\") "
Oct 30 21:23:58.936298 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:58.936119    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94c10451-978e-44bb-8fa5-864dcd6f6d93-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c10451-978e-44bb-8fa5-864dcd6f6d93-workload-socket\") pod \"94c10451-978e-44bb-8fa5-864dcd6f6d93\" (UID: \"94c10451-978e-44bb-8fa5-864dcd6f6d93\") "
Oct 30 21:23:58.936298 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:58.936211    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94c10451-978e-44bb-8fa5-864dcd6f6d93-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94c10451-978e-44bb-8fa5-864dcd6f6d93-istiod-ca-cert\") pod \"94c10451-978e-44bb-8fa5-864dcd6f6d93\" (UID: \"94c10451-978e-44bb-8fa5-864dcd6f6d93\") "
Oct 30 21:23:58.936298 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:58.936210    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94c10451-978e-44bb-8fa5-864dcd6f6d93-credential-socket" pod "94c10451-978e-44bb-8fa5-864dcd6f6d93" (UID: "94c10451-978e-44bb-8fa5-864dcd6f6d93"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:58.936509 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:58.936298    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94c10451-978e-44bb-8fa5-864dcd6f6d93-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94c10451-978e-44bb-8fa5-864dcd6f6d93-workload-certs\") pod \"94c10451-978e-44bb-8fa5-864dcd6f6d93\" (UID: \"94c10451-978e-44bb-8fa5-864dcd6f6d93\") "
Oct 30 21:23:58.936509 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:58.936493    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94c10451-978e-44bb-8fa5-864dcd6f6d93-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94c10451-978e-44bb-8fa5-864dcd6f6d93-istio-envoy\") pod \"94c10451-978e-44bb-8fa5-864dcd6f6d93\" (UID: \"94c10451-978e-44bb-8fa5-864dcd6f6d93\") "
Oct 30 21:23:58.936621 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:58.936542    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/454f552c-5306-44d0-8d5d-df9555872183-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/454f552c-5306-44d0-8d5d-df9555872183-istio-data\") pod \"454f552c-5306-44d0-8d5d-df9555872183\" (UID: \"454f552c-5306-44d0-8d5d-df9555872183\") "
Oct 30 21:23:58.936621 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:58.936577    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94c10451-978e-44bb-8fa5-864dcd6f6d93-kube-api-access-n7v5w\" (UniqueName: \"kubernetes.io/projected/94c10451-978e-44bb-8fa5-864dcd6f6d93-kube-api-access-n7v5w\") pod \"94c10451-978e-44bb-8fa5-864dcd6f6d93\" (UID: \"94c10451-978e-44bb-8fa5-864dcd6f6d93\") "
Oct 30 21:23:58.936723 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:58.936570    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/454f552c-5306-44d0-8d5d-df9555872183-istiod-ca-cert" pod "454f552c-5306-44d0-8d5d-df9555872183" (UID: "454f552c-5306-44d0-8d5d-df9555872183"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:58.937660 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:58.937039    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/454f552c-5306-44d0-8d5d-df9555872183-workload-certs" pod "454f552c-5306-44d0-8d5d-df9555872183" (UID: "454f552c-5306-44d0-8d5d-df9555872183"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:58.937660 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:58.937256    5713 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c10451-978e-44bb-8fa5-864dcd6f6d93-credential-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:23:58.937660 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:58.937276    5713 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/454f552c-5306-44d0-8d5d-df9555872183-istiod-ca-cert\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:23:58.937660 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:58.937292    5713 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/454f552c-5306-44d0-8d5d-df9555872183-workload-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:23:58.937660 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:58.937286    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94c10451-978e-44bb-8fa5-864dcd6f6d93-workload-socket" pod "94c10451-978e-44bb-8fa5-864dcd6f6d93" (UID: "94c10451-978e-44bb-8fa5-864dcd6f6d93"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:58.937660 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:58.937444    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94c10451-978e-44bb-8fa5-864dcd6f6d93-workload-certs" pod "94c10451-978e-44bb-8fa5-864dcd6f6d93" (UID: "94c10451-978e-44bb-8fa5-864dcd6f6d93"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:58.937660 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:58.937448    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/454f552c-5306-44d0-8d5d-df9555872183-workload-socket" pod "454f552c-5306-44d0-8d5d-df9555872183" (UID: "454f552c-5306-44d0-8d5d-df9555872183"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:58.937660 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:58.937612    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/454f552c-5306-44d0-8d5d-df9555872183-credential-socket" pod "454f552c-5306-44d0-8d5d-df9555872183" (UID: "454f552c-5306-44d0-8d5d-df9555872183"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:58.938251 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:58.937892    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94c10451-978e-44bb-8fa5-864dcd6f6d93-istiod-ca-cert" pod "94c10451-978e-44bb-8fa5-864dcd6f6d93" (UID: "94c10451-978e-44bb-8fa5-864dcd6f6d93"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:58.938756 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:58.938504    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94c10451-978e-44bb-8fa5-864dcd6f6d93-istio-data" pod "94c10451-978e-44bb-8fa5-864dcd6f6d93" (UID: "94c10451-978e-44bb-8fa5-864dcd6f6d93"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:58.938756 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:58.938704    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/454f552c-5306-44d0-8d5d-df9555872183-istio-data" pod "454f552c-5306-44d0-8d5d-df9555872183" (UID: "454f552c-5306-44d0-8d5d-df9555872183"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:58.938952 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:58.938853    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/454f552c-5306-44d0-8d5d-df9555872183-istio-token" pod "454f552c-5306-44d0-8d5d-df9555872183" (UID: "454f552c-5306-44d0-8d5d-df9555872183"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:58.940823 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:58.940772    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94c10451-978e-44bb-8fa5-864dcd6f6d93-istio-envoy" pod "94c10451-978e-44bb-8fa5-864dcd6f6d93" (UID: "94c10451-978e-44bb-8fa5-864dcd6f6d93"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:58.941082 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:58.941058    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/454f552c-5306-44d0-8d5d-df9555872183-istio-podinfo" pod "454f552c-5306-44d0-8d5d-df9555872183" (UID: "454f552c-5306-44d0-8d5d-df9555872183"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:23:58.941183 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:58.941167    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94c10451-978e-44bb-8fa5-864dcd6f6d93-istio-token" pod "94c10451-978e-44bb-8fa5-864dcd6f6d93" (UID: "94c10451-978e-44bb-8fa5-864dcd6f6d93"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:58.941660 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:58.941636    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94c10451-978e-44bb-8fa5-864dcd6f6d93-kube-api-access-n7v5w" pod "94c10451-978e-44bb-8fa5-864dcd6f6d93" (UID: "94c10451-978e-44bb-8fa5-864dcd6f6d93"). InnerVolumeSpecName "kube-api-access-n7v5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:58.941761 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:58.941727    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/454f552c-5306-44d0-8d5d-df9555872183-kube-api-access-8s8pb" pod "454f552c-5306-44d0-8d5d-df9555872183" (UID: "454f552c-5306-44d0-8d5d-df9555872183"). InnerVolumeSpecName "kube-api-access-8s8pb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:58.942700 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:58.942678    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/94c10451-978e-44bb-8fa5-864dcd6f6d93-istio-podinfo" pod "94c10451-978e-44bb-8fa5-864dcd6f6d93" (UID: "94c10451-978e-44bb-8fa5-864dcd6f6d93"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:23:58.943043 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:58.943023    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/454f552c-5306-44d0-8d5d-df9555872183-istio-envoy" pod "454f552c-5306-44d0-8d5d-df9555872183" (UID: "454f552c-5306-44d0-8d5d-df9555872183"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:59.038847 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:59.038520    5713 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94c10451-978e-44bb-8fa5-864dcd6f6d93-istiod-ca-cert\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:23:59.038847 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:59.038560    5713 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94c10451-978e-44bb-8fa5-864dcd6f6d93-workload-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:23:59.038847 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:59.038575    5713 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94c10451-978e-44bb-8fa5-864dcd6f6d93-istio-envoy\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:23:59.038847 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:59.038589    5713 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/454f552c-5306-44d0-8d5d-df9555872183-istio-data\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:23:59.038847 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:59.038606    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7v5w\" (UniqueName: \"kubernetes.io/projected/94c10451-978e-44bb-8fa5-864dcd6f6d93-kube-api-access-n7v5w\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:23:59.038847 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:59.038622    5713 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/454f552c-5306-44d0-8d5d-df9555872183-istio-podinfo\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:23:59.038847 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:59.038636    5713 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94c10451-978e-44bb-8fa5-864dcd6f6d93-istio-token\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:23:59.038847 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:59.038651    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8s8pb\" (UniqueName: \"kubernetes.io/projected/454f552c-5306-44d0-8d5d-df9555872183-kube-api-access-8s8pb\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:23:59.038847 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:59.038666    5713 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/454f552c-5306-44d0-8d5d-df9555872183-istio-token\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:23:59.038847 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:59.038680    5713 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/454f552c-5306-44d0-8d5d-df9555872183-workload-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:23:59.038847 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:59.038697    5713 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/454f552c-5306-44d0-8d5d-df9555872183-istio-envoy\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:23:59.038847 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:59.038711    5713 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94c10451-978e-44bb-8fa5-864dcd6f6d93-istio-podinfo\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:23:59.038847 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:59.038726    5713 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/454f552c-5306-44d0-8d5d-df9555872183-credential-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:23:59.038847 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:59.038740    5713 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94c10451-978e-44bb-8fa5-864dcd6f6d93-istio-data\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:23:59.038847 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:59.038754    5713 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c10451-978e-44bb-8fa5-864dcd6f6d93-workload-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:23:59.718559 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:59.718528    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-vmz5r"
Oct 30 21:23:59.719097 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:59.718530    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-vmz5r" event={"ID":"94c10451-978e-44bb-8fa5-864dcd6f6d93","Type":"ContainerDied","Data":"689b180edf50240abb26217d148050cc2053a7aa34d6b5c9e116bc5194013ff8"}
Oct 30 21:23:59.719097 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:59.718750    5713 scope.go:122] "RemoveContainer" containerID="44a3d1cade193a7daf80aed75493a7b2326af8ef29549b67503878efb8dd6cc3"
Oct 30 21:23:59.720368 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:59.720270    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5hlrx"
Oct 30 21:23:59.720368 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:59.720265    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5hlrx" event={"ID":"454f552c-5306-44d0-8d5d-df9555872183","Type":"ContainerDied","Data":"fedf1e6aa57b558e333b025c93d60ce263d124783b5ffb141614307b5d219ff5"}
Oct 30 21:23:59.730206 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:59.730185    5713 scope.go:122] "RemoveContainer" containerID="953db0957b097a6cfee8cd7bee85597a6d1982031c8480cc7ce33894bf1d89cd"
Oct 30 21:23:59.885305 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:59.885272    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-vmz5r"]
Oct 30 21:23:59.935142 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:59.935114    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-vmz5r"]
Oct 30 21:23:59.998026 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:23:59.997934    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5hlrx"]
Oct 30 21:24:00.000726 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:00.000698    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-5hlrx"]
Oct 30 21:24:01.064254 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:01.064223    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="454f552c-5306-44d0-8d5d-df9555872183" path="/var/lib/kubelet/pods/454f552c-5306-44d0-8d5d-df9555872183/volumes"
Oct 30 21:24:01.064903 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:01.064886    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94c10451-978e-44bb-8fa5-864dcd6f6d93" path="/var/lib/kubelet/pods/94c10451-978e-44bb-8fa5-864dcd6f6d93/volumes"
Oct 30 21:24:02.924682 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:02.924640    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-f0fe13/test-model-f0fe13-kserve-c746dc965-tkz9t" podUID="f4c9a7f4-79ff-4f0b-855a-d0850f5a3115" containerName="main" containerID="cri-o://e1f7c78a6675631615f2ed193d5d8235024d0436ce9c4a483007adfb56aba349" gracePeriod=45
Oct 30 21:24:03.184353 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:03.184328    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f0fe13/test-model-f0fe13-kserve-c746dc965-tkz9t"
Oct 30 21:24:03.285072 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:03.285032    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4c9a7f4-79ff-4f0b-855a-d0850f5a3115-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4c9a7f4-79ff-4f0b-855a-d0850f5a3115-model-cache\") pod \"f4c9a7f4-79ff-4f0b-855a-d0850f5a3115\" (UID: \"f4c9a7f4-79ff-4f0b-855a-d0850f5a3115\") "
Oct 30 21:24:03.285072 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:03.285093    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4c9a7f4-79ff-4f0b-855a-d0850f5a3115-kube-api-access-xqp6c\" (UniqueName: \"kubernetes.io/projected/f4c9a7f4-79ff-4f0b-855a-d0850f5a3115-kube-api-access-xqp6c\") pod \"f4c9a7f4-79ff-4f0b-855a-d0850f5a3115\" (UID: \"f4c9a7f4-79ff-4f0b-855a-d0850f5a3115\") "
Oct 30 21:24:03.285338 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:03.285128    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4c9a7f4-79ff-4f0b-855a-d0850f5a3115-home\" (UniqueName: \"kubernetes.io/empty-dir/f4c9a7f4-79ff-4f0b-855a-d0850f5a3115-home\") pod \"f4c9a7f4-79ff-4f0b-855a-d0850f5a3115\" (UID: \"f4c9a7f4-79ff-4f0b-855a-d0850f5a3115\") "
Oct 30 21:24:03.285338 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:03.285186    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4c9a7f4-79ff-4f0b-855a-d0850f5a3115-tls-certs\" (UniqueName: \"kubernetes.io/secret/f4c9a7f4-79ff-4f0b-855a-d0850f5a3115-tls-certs\") pod \"f4c9a7f4-79ff-4f0b-855a-d0850f5a3115\" (UID: \"f4c9a7f4-79ff-4f0b-855a-d0850f5a3115\") "
Oct 30 21:24:03.285338 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:03.285229    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4c9a7f4-79ff-4f0b-855a-d0850f5a3115-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4c9a7f4-79ff-4f0b-855a-d0850f5a3115-tmp-dir\") pod \"f4c9a7f4-79ff-4f0b-855a-d0850f5a3115\" (UID: \"f4c9a7f4-79ff-4f0b-855a-d0850f5a3115\") "
Oct 30 21:24:03.285338 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:03.285253    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4c9a7f4-79ff-4f0b-855a-d0850f5a3115-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4c9a7f4-79ff-4f0b-855a-d0850f5a3115-dshm\") pod \"f4c9a7f4-79ff-4f0b-855a-d0850f5a3115\" (UID: \"f4c9a7f4-79ff-4f0b-855a-d0850f5a3115\") "
Oct 30 21:24:03.285338 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:03.285304    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4c9a7f4-79ff-4f0b-855a-d0850f5a3115-model-cache" pod "f4c9a7f4-79ff-4f0b-855a-d0850f5a3115" (UID: "f4c9a7f4-79ff-4f0b-855a-d0850f5a3115"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:24:03.285624 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:03.285413    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4c9a7f4-79ff-4f0b-855a-d0850f5a3115-home" pod "f4c9a7f4-79ff-4f0b-855a-d0850f5a3115" (UID: "f4c9a7f4-79ff-4f0b-855a-d0850f5a3115"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:24:03.285624 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:03.285483    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4c9a7f4-79ff-4f0b-855a-d0850f5a3115-tmp-dir" pod "f4c9a7f4-79ff-4f0b-855a-d0850f5a3115" (UID: "f4c9a7f4-79ff-4f0b-855a-d0850f5a3115"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:24:03.285736 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:03.285634    5713 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4c9a7f4-79ff-4f0b-855a-d0850f5a3115-model-cache\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:03.285736 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:03.285648    5713 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4c9a7f4-79ff-4f0b-855a-d0850f5a3115-home\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:03.285736 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:03.285658    5713 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4c9a7f4-79ff-4f0b-855a-d0850f5a3115-tmp-dir\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:03.287601 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:03.287560    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4c9a7f4-79ff-4f0b-855a-d0850f5a3115-tls-certs" pod "f4c9a7f4-79ff-4f0b-855a-d0850f5a3115" (UID: "f4c9a7f4-79ff-4f0b-855a-d0850f5a3115"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:24:03.287601 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:03.287577    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4c9a7f4-79ff-4f0b-855a-d0850f5a3115-kube-api-access-xqp6c" pod "f4c9a7f4-79ff-4f0b-855a-d0850f5a3115" (UID: "f4c9a7f4-79ff-4f0b-855a-d0850f5a3115"). InnerVolumeSpecName "kube-api-access-xqp6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:24:03.287744 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:03.287626    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4c9a7f4-79ff-4f0b-855a-d0850f5a3115-dshm" pod "f4c9a7f4-79ff-4f0b-855a-d0850f5a3115" (UID: "f4c9a7f4-79ff-4f0b-855a-d0850f5a3115"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:24:03.386438 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:03.386406    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqp6c\" (UniqueName: \"kubernetes.io/projected/f4c9a7f4-79ff-4f0b-855a-d0850f5a3115-kube-api-access-xqp6c\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:03.386438 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:03.386432    5713 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4c9a7f4-79ff-4f0b-855a-d0850f5a3115-tls-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:03.386438 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:03.386441    5713 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4c9a7f4-79ff-4f0b-855a-d0850f5a3115-dshm\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:03.751245 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:03.751207    5713 generic.go:356] "Generic (PLEG): container finished" podID="f4c9a7f4-79ff-4f0b-855a-d0850f5a3115" containerID="e1f7c78a6675631615f2ed193d5d8235024d0436ce9c4a483007adfb56aba349" exitCode=0
Oct 30 21:24:03.751245 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:03.751248    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f0fe13/test-model-f0fe13-kserve-c746dc965-tkz9t"
Oct 30 21:24:03.751467 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:03.751296    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f0fe13/test-model-f0fe13-kserve-c746dc965-tkz9t" event={"ID":"f4c9a7f4-79ff-4f0b-855a-d0850f5a3115","Type":"ContainerDied","Data":"e1f7c78a6675631615f2ed193d5d8235024d0436ce9c4a483007adfb56aba349"}
Oct 30 21:24:03.751467 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:03.751336    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f0fe13/test-model-f0fe13-kserve-c746dc965-tkz9t" event={"ID":"f4c9a7f4-79ff-4f0b-855a-d0850f5a3115","Type":"ContainerDied","Data":"90550b95ba0bda4247fb9a8180460e3c393a543a52598882dfcf74a7713ba045"}
Oct 30 21:24:03.751467 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:03.751361    5713 scope.go:122] "RemoveContainer" containerID="e1f7c78a6675631615f2ed193d5d8235024d0436ce9c4a483007adfb56aba349"
Oct 30 21:24:03.762005 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:03.761985    5713 scope.go:122] "RemoveContainer" containerID="e1f7c78a6675631615f2ed193d5d8235024d0436ce9c4a483007adfb56aba349"
Oct 30 21:24:03.762291 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:24:03.762267    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1f7c78a6675631615f2ed193d5d8235024d0436ce9c4a483007adfb56aba349\": container with ID starting with e1f7c78a6675631615f2ed193d5d8235024d0436ce9c4a483007adfb56aba349 not found: ID does not exist" containerID="e1f7c78a6675631615f2ed193d5d8235024d0436ce9c4a483007adfb56aba349"
Oct 30 21:24:03.762341 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:03.762304    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1f7c78a6675631615f2ed193d5d8235024d0436ce9c4a483007adfb56aba349"} err="failed to get container status \"e1f7c78a6675631615f2ed193d5d8235024d0436ce9c4a483007adfb56aba349\": rpc error: code = NotFound desc = could not find container \"e1f7c78a6675631615f2ed193d5d8235024d0436ce9c4a483007adfb56aba349\": container with ID starting with e1f7c78a6675631615f2ed193d5d8235024d0436ce9c4a483007adfb56aba349 not found: ID does not exist"
Oct 30 21:24:03.914461 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:03.914397    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f0fe13/test-model-f0fe13-kserve-c746dc965-tkz9t"]
Oct 30 21:24:03.918331 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:03.918301    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-f0fe13/test-model-f0fe13-kserve-c746dc965-tkz9t"]
Oct 30 21:24:05.064366 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:05.064333    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4c9a7f4-79ff-4f0b-855a-d0850f5a3115" path="/var/lib/kubelet/pods/f4c9a7f4-79ff-4f0b-855a-d0850f5a3115/volumes"
Oct 30 21:24:06.394907 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:06.394871    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-08df98-699dbdd999-t8wvt" podUID="aec20547-eca3-464b-9c2d-974ff1673459" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:24:10.865877 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:10.865845    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f0fe13-sg5xm"]
Oct 30 21:24:10.866352 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:10.866338    5713 cpu_manager.go:414] "containerMap: removing container" podUID="afaab19a-aaae-4a24-9e43-0a7e4594641d" containerName="revoke-keys"
Oct 30 21:24:10.866408 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:10.866353    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afaab19a-aaae-4a24-9e43-0a7e4594641d" containerName="revoke-keys"
Oct 30 21:24:10.866408 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:10.866374    5713 cpu_manager.go:414] "containerMap: removing container" podUID="454f552c-5306-44d0-8d5d-df9555872183" containerName="istio-proxy"
Oct 30 21:24:10.866408 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:10.866378    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="454f552c-5306-44d0-8d5d-df9555872183" containerName="istio-proxy"
Oct 30 21:24:10.866408 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:10.866395    5713 cpu_manager.go:414] "containerMap: removing container" podUID="94c10451-978e-44bb-8fa5-864dcd6f6d93" containerName="istio-proxy"
Oct 30 21:24:10.866408 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:10.866400    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94c10451-978e-44bb-8fa5-864dcd6f6d93" containerName="istio-proxy"
Oct 30 21:24:10.866578 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:10.866419    5713 cpu_manager.go:414] "containerMap: removing container" podUID="3b0becd4-be23-402c-8870-a49d40b04f90" containerName="payload-processing"
Oct 30 21:24:10.866578 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:10.866424    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b0becd4-be23-402c-8870-a49d40b04f90" containerName="payload-processing"
Oct 30 21:24:10.866578 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:10.866430    5713 cpu_manager.go:414] "containerMap: removing container" podUID="f4c9a7f4-79ff-4f0b-855a-d0850f5a3115" containerName="main"
Oct 30 21:24:10.866578 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:10.866434    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4c9a7f4-79ff-4f0b-855a-d0850f5a3115" containerName="main"
Oct 30 21:24:10.866578 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:10.866511    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b0becd4-be23-402c-8870-a49d40b04f90" containerName="payload-processing"
Oct 30 21:24:10.866578 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:10.866521    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4c9a7f4-79ff-4f0b-855a-d0850f5a3115" containerName="main"
Oct 30 21:24:10.866578 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:10.866531    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="afaab19a-aaae-4a24-9e43-0a7e4594641d" containerName="revoke-keys"
Oct 30 21:24:10.866578 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:10.866539    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="454f552c-5306-44d0-8d5d-df9555872183" containerName="istio-proxy"
Oct 30 21:24:10.866578 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:10.866547    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="94c10451-978e-44bb-8fa5-864dcd6f6d93" containerName="istio-proxy"
Oct 30 21:24:10.871177 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:10.871159    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f0fe13-sg5xm"
Oct 30 21:24:10.923536 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:10.923507    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:24:10.923671 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:10.923588    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:24:10.926826 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:10.926799    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f0fe13-sg5xm"]
Oct 30 21:24:10.972466 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:10.972418    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:24:10.973006 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:10.972983    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:24:10.983509 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:10.983473    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vsdq\" (UniqueName: \"kubernetes.io/projected/b6d68dc1-ba46-490a-98ac-09208798fb71-kube-api-access-4vsdq\") pod \"maas-api-revoke-keys-e2e-inf-a-f0fe13-sg5xm\" (UID: \"b6d68dc1-ba46-490a-98ac-09208798fb71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f0fe13-sg5xm"
Oct 30 21:24:10.983691 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:10.983671    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6d68dc1-ba46-490a-98ac-09208798fb71-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f0fe13-sg5xm\" (UID: \"b6d68dc1-ba46-490a-98ac-09208798fb71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f0fe13-sg5xm"
Oct 30 21:24:10.983760 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:10.983736    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6d68dc1-ba46-490a-98ac-09208798fb71-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f0fe13-sg5xm\" (UID: \"b6d68dc1-ba46-490a-98ac-09208798fb71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f0fe13-sg5xm"
Oct 30 21:24:11.085089 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:11.085041    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vsdq\" (UniqueName: \"kubernetes.io/projected/b6d68dc1-ba46-490a-98ac-09208798fb71-kube-api-access-4vsdq\") pod \"maas-api-revoke-keys-e2e-inf-a-f0fe13-sg5xm\" (UID: \"b6d68dc1-ba46-490a-98ac-09208798fb71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f0fe13-sg5xm"
Oct 30 21:24:11.085245 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:11.085224    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6d68dc1-ba46-490a-98ac-09208798fb71-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f0fe13-sg5xm\" (UID: \"b6d68dc1-ba46-490a-98ac-09208798fb71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f0fe13-sg5xm"
Oct 30 21:24:11.085322 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:11.085280    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6d68dc1-ba46-490a-98ac-09208798fb71-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f0fe13-sg5xm\" (UID: \"b6d68dc1-ba46-490a-98ac-09208798fb71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f0fe13-sg5xm"
Oct 30 21:24:11.085905 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:11.085881    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6d68dc1-ba46-490a-98ac-09208798fb71-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f0fe13-sg5xm\" (UID: \"b6d68dc1-ba46-490a-98ac-09208798fb71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f0fe13-sg5xm"
Oct 30 21:24:11.087731 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:11.087703    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6d68dc1-ba46-490a-98ac-09208798fb71-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f0fe13-sg5xm\" (UID: \"b6d68dc1-ba46-490a-98ac-09208798fb71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f0fe13-sg5xm"
Oct 30 21:24:11.141373 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:11.141318    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vsdq\" (UniqueName: \"kubernetes.io/projected/b6d68dc1-ba46-490a-98ac-09208798fb71-kube-api-access-4vsdq\") pod \"maas-api-revoke-keys-e2e-inf-a-f0fe13-sg5xm\" (UID: \"b6d68dc1-ba46-490a-98ac-09208798fb71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f0fe13-sg5xm"
Oct 30 21:24:11.184350 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:11.184315    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f0fe13-sg5xm"
Oct 30 21:24:11.564953 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:11.564926    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f0fe13-sg5xm"]
Oct 30 21:24:11.566585 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:24:11.566556    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6d68dc1_ba46_490a_98ac_09208798fb71.slice/crio-13d657f0e3d247de53336969d4b684c2aa00a12b537ba6525408adf154ee8a7c WatchSource:0}: Error finding container 13d657f0e3d247de53336969d4b684c2aa00a12b537ba6525408adf154ee8a7c: Status 404 returned error can't find the container with id 13d657f0e3d247de53336969d4b684c2aa00a12b537ba6525408adf154ee8a7c
Oct 30 21:24:11.814006 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:11.812870    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f0fe13-sg5xm" event={"ID":"b6d68dc1-ba46-490a-98ac-09208798fb71","Type":"ContainerStarted","Data":"13d657f0e3d247de53336969d4b684c2aa00a12b537ba6525408adf154ee8a7c"}
Oct 30 21:24:12.821738 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:12.821637    5713 generic.go:356] "Generic (PLEG): container finished" podID="b6d68dc1-ba46-490a-98ac-09208798fb71" containerID="294f2f5bfe666597c321a5672f94a0f813f1bc4c485ac620b525267466f9302c" exitCode=0
Oct 30 21:24:12.821738 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:12.821721    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f0fe13-sg5xm" event={"ID":"b6d68dc1-ba46-490a-98ac-09208798fb71","Type":"ContainerDied","Data":"294f2f5bfe666597c321a5672f94a0f813f1bc4c485ac620b525267466f9302c"}
Oct 30 21:24:13.975693 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:13.975665    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f0fe13-sg5xm"
Oct 30 21:24:14.122018 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:14.121929    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6d68dc1-ba46-490a-98ac-09208798fb71-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6d68dc1-ba46-490a-98ac-09208798fb71-maas-api-openshift-ca\") pod \"b6d68dc1-ba46-490a-98ac-09208798fb71\" (UID: \"b6d68dc1-ba46-490a-98ac-09208798fb71\") "
Oct 30 21:24:14.122018 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:14.121973    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6d68dc1-ba46-490a-98ac-09208798fb71-kube-api-access-4vsdq\" (UniqueName: \"kubernetes.io/projected/b6d68dc1-ba46-490a-98ac-09208798fb71-kube-api-access-4vsdq\") pod \"b6d68dc1-ba46-490a-98ac-09208798fb71\" (UID: \"b6d68dc1-ba46-490a-98ac-09208798fb71\") "
Oct 30 21:24:14.122265 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:14.122141    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6d68dc1-ba46-490a-98ac-09208798fb71-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6d68dc1-ba46-490a-98ac-09208798fb71-maas-api-xks-ca\") pod \"b6d68dc1-ba46-490a-98ac-09208798fb71\" (UID: \"b6d68dc1-ba46-490a-98ac-09208798fb71\") "
Oct 30 21:24:14.122388 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:14.122357    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6d68dc1-ba46-490a-98ac-09208798fb71-maas-api-openshift-ca" pod "b6d68dc1-ba46-490a-98ac-09208798fb71" (UID: "b6d68dc1-ba46-490a-98ac-09208798fb71"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:24:14.122700 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:14.122677    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6d68dc1-ba46-490a-98ac-09208798fb71-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:14.124506 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:14.124483    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6d68dc1-ba46-490a-98ac-09208798fb71-maas-api-xks-ca" pod "b6d68dc1-ba46-490a-98ac-09208798fb71" (UID: "b6d68dc1-ba46-490a-98ac-09208798fb71"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:24:14.124832 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:14.124815    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6d68dc1-ba46-490a-98ac-09208798fb71-kube-api-access-4vsdq" pod "b6d68dc1-ba46-490a-98ac-09208798fb71" (UID: "b6d68dc1-ba46-490a-98ac-09208798fb71"). InnerVolumeSpecName "kube-api-access-4vsdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:24:14.224306 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:14.224277    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6d68dc1-ba46-490a-98ac-09208798fb71-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:14.224306 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:14.224302    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vsdq\" (UniqueName: \"kubernetes.io/projected/b6d68dc1-ba46-490a-98ac-09208798fb71-kube-api-access-4vsdq\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:14.841408 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:14.841371    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f0fe13-sg5xm" event={"ID":"b6d68dc1-ba46-490a-98ac-09208798fb71","Type":"ContainerDied","Data":"13d657f0e3d247de53336969d4b684c2aa00a12b537ba6525408adf154ee8a7c"}
Oct 30 21:24:14.841614 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:14.841433    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13d657f0e3d247de53336969d4b684c2aa00a12b537ba6525408adf154ee8a7c"
Oct 30 21:24:14.841614 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:14.841386    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f0fe13-sg5xm"
Oct 30 21:24:16.116922 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:16.116867    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-08df98-699dbdd999-t8wvt"
Oct 30 21:24:16.249062 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:16.249025    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aec20547-eca3-464b-9c2d-974ff1673459-kube-api-access-5h5nx\" (UniqueName: \"kubernetes.io/projected/aec20547-eca3-464b-9c2d-974ff1673459-kube-api-access-5h5nx\") pod \"aec20547-eca3-464b-9c2d-974ff1673459\" (UID: \"aec20547-eca3-464b-9c2d-974ff1673459\") "
Oct 30 21:24:16.249296 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:16.249278    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aec20547-eca3-464b-9c2d-974ff1673459-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aec20547-eca3-464b-9c2d-974ff1673459-plugins-config-volume\") pod \"aec20547-eca3-464b-9c2d-974ff1673459\" (UID: \"aec20547-eca3-464b-9c2d-974ff1673459\") "
Oct 30 21:24:16.249698 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:16.249666    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aec20547-eca3-464b-9c2d-974ff1673459-plugins-config-volume" pod "aec20547-eca3-464b-9c2d-974ff1673459" (UID: "aec20547-eca3-464b-9c2d-974ff1673459"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:24:16.251730 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:16.251688    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aec20547-eca3-464b-9c2d-974ff1673459-kube-api-access-5h5nx" pod "aec20547-eca3-464b-9c2d-974ff1673459" (UID: "aec20547-eca3-464b-9c2d-974ff1673459"). InnerVolumeSpecName "kube-api-access-5h5nx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:24:16.351338 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:16.351307    5713 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aec20547-eca3-464b-9c2d-974ff1673459-plugins-config-volume\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:16.351338 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:16.351336    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5h5nx\" (UniqueName: \"kubernetes.io/projected/aec20547-eca3-464b-9c2d-974ff1673459-kube-api-access-5h5nx\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:16.544081 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:16.544040    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1b889bd-gqxc5"]
Oct 30 21:24:16.547021 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:16.546991    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d1b889bd-gqxc5"]
Oct 30 21:24:16.856934 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:16.856841    5713 generic.go:356] "Generic (PLEG): container finished" podID="aec20547-eca3-464b-9c2d-974ff1673459" containerID="a175d0a592d30634c21dc1848654e188d7c1d47fdf5585e6b389fed68d368dc5" exitCode=0
Oct 30 21:24:16.856934 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:16.856901    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-08df98-699dbdd999-t8wvt"
Oct 30 21:24:16.857182 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:16.856931    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-08df98-699dbdd999-t8wvt" event={"ID":"aec20547-eca3-464b-9c2d-974ff1673459","Type":"ContainerDied","Data":"a175d0a592d30634c21dc1848654e188d7c1d47fdf5585e6b389fed68d368dc5"}
Oct 30 21:24:16.857182 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:16.856988    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-08df98-699dbdd999-t8wvt" event={"ID":"aec20547-eca3-464b-9c2d-974ff1673459","Type":"ContainerDied","Data":"61dc6907418f316f391f91df3206b6b365b852c83ba6da6b6835bdd99df23db7"}
Oct 30 21:24:16.857182 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:16.857013    5713 scope.go:122] "RemoveContainer" containerID="a175d0a592d30634c21dc1848654e188d7c1d47fdf5585e6b389fed68d368dc5"
Oct 30 21:24:16.868158 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:16.868137    5713 scope.go:122] "RemoveContainer" containerID="a175d0a592d30634c21dc1848654e188d7c1d47fdf5585e6b389fed68d368dc5"
Oct 30 21:24:16.868449 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:24:16.868426    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a175d0a592d30634c21dc1848654e188d7c1d47fdf5585e6b389fed68d368dc5\": container with ID starting with a175d0a592d30634c21dc1848654e188d7c1d47fdf5585e6b389fed68d368dc5 not found: ID does not exist" containerID="a175d0a592d30634c21dc1848654e188d7c1d47fdf5585e6b389fed68d368dc5"
Oct 30 21:24:16.868495 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:16.868457    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a175d0a592d30634c21dc1848654e188d7c1d47fdf5585e6b389fed68d368dc5"} err="failed to get container status \"a175d0a592d30634c21dc1848654e188d7c1d47fdf5585e6b389fed68d368dc5\": rpc error: code = NotFound desc = could not find container \"a175d0a592d30634c21dc1848654e188d7c1d47fdf5585e6b389fed68d368dc5\": container with ID starting with a175d0a592d30634c21dc1848654e188d7c1d47fdf5585e6b389fed68d368dc5 not found: ID does not exist"
Oct 30 21:24:17.022338 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:17.022303    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-08df98-699dbdd999-t8wvt"]
Oct 30 21:24:17.026067 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:17.026041    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-08df98-699dbdd999-t8wvt"]
Oct 30 21:24:17.064631 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:17.064603    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aec20547-eca3-464b-9c2d-974ff1673459" path="/var/lib/kubelet/pods/aec20547-eca3-464b-9c2d-974ff1673459/volumes"
Oct 30 21:24:17.065086 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:17.065071    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9eaeca5-7abf-4af7-90e3-b190c2db7fbe" path="/var/lib/kubelet/pods/e9eaeca5-7abf-4af7-90e3-b190c2db7fbe/volumes"
Oct 30 21:24:21.923129 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:21.923098    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-d1b889bd/e2e-lifecycle-model-d1b889bd-kserve-6bf5bc48d5-7cw4n"]
Oct 30 21:24:22.893607 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:22.893564    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-08df98-969r5"]
Oct 30 21:24:22.944194 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:22.944163    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-08df98-969r5"]
Oct 30 21:24:23.063206 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:23.063167    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e8c5c3f-1ca9-4434-a0cb-c45f0e249fbb" path="/var/lib/kubelet/pods/3e8c5c3f-1ca9-4434-a0cb-c45f0e249fbb/volumes"
Oct 30 21:24:24.459978 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:24:24.459935    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:24:29.780247 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:29.780197    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x"]
Oct 30 21:24:29.781332 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:29.781258    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x" podUID="fbb39bfd-2716-4ded-9e69-6131383afdb9" containerName="istio-proxy" containerID="cri-o://4f5c7a7b969f241216c7caa753d8eb44f3df9c8acfe8f66d904e8f018377425d" gracePeriod=30
Oct 30 21:24:30.531184 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:30.531146    5713 patch_prober.go:29] interesting pod/e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:24:30.531365 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:30.531203    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x" podUID="fbb39bfd-2716-4ded-9e69-6131383afdb9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:24:30.810493 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:30.810401    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-682999-gkt8n"]
Oct 30 21:24:30.811192 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:30.811170    5713 cpu_manager.go:414] "containerMap: removing container" podUID="b6d68dc1-ba46-490a-98ac-09208798fb71" containerName="revoke-keys"
Oct 30 21:24:30.811296 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:30.811194    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6d68dc1-ba46-490a-98ac-09208798fb71" containerName="revoke-keys"
Oct 30 21:24:30.811296 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:30.811234    5713 cpu_manager.go:414] "containerMap: removing container" podUID="aec20547-eca3-464b-9c2d-974ff1673459" containerName="payload-pre-processing"
Oct 30 21:24:30.811296 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:30.811243    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aec20547-eca3-464b-9c2d-974ff1673459" containerName="payload-pre-processing"
Oct 30 21:24:30.811461 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:30.811396    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6d68dc1-ba46-490a-98ac-09208798fb71" containerName="revoke-keys"
Oct 30 21:24:30.811461 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:30.811413    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="aec20547-eca3-464b-9c2d-974ff1673459" containerName="payload-pre-processing"
Oct 30 21:24:30.816383 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:30.816359    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-682999-gkt8n"
Oct 30 21:24:30.874161 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:30.874129    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-682999-gkt8n"]
Oct 30 21:24:30.919623 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:30.919571    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:24:30.919623 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:30.919572    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:24:30.919865 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:30.919639    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:24:30.919865 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:30.919641    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:24:30.933188 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:30.933158    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d0b55cf-0697-4ecb-aaa2-479b968fbd2f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-682999-gkt8n\" (UID: \"5d0b55cf-0697-4ecb-aaa2-479b968fbd2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-682999-gkt8n"
Oct 30 21:24:30.933308 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:30.933196    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-658rm\" (UniqueName: \"kubernetes.io/projected/5d0b55cf-0697-4ecb-aaa2-479b968fbd2f-kube-api-access-658rm\") pod \"maas-api-revoke-keys-e2e-worker-w3-682999-gkt8n\" (UID: \"5d0b55cf-0697-4ecb-aaa2-479b968fbd2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-682999-gkt8n"
Oct 30 21:24:30.933308 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:30.933237    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d0b55cf-0697-4ecb-aaa2-479b968fbd2f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-682999-gkt8n\" (UID: \"5d0b55cf-0697-4ecb-aaa2-479b968fbd2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-682999-gkt8n"
Oct 30 21:24:31.034170 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:31.034139    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d0b55cf-0697-4ecb-aaa2-479b968fbd2f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-682999-gkt8n\" (UID: \"5d0b55cf-0697-4ecb-aaa2-479b968fbd2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-682999-gkt8n"
Oct 30 21:24:31.034170 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:31.034175    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-658rm\" (UniqueName: \"kubernetes.io/projected/5d0b55cf-0697-4ecb-aaa2-479b968fbd2f-kube-api-access-658rm\") pod \"maas-api-revoke-keys-e2e-worker-w3-682999-gkt8n\" (UID: \"5d0b55cf-0697-4ecb-aaa2-479b968fbd2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-682999-gkt8n"
Oct 30 21:24:31.034505 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:31.034213    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d0b55cf-0697-4ecb-aaa2-479b968fbd2f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-682999-gkt8n\" (UID: \"5d0b55cf-0697-4ecb-aaa2-479b968fbd2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-682999-gkt8n"
Oct 30 21:24:31.034884 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:31.034857    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d0b55cf-0697-4ecb-aaa2-479b968fbd2f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-682999-gkt8n\" (UID: \"5d0b55cf-0697-4ecb-aaa2-479b968fbd2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-682999-gkt8n"
Oct 30 21:24:31.036661 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:31.036642    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d0b55cf-0697-4ecb-aaa2-479b968fbd2f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-682999-gkt8n\" (UID: \"5d0b55cf-0697-4ecb-aaa2-479b968fbd2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-682999-gkt8n"
Oct 30 21:24:31.095561 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:31.095492    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-658rm\" (UniqueName: \"kubernetes.io/projected/5d0b55cf-0697-4ecb-aaa2-479b968fbd2f-kube-api-access-658rm\") pod \"maas-api-revoke-keys-e2e-worker-w3-682999-gkt8n\" (UID: \"5d0b55cf-0697-4ecb-aaa2-479b968fbd2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-682999-gkt8n"
Oct 30 21:24:31.129540 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:31.129504    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-682999-gkt8n"
Oct 30 21:24:31.505613 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:31.505581    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-682999-gkt8n"]
Oct 30 21:24:31.507068 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:24:31.507038    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d0b55cf_0697_4ecb_aaa2_479b968fbd2f.slice/crio-181e57ba1dbadda10c31f9ec7b543018b46ec783b11c221287f373cc45d5f858 WatchSource:0}: Error finding container 181e57ba1dbadda10c31f9ec7b543018b46ec783b11c221287f373cc45d5f858: Status 404 returned error can't find the container with id 181e57ba1dbadda10c31f9ec7b543018b46ec783b11c221287f373cc45d5f858
Oct 30 21:24:31.971352 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:31.971312    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-682999-gkt8n" event={"ID":"5d0b55cf-0697-4ecb-aaa2-479b968fbd2f","Type":"ContainerStarted","Data":"60a574a7ffcf27b3657bb97d5bc31db90f72e7a4eeb445944e703423d6f600b2"}
Oct 30 21:24:31.971738 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:31.971378    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-682999-gkt8n" event={"ID":"5d0b55cf-0697-4ecb-aaa2-479b968fbd2f","Type":"ContainerStarted","Data":"181e57ba1dbadda10c31f9ec7b543018b46ec783b11c221287f373cc45d5f858"}
Oct 30 21:24:32.086779 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:32.086722    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-682999-gkt8n" podStartSLOduration=2.086703222 podStartE2EDuration="2.086703222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:32.083210426 +0000 UTC m=+2755.147986830" watchObservedRunningTime="2026-10-30 21:24:32.086703222 +0000 UTC m=+2755.151479621"
Oct 30 21:24:32.981423 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:32.981379    5713 generic.go:356] "Generic (PLEG): container finished" podID="5d0b55cf-0697-4ecb-aaa2-479b968fbd2f" containerID="60a574a7ffcf27b3657bb97d5bc31db90f72e7a4eeb445944e703423d6f600b2" exitCode=0
Oct 30 21:24:32.981897 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:32.981454    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-682999-gkt8n" event={"ID":"5d0b55cf-0697-4ecb-aaa2-479b968fbd2f","Type":"ContainerDied","Data":"60a574a7ffcf27b3657bb97d5bc31db90f72e7a4eeb445944e703423d6f600b2"}
Oct 30 21:24:34.137030 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:34.137001    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-682999-gkt8n"
Oct 30 21:24:34.270103 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:34.270022    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d0b55cf-0697-4ecb-aaa2-479b968fbd2f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d0b55cf-0697-4ecb-aaa2-479b968fbd2f-maas-api-xks-ca\") pod \"5d0b55cf-0697-4ecb-aaa2-479b968fbd2f\" (UID: \"5d0b55cf-0697-4ecb-aaa2-479b968fbd2f\") "
Oct 30 21:24:34.270424 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:34.270401    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d0b55cf-0697-4ecb-aaa2-479b968fbd2f-kube-api-access-658rm\" (UniqueName: \"kubernetes.io/projected/5d0b55cf-0697-4ecb-aaa2-479b968fbd2f-kube-api-access-658rm\") pod \"5d0b55cf-0697-4ecb-aaa2-479b968fbd2f\" (UID: \"5d0b55cf-0697-4ecb-aaa2-479b968fbd2f\") "
Oct 30 21:24:34.270589 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:34.270546    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d0b55cf-0697-4ecb-aaa2-479b968fbd2f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d0b55cf-0697-4ecb-aaa2-479b968fbd2f-maas-api-openshift-ca\") pod \"5d0b55cf-0697-4ecb-aaa2-479b968fbd2f\" (UID: \"5d0b55cf-0697-4ecb-aaa2-479b968fbd2f\") "
Oct 30 21:24:34.271460 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:34.271432    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d0b55cf-0697-4ecb-aaa2-479b968fbd2f-maas-api-openshift-ca" pod "5d0b55cf-0697-4ecb-aaa2-479b968fbd2f" (UID: "5d0b55cf-0697-4ecb-aaa2-479b968fbd2f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:24:34.277233 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:34.277197    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d0b55cf-0697-4ecb-aaa2-479b968fbd2f-maas-api-xks-ca" pod "5d0b55cf-0697-4ecb-aaa2-479b968fbd2f" (UID: "5d0b55cf-0697-4ecb-aaa2-479b968fbd2f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:24:34.290431 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:34.290390    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d0b55cf-0697-4ecb-aaa2-479b968fbd2f-kube-api-access-658rm" pod "5d0b55cf-0697-4ecb-aaa2-479b968fbd2f" (UID: "5d0b55cf-0697-4ecb-aaa2-479b968fbd2f"). InnerVolumeSpecName "kube-api-access-658rm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:24:34.372848 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:34.372780    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d0b55cf-0697-4ecb-aaa2-479b968fbd2f-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:34.372848 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:34.372846    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-658rm\" (UniqueName: \"kubernetes.io/projected/5d0b55cf-0697-4ecb-aaa2-479b968fbd2f-kube-api-access-658rm\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:34.372848 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:34.372858    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d0b55cf-0697-4ecb-aaa2-479b968fbd2f-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:34.416623 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:34.416586    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f0fe13-sg5xm"]
Oct 30 21:24:34.419284 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:34.419261    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f0fe13-sg5xm"]
Oct 30 21:24:35.001767 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:35.001722    5713 generic.go:356] "Generic (PLEG): container finished" podID="fbb39bfd-2716-4ded-9e69-6131383afdb9" containerID="4f5c7a7b969f241216c7caa753d8eb44f3df9c8acfe8f66d904e8f018377425d" exitCode=0
Oct 30 21:24:35.001981 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:35.001779    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x" event={"ID":"fbb39bfd-2716-4ded-9e69-6131383afdb9","Type":"ContainerDied","Data":"4f5c7a7b969f241216c7caa753d8eb44f3df9c8acfe8f66d904e8f018377425d"}
Oct 30 21:24:35.001981 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:35.001865    5713 scope.go:122] "RemoveContainer" containerID="ae817ded885d8d06555408698bfad5fe2bb80a8ef99e711f5330eda0f7763afe"
Oct 30 21:24:35.003948 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:35.003921    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-682999-gkt8n" event={"ID":"5d0b55cf-0697-4ecb-aaa2-479b968fbd2f","Type":"ContainerDied","Data":"181e57ba1dbadda10c31f9ec7b543018b46ec783b11c221287f373cc45d5f858"}
Oct 30 21:24:35.004108 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:35.003946    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-682999-gkt8n"
Oct 30 21:24:35.004108 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:35.004049    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="181e57ba1dbadda10c31f9ec7b543018b46ec783b11c221287f373cc45d5f858"
Oct 30 21:24:35.039249 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:35.039225    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x"
Oct 30 21:24:35.063731 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:35.063658    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6d68dc1-ba46-490a-98ac-09208798fb71" path="/var/lib/kubelet/pods/b6d68dc1-ba46-490a-98ac-09208798fb71/volumes"
Oct 30 21:24:35.080996 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:35.080966    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbb39bfd-2716-4ded-9e69-6131383afdb9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbb39bfd-2716-4ded-9e69-6131383afdb9-istio-data\") pod \"fbb39bfd-2716-4ded-9e69-6131383afdb9\" (UID: \"fbb39bfd-2716-4ded-9e69-6131383afdb9\") "
Oct 30 21:24:35.081187 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:35.081047    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbb39bfd-2716-4ded-9e69-6131383afdb9-istio-token\" (UniqueName: \"kubernetes.io/projected/fbb39bfd-2716-4ded-9e69-6131383afdb9-istio-token\") pod \"fbb39bfd-2716-4ded-9e69-6131383afdb9\" (UID: \"fbb39bfd-2716-4ded-9e69-6131383afdb9\") "
Oct 30 21:24:35.081187 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:35.081080    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbb39bfd-2716-4ded-9e69-6131383afdb9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbb39bfd-2716-4ded-9e69-6131383afdb9-workload-certs\") pod \"fbb39bfd-2716-4ded-9e69-6131383afdb9\" (UID: \"fbb39bfd-2716-4ded-9e69-6131383afdb9\") "
Oct 30 21:24:35.081187 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:35.081129    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbb39bfd-2716-4ded-9e69-6131383afdb9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb39bfd-2716-4ded-9e69-6131383afdb9-workload-socket\") pod \"fbb39bfd-2716-4ded-9e69-6131383afdb9\" (UID: \"fbb39bfd-2716-4ded-9e69-6131383afdb9\") "
Oct 30 21:24:35.081187 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:35.081164    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbb39bfd-2716-4ded-9e69-6131383afdb9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb39bfd-2716-4ded-9e69-6131383afdb9-credential-socket\") pod \"fbb39bfd-2716-4ded-9e69-6131383afdb9\" (UID: \"fbb39bfd-2716-4ded-9e69-6131383afdb9\") "
Oct 30 21:24:35.081408 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:35.081242    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbb39bfd-2716-4ded-9e69-6131383afdb9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbb39bfd-2716-4ded-9e69-6131383afdb9-istio-envoy\") pod \"fbb39bfd-2716-4ded-9e69-6131383afdb9\" (UID: \"fbb39bfd-2716-4ded-9e69-6131383afdb9\") "
Oct 30 21:24:35.081408 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:35.081274    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbb39bfd-2716-4ded-9e69-6131383afdb9-istio-data" pod "fbb39bfd-2716-4ded-9e69-6131383afdb9" (UID: "fbb39bfd-2716-4ded-9e69-6131383afdb9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:24:35.081408 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:35.081351    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbb39bfd-2716-4ded-9e69-6131383afdb9-kube-api-access-n5n78\" (UniqueName: \"kubernetes.io/projected/fbb39bfd-2716-4ded-9e69-6131383afdb9-kube-api-access-n5n78\") pod \"fbb39bfd-2716-4ded-9e69-6131383afdb9\" (UID: \"fbb39bfd-2716-4ded-9e69-6131383afdb9\") "
Oct 30 21:24:35.081408 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:35.081400    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fbb39bfd-2716-4ded-9e69-6131383afdb9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbb39bfd-2716-4ded-9e69-6131383afdb9-istio-podinfo\") pod \"fbb39bfd-2716-4ded-9e69-6131383afdb9\" (UID: \"fbb39bfd-2716-4ded-9e69-6131383afdb9\") "
Oct 30 21:24:35.081700 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:35.081669    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbb39bfd-2716-4ded-9e69-6131383afdb9-workload-certs" pod "fbb39bfd-2716-4ded-9e69-6131383afdb9" (UID: "fbb39bfd-2716-4ded-9e69-6131383afdb9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:24:35.081882 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:35.081708    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbb39bfd-2716-4ded-9e69-6131383afdb9-workload-socket" pod "fbb39bfd-2716-4ded-9e69-6131383afdb9" (UID: "fbb39bfd-2716-4ded-9e69-6131383afdb9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:24:35.081882 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:35.081757    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbb39bfd-2716-4ded-9e69-6131383afdb9-credential-socket" pod "fbb39bfd-2716-4ded-9e69-6131383afdb9" (UID: "fbb39bfd-2716-4ded-9e69-6131383afdb9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:24:35.082015 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:35.081876    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbb39bfd-2716-4ded-9e69-6131383afdb9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbb39bfd-2716-4ded-9e69-6131383afdb9-istiod-ca-cert\") pod \"fbb39bfd-2716-4ded-9e69-6131383afdb9\" (UID: \"fbb39bfd-2716-4ded-9e69-6131383afdb9\") "
Oct 30 21:24:35.082191 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:35.082165    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbb39bfd-2716-4ded-9e69-6131383afdb9-istiod-ca-cert" pod "fbb39bfd-2716-4ded-9e69-6131383afdb9" (UID: "fbb39bfd-2716-4ded-9e69-6131383afdb9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:24:35.082391 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:35.082374    5713 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbb39bfd-2716-4ded-9e69-6131383afdb9-istio-data\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:35.082478 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:35.082398    5713 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbb39bfd-2716-4ded-9e69-6131383afdb9-workload-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:35.082478 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:35.082413    5713 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb39bfd-2716-4ded-9e69-6131383afdb9-workload-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:35.082478 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:35.082426    5713 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb39bfd-2716-4ded-9e69-6131383afdb9-credential-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:35.082478 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:35.082441    5713 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbb39bfd-2716-4ded-9e69-6131383afdb9-istiod-ca-cert\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:35.084010 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:35.083986    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbb39bfd-2716-4ded-9e69-6131383afdb9-kube-api-access-n5n78" pod "fbb39bfd-2716-4ded-9e69-6131383afdb9" (UID: "fbb39bfd-2716-4ded-9e69-6131383afdb9"). InnerVolumeSpecName "kube-api-access-n5n78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:24:35.084198 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:35.084181    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbb39bfd-2716-4ded-9e69-6131383afdb9-istio-token" pod "fbb39bfd-2716-4ded-9e69-6131383afdb9" (UID: "fbb39bfd-2716-4ded-9e69-6131383afdb9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:24:35.084302 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:35.084280    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbb39bfd-2716-4ded-9e69-6131383afdb9-istio-envoy" pod "fbb39bfd-2716-4ded-9e69-6131383afdb9" (UID: "fbb39bfd-2716-4ded-9e69-6131383afdb9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:24:35.084356 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:35.084295    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fbb39bfd-2716-4ded-9e69-6131383afdb9-istio-podinfo" pod "fbb39bfd-2716-4ded-9e69-6131383afdb9" (UID: "fbb39bfd-2716-4ded-9e69-6131383afdb9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:24:35.183992 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:35.183954    5713 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbb39bfd-2716-4ded-9e69-6131383afdb9-istio-token\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:35.183992 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:35.183991    5713 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbb39bfd-2716-4ded-9e69-6131383afdb9-istio-envoy\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:35.183992 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:35.184006    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5n78\" (UniqueName: \"kubernetes.io/projected/fbb39bfd-2716-4ded-9e69-6131383afdb9-kube-api-access-n5n78\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:35.183992 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:35.184021    5713 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbb39bfd-2716-4ded-9e69-6131383afdb9-istio-podinfo\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:35.221923 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:35.221878    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-682999/distinct-2-w3-682999-kserve-7bc687885b-gq5x7"]
Oct 30 21:24:35.226325 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:35.226297    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-682999/distinct-w3-682999-kserve-74d49d4955-nndlc"]
Oct 30 21:24:35.237839 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:35.237807    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-682999/premium-w3-682999-kserve-5f47cf9b8d-xqnnr"]
Oct 30 21:24:35.248622 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:35.248590    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-682999/simulator-w3-682999-kserve-57cfcb6f8f-6mgrg"]
Oct 30 21:24:36.013934 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:36.013900    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x"
Oct 30 21:24:36.013934 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:36.013915    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x" event={"ID":"fbb39bfd-2716-4ded-9e69-6131383afdb9","Type":"ContainerDied","Data":"725f37863f4271aceddeaa7bad0a2cb14c194845c0b17376144cea81d200874f"}
Oct 30 21:24:36.014197 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:36.013978    5713 scope.go:122] "RemoveContainer" containerID="4f5c7a7b969f241216c7caa753d8eb44f3df9c8acfe8f66d904e8f018377425d"
Oct 30 21:24:36.178624 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:36.178591    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x"]
Oct 30 21:24:36.185407 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:36.185380    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-08df98-openshift-default-cdd775fcf-pql4x"]
Oct 30 21:24:36.944397 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:36.944361    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-d1b889bd/e2e-lifecycle-model-d1b889bd-kserve-6bf5bc48d5-7cw4n" podUID="2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4" containerName="main" containerID="cri-o://4f9633f73e188265e5106159856da1af2f1af108745645bcf89f191e54061ca0" gracePeriod=45
Oct 30 21:24:37.074225 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:37.074180    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbb39bfd-2716-4ded-9e69-6131383afdb9" path="/var/lib/kubelet/pods/fbb39bfd-2716-4ded-9e69-6131383afdb9/volumes"
Oct 30 21:24:37.213555 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:37.213529    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d1b889bd/e2e-lifecycle-model-d1b889bd-kserve-6bf5bc48d5-7cw4n"
Oct 30 21:24:37.309110 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:37.309078    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4-tmp-dir\") pod \"2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4\" (UID: \"2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4\") "
Oct 30 21:24:37.309314 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:37.309131    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4-home\" (UniqueName: \"kubernetes.io/empty-dir/2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4-home\") pod \"2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4\" (UID: \"2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4\") "
Oct 30 21:24:37.309314 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:37.309188    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4-dshm\") pod \"2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4\" (UID: \"2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4\") "
Oct 30 21:24:37.309314 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:37.309255    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4-kube-api-access-49l52\" (UniqueName: \"kubernetes.io/projected/2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4-kube-api-access-49l52\") pod \"2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4\" (UID: \"2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4\") "
Oct 30 21:24:37.309510 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:37.309357    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4-tls-certs\" (UniqueName: \"kubernetes.io/secret/2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4-tls-certs\") pod \"2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4\" (UID: \"2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4\") "
Oct 30 21:24:37.309510 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:37.309391    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4-tmp-dir" pod "2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4" (UID: "2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:24:37.309510 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:37.309442    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4-model-cache\") pod \"2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4\" (UID: \"2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4\") "
Oct 30 21:24:37.309510 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:37.309465    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4-home" pod "2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4" (UID: "2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:24:37.309698 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:37.309679    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4-model-cache" pod "2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4" (UID: "2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:24:37.310006 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:37.309959    5713 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4-model-cache\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:37.310006 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:37.309979    5713 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4-tmp-dir\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:37.310006 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:37.309989    5713 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4-home\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:37.311742 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:37.311717    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4-kube-api-access-49l52" pod "2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4" (UID: "2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4"). InnerVolumeSpecName "kube-api-access-49l52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:24:37.311883 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:37.311745    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4-dshm" pod "2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4" (UID: "2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:24:37.311883 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:37.311774    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4-tls-certs" pod "2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4" (UID: "2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:24:37.411285 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:37.411246    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49l52\" (UniqueName: \"kubernetes.io/projected/2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4-kube-api-access-49l52\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:37.411285 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:37.411278    5713 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4-tls-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:37.411285 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:37.411288    5713 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4-dshm\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:38.031607 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:38.031571    5713 generic.go:356] "Generic (PLEG): container finished" podID="2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4" containerID="4f9633f73e188265e5106159856da1af2f1af108745645bcf89f191e54061ca0" exitCode=0
Oct 30 21:24:38.031824 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:38.031620    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d1b889bd/e2e-lifecycle-model-d1b889bd-kserve-6bf5bc48d5-7cw4n"
Oct 30 21:24:38.031824 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:38.031648    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d1b889bd/e2e-lifecycle-model-d1b889bd-kserve-6bf5bc48d5-7cw4n" event={"ID":"2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4","Type":"ContainerDied","Data":"4f9633f73e188265e5106159856da1af2f1af108745645bcf89f191e54061ca0"}
Oct 30 21:24:38.031824 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:38.031689    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d1b889bd/e2e-lifecycle-model-d1b889bd-kserve-6bf5bc48d5-7cw4n" event={"ID":"2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4","Type":"ContainerDied","Data":"fad24492435369d437e948c391dc52aa2ce1429e3ac61a2871df4ac16b762f47"}
Oct 30 21:24:38.031824 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:38.031707    5713 scope.go:122] "RemoveContainer" containerID="4f9633f73e188265e5106159856da1af2f1af108745645bcf89f191e54061ca0"
Oct 30 21:24:38.043047 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:38.043030    5713 scope.go:122] "RemoveContainer" containerID="4f9633f73e188265e5106159856da1af2f1af108745645bcf89f191e54061ca0"
Oct 30 21:24:38.043353 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:24:38.043334    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f9633f73e188265e5106159856da1af2f1af108745645bcf89f191e54061ca0\": container with ID starting with 4f9633f73e188265e5106159856da1af2f1af108745645bcf89f191e54061ca0 not found: ID does not exist" containerID="4f9633f73e188265e5106159856da1af2f1af108745645bcf89f191e54061ca0"
Oct 30 21:24:38.043444 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:38.043362    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f9633f73e188265e5106159856da1af2f1af108745645bcf89f191e54061ca0"} err="failed to get container status \"4f9633f73e188265e5106159856da1af2f1af108745645bcf89f191e54061ca0\": rpc error: code = NotFound desc = could not find container \"4f9633f73e188265e5106159856da1af2f1af108745645bcf89f191e54061ca0\": container with ID starting with 4f9633f73e188265e5106159856da1af2f1af108745645bcf89f191e54061ca0 not found: ID does not exist"
Oct 30 21:24:38.206738 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:38.206704    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-d1b889bd/e2e-lifecycle-model-d1b889bd-kserve-6bf5bc48d5-7cw4n"]
Oct 30 21:24:38.256230 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:38.256196    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-d1b889bd/e2e-lifecycle-model-d1b889bd-kserve-6bf5bc48d5-7cw4n"]
Oct 30 21:24:39.064018 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:39.063986    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4" path="/var/lib/kubelet/pods/2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4/volumes"
Oct 30 21:24:39.633995 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:39.633961    5713 scope.go:122] "RemoveContainer" containerID="2b3a49e1fcd7eaed2980b586981883d2afb42ff68aaff949048e56b567aa6580"
Oct 30 21:24:39.644549 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:39.644527    5713 scope.go:122] "RemoveContainer" containerID="f7a39a388676aa241a71f37c4b8390dd39e2568691d2ae4575173bc1a1b5c885"
Oct 30 21:24:39.654090 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:39.654068    5713 scope.go:122] "RemoveContainer" containerID="b2909abd83ca144cbdc9fac802b0462261d8d2ca99f0355eff29fe1877da048a"
Oct 30 21:24:39.663363 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:39.663339    5713 scope.go:122] "RemoveContainer" containerID="f14de084b4e59c5b10af5ded67c3cbba9c64401cc2e0d577edf5beec2dcccb49"
Oct 30 21:24:39.672286 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:39.672266    5713 scope.go:122] "RemoveContainer" containerID="346d32966dbefe5e0cbc3530b88e8c0c96ac817ceba60b93c76d2cba1bbace0b"
Oct 30 21:24:41.015205 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:41.015163    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg"]
Oct 30 21:24:41.015719 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:41.015444    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg" podUID="d0dff592-d5a1-4832-8104-46cc5fd973c5" containerName="istio-proxy" containerID="cri-o://eb0fe2865c319539aa0fcaba4ec26a0846a4830c675c707d4ee10ebde1983082" gracePeriod=30
Oct 30 21:24:41.851244 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:41.851208    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-639b07-8zrvs"]
Oct 30 21:24:41.851743 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:41.851729    5713 cpu_manager.go:414] "containerMap: removing container" podUID="2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4" containerName="main"
Oct 30 21:24:41.851831 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:41.851744    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4" containerName="main"
Oct 30 21:24:41.851831 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:41.851763    5713 cpu_manager.go:414] "containerMap: removing container" podUID="fbb39bfd-2716-4ded-9e69-6131383afdb9" containerName="istio-proxy"
Oct 30 21:24:41.851831 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:41.851768    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbb39bfd-2716-4ded-9e69-6131383afdb9" containerName="istio-proxy"
Oct 30 21:24:41.851831 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:41.851783    5713 cpu_manager.go:414] "containerMap: removing container" podUID="fbb39bfd-2716-4ded-9e69-6131383afdb9" containerName="istio-proxy"
Oct 30 21:24:41.851831 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:41.851806    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbb39bfd-2716-4ded-9e69-6131383afdb9" containerName="istio-proxy"
Oct 30 21:24:41.851831 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:41.851825    5713 cpu_manager.go:414] "containerMap: removing container" podUID="5d0b55cf-0697-4ecb-aaa2-479b968fbd2f" containerName="revoke-keys"
Oct 30 21:24:41.851831 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:41.851832    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d0b55cf-0697-4ecb-aaa2-479b968fbd2f" containerName="revoke-keys"
Oct 30 21:24:41.852054 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:41.851919    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbb39bfd-2716-4ded-9e69-6131383afdb9" containerName="istio-proxy"
Oct 30 21:24:41.852054 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:41.851927    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d0b55cf-0697-4ecb-aaa2-479b968fbd2f" containerName="revoke-keys"
Oct 30 21:24:41.852054 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:41.851939    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ac9ab3e-0d2f-469b-8538-3ec0ace4ecc4" containerName="main"
Oct 30 21:24:41.857367 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:41.857344    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-639b07-8zrvs"
Oct 30 21:24:41.910881 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:41.910847    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-639b07-8zrvs"]
Oct 30 21:24:41.957081 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:41.957047    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:24:41.957081 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:41.957069    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:24:41.957347 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:41.957111    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:24:41.958727 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:41.958708    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:24:41.963354 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:41.963329    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b559e33d-9df3-49f7-a9f3-b97c1c79c0e9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-639b07-8zrvs\" (UID: \"b559e33d-9df3-49f7-a9f3-b97c1c79c0e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-639b07-8zrvs"
Oct 30 21:24:41.963463 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:41.963410    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwm8m\" (UniqueName: \"kubernetes.io/projected/b559e33d-9df3-49f7-a9f3-b97c1c79c0e9-kube-api-access-nwm8m\") pod \"maas-api-revoke-keys-e2e-inf-b-639b07-8zrvs\" (UID: \"b559e33d-9df3-49f7-a9f3-b97c1c79c0e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-639b07-8zrvs"
Oct 30 21:24:41.963518 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:41.963459    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b559e33d-9df3-49f7-a9f3-b97c1c79c0e9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-639b07-8zrvs\" (UID: \"b559e33d-9df3-49f7-a9f3-b97c1c79c0e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-639b07-8zrvs"
Oct 30 21:24:42.064969 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:42.064936    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b559e33d-9df3-49f7-a9f3-b97c1c79c0e9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-639b07-8zrvs\" (UID: \"b559e33d-9df3-49f7-a9f3-b97c1c79c0e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-639b07-8zrvs"
Oct 30 21:24:42.065494 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:42.065071    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b559e33d-9df3-49f7-a9f3-b97c1c79c0e9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-639b07-8zrvs\" (UID: \"b559e33d-9df3-49f7-a9f3-b97c1c79c0e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-639b07-8zrvs"
Oct 30 21:24:42.065494 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:42.065114    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwm8m\" (UniqueName: \"kubernetes.io/projected/b559e33d-9df3-49f7-a9f3-b97c1c79c0e9-kube-api-access-nwm8m\") pod \"maas-api-revoke-keys-e2e-inf-b-639b07-8zrvs\" (UID: \"b559e33d-9df3-49f7-a9f3-b97c1c79c0e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-639b07-8zrvs"
Oct 30 21:24:42.065736 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:42.065716    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b559e33d-9df3-49f7-a9f3-b97c1c79c0e9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-639b07-8zrvs\" (UID: \"b559e33d-9df3-49f7-a9f3-b97c1c79c0e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-639b07-8zrvs"
Oct 30 21:24:42.067688 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:42.067662    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b559e33d-9df3-49f7-a9f3-b97c1c79c0e9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-639b07-8zrvs\" (UID: \"b559e33d-9df3-49f7-a9f3-b97c1c79c0e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-639b07-8zrvs"
Oct 30 21:24:42.122377 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:42.122306    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwm8m\" (UniqueName: \"kubernetes.io/projected/b559e33d-9df3-49f7-a9f3-b97c1c79c0e9-kube-api-access-nwm8m\") pod \"maas-api-revoke-keys-e2e-inf-b-639b07-8zrvs\" (UID: \"b559e33d-9df3-49f7-a9f3-b97c1c79c0e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-639b07-8zrvs"
Oct 30 21:24:42.170478 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:42.170445    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-639b07-8zrvs"
Oct 30 21:24:42.549370 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:42.549341    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-639b07-8zrvs"]
Oct 30 21:24:42.550893 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:24:42.550855    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb559e33d_9df3_49f7_a9f3_b97c1c79c0e9.slice/crio-f66827d8361e579709bfb35fef4c49e314be76356961075557adae9c553b460b WatchSource:0}: Error finding container f66827d8361e579709bfb35fef4c49e314be76356961075557adae9c553b460b: Status 404 returned error can't find the container with id f66827d8361e579709bfb35fef4c49e314be76356961075557adae9c553b460b
Oct 30 21:24:43.068077 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:43.068035    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-639b07-8zrvs" event={"ID":"b559e33d-9df3-49f7-a9f3-b97c1c79c0e9","Type":"ContainerStarted","Data":"4f8466573277a3a7fdb4fd53190d81fceda4627cfbe907c798bbae1b10e8f37d"}
Oct 30 21:24:43.068586 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:43.068098    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-639b07-8zrvs" event={"ID":"b559e33d-9df3-49f7-a9f3-b97c1c79c0e9","Type":"ContainerStarted","Data":"f66827d8361e579709bfb35fef4c49e314be76356961075557adae9c553b460b"}
Oct 30 21:24:43.184484 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:43.184438    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-639b07-8zrvs" podStartSLOduration=2.184424793 podStartE2EDuration="2.184424793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:43.181038305 +0000 UTC m=+2766.245814704" watchObservedRunningTime="2026-10-30 21:24:43.184424793 +0000 UTC m=+2766.249201219"
Oct 30 21:24:43.668004 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:43.667974    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-682999-5489c955b7-dzmt9"]
Oct 30 21:24:44.077592 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:44.077556    5713 generic.go:356] "Generic (PLEG): container finished" podID="b559e33d-9df3-49f7-a9f3-b97c1c79c0e9" containerID="4f8466573277a3a7fdb4fd53190d81fceda4627cfbe907c798bbae1b10e8f37d" exitCode=0
Oct 30 21:24:44.077989 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:44.077638    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-639b07-8zrvs" event={"ID":"b559e33d-9df3-49f7-a9f3-b97c1c79c0e9","Type":"ContainerDied","Data":"4f8466573277a3a7fdb4fd53190d81fceda4627cfbe907c798bbae1b10e8f37d"}
Oct 30 21:24:45.240197 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:45.240170    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-639b07-8zrvs"
Oct 30 21:24:45.304865 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:45.304821    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b559e33d-9df3-49f7-a9f3-b97c1c79c0e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b559e33d-9df3-49f7-a9f3-b97c1c79c0e9-maas-api-xks-ca\") pod \"b559e33d-9df3-49f7-a9f3-b97c1c79c0e9\" (UID: \"b559e33d-9df3-49f7-a9f3-b97c1c79c0e9\") "
Oct 30 21:24:45.305035 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:45.304951    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b559e33d-9df3-49f7-a9f3-b97c1c79c0e9-kube-api-access-nwm8m\" (UniqueName: \"kubernetes.io/projected/b559e33d-9df3-49f7-a9f3-b97c1c79c0e9-kube-api-access-nwm8m\") pod \"b559e33d-9df3-49f7-a9f3-b97c1c79c0e9\" (UID: \"b559e33d-9df3-49f7-a9f3-b97c1c79c0e9\") "
Oct 30 21:24:45.305035 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:45.304998    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b559e33d-9df3-49f7-a9f3-b97c1c79c0e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b559e33d-9df3-49f7-a9f3-b97c1c79c0e9-maas-api-openshift-ca\") pod \"b559e33d-9df3-49f7-a9f3-b97c1c79c0e9\" (UID: \"b559e33d-9df3-49f7-a9f3-b97c1c79c0e9\") "
Oct 30 21:24:45.305419 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:45.305389    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b559e33d-9df3-49f7-a9f3-b97c1c79c0e9-maas-api-openshift-ca" pod "b559e33d-9df3-49f7-a9f3-b97c1c79c0e9" (UID: "b559e33d-9df3-49f7-a9f3-b97c1c79c0e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:24:45.307064 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:45.307038    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b559e33d-9df3-49f7-a9f3-b97c1c79c0e9-maas-api-xks-ca" pod "b559e33d-9df3-49f7-a9f3-b97c1c79c0e9" (UID: "b559e33d-9df3-49f7-a9f3-b97c1c79c0e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:24:45.307222 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:45.307202    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b559e33d-9df3-49f7-a9f3-b97c1c79c0e9-kube-api-access-nwm8m" pod "b559e33d-9df3-49f7-a9f3-b97c1c79c0e9" (UID: "b559e33d-9df3-49f7-a9f3-b97c1c79c0e9"). InnerVolumeSpecName "kube-api-access-nwm8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:24:45.406850 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:45.406739    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwm8m\" (UniqueName: \"kubernetes.io/projected/b559e33d-9df3-49f7-a9f3-b97c1c79c0e9-kube-api-access-nwm8m\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:45.406850 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:45.406778    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b559e33d-9df3-49f7-a9f3-b97c1c79c0e9-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:45.406850 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:45.406805    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b559e33d-9df3-49f7-a9f3-b97c1c79c0e9-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:46.092180 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:46.092149    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-639b07-8zrvs"
Oct 30 21:24:46.092180 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:46.092164    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-639b07-8zrvs" event={"ID":"b559e33d-9df3-49f7-a9f3-b97c1c79c0e9","Type":"ContainerDied","Data":"f66827d8361e579709bfb35fef4c49e314be76356961075557adae9c553b460b"}
Oct 30 21:24:46.092403 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:46.092211    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f66827d8361e579709bfb35fef4c49e314be76356961075557adae9c553b460b"
Oct 30 21:24:46.301365 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:46.301308    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg"
Oct 30 21:24:46.417221 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:46.417187    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0dff592-d5a1-4832-8104-46cc5fd973c5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0dff592-d5a1-4832-8104-46cc5fd973c5-workload-socket\") pod \"d0dff592-d5a1-4832-8104-46cc5fd973c5\" (UID: \"d0dff592-d5a1-4832-8104-46cc5fd973c5\") "
Oct 30 21:24:46.417736 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:46.417235    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d0dff592-d5a1-4832-8104-46cc5fd973c5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0dff592-d5a1-4832-8104-46cc5fd973c5-istio-podinfo\") pod \"d0dff592-d5a1-4832-8104-46cc5fd973c5\" (UID: \"d0dff592-d5a1-4832-8104-46cc5fd973c5\") "
Oct 30 21:24:46.417736 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:46.417404    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0dff592-d5a1-4832-8104-46cc5fd973c5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0dff592-d5a1-4832-8104-46cc5fd973c5-istio-envoy\") pod \"d0dff592-d5a1-4832-8104-46cc5fd973c5\" (UID: \"d0dff592-d5a1-4832-8104-46cc5fd973c5\") "
Oct 30 21:24:46.417736 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:46.417596    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0dff592-d5a1-4832-8104-46cc5fd973c5-istio-token\" (UniqueName: \"kubernetes.io/projected/d0dff592-d5a1-4832-8104-46cc5fd973c5-istio-token\") pod \"d0dff592-d5a1-4832-8104-46cc5fd973c5\" (UID: \"d0dff592-d5a1-4832-8104-46cc5fd973c5\") "
Oct 30 21:24:46.417736 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:46.417674    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0dff592-d5a1-4832-8104-46cc5fd973c5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0dff592-d5a1-4832-8104-46cc5fd973c5-credential-socket\") pod \"d0dff592-d5a1-4832-8104-46cc5fd973c5\" (UID: \"d0dff592-d5a1-4832-8104-46cc5fd973c5\") "
Oct 30 21:24:46.418601 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:46.417763    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0dff592-d5a1-4832-8104-46cc5fd973c5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0dff592-d5a1-4832-8104-46cc5fd973c5-istio-data\") pod \"d0dff592-d5a1-4832-8104-46cc5fd973c5\" (UID: \"d0dff592-d5a1-4832-8104-46cc5fd973c5\") "
Oct 30 21:24:46.418601 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:46.417905    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0dff592-d5a1-4832-8104-46cc5fd973c5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0dff592-d5a1-4832-8104-46cc5fd973c5-istiod-ca-cert\") pod \"d0dff592-d5a1-4832-8104-46cc5fd973c5\" (UID: \"d0dff592-d5a1-4832-8104-46cc5fd973c5\") "
Oct 30 21:24:46.418601 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:46.417933    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0dff592-d5a1-4832-8104-46cc5fd973c5-workload-socket" pod "d0dff592-d5a1-4832-8104-46cc5fd973c5" (UID: "d0dff592-d5a1-4832-8104-46cc5fd973c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:24:46.418601 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:46.418055    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0dff592-d5a1-4832-8104-46cc5fd973c5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0dff592-d5a1-4832-8104-46cc5fd973c5-workload-certs\") pod \"d0dff592-d5a1-4832-8104-46cc5fd973c5\" (UID: \"d0dff592-d5a1-4832-8104-46cc5fd973c5\") "
Oct 30 21:24:46.418601 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:46.418147    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0dff592-d5a1-4832-8104-46cc5fd973c5-kube-api-access-2tcj4\" (UniqueName: \"kubernetes.io/projected/d0dff592-d5a1-4832-8104-46cc5fd973c5-kube-api-access-2tcj4\") pod \"d0dff592-d5a1-4832-8104-46cc5fd973c5\" (UID: \"d0dff592-d5a1-4832-8104-46cc5fd973c5\") "
Oct 30 21:24:46.418912 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:46.418605    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0dff592-d5a1-4832-8104-46cc5fd973c5-istio-data" pod "d0dff592-d5a1-4832-8104-46cc5fd973c5" (UID: "d0dff592-d5a1-4832-8104-46cc5fd973c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:24:46.418912 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:46.418674    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0dff592-d5a1-4832-8104-46cc5fd973c5-workload-certs" pod "d0dff592-d5a1-4832-8104-46cc5fd973c5" (UID: "d0dff592-d5a1-4832-8104-46cc5fd973c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:24:46.418912 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:46.418827    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0dff592-d5a1-4832-8104-46cc5fd973c5-credential-socket" pod "d0dff592-d5a1-4832-8104-46cc5fd973c5" (UID: "d0dff592-d5a1-4832-8104-46cc5fd973c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:24:46.419247 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:46.419061    5713 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0dff592-d5a1-4832-8104-46cc5fd973c5-workload-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:46.419247 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:46.419079    5713 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0dff592-d5a1-4832-8104-46cc5fd973c5-credential-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:46.419247 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:46.419093    5713 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0dff592-d5a1-4832-8104-46cc5fd973c5-istio-data\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:46.419247 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:46.419107    5713 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0dff592-d5a1-4832-8104-46cc5fd973c5-workload-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:46.419621 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:46.419586    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0dff592-d5a1-4832-8104-46cc5fd973c5-istiod-ca-cert" pod "d0dff592-d5a1-4832-8104-46cc5fd973c5" (UID: "d0dff592-d5a1-4832-8104-46cc5fd973c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:24:46.421398 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:46.421297    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d0dff592-d5a1-4832-8104-46cc5fd973c5-istio-podinfo" pod "d0dff592-d5a1-4832-8104-46cc5fd973c5" (UID: "d0dff592-d5a1-4832-8104-46cc5fd973c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:24:46.421398 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:46.421377    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0dff592-d5a1-4832-8104-46cc5fd973c5-istio-envoy" pod "d0dff592-d5a1-4832-8104-46cc5fd973c5" (UID: "d0dff592-d5a1-4832-8104-46cc5fd973c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:24:46.421649 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:46.421623    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0dff592-d5a1-4832-8104-46cc5fd973c5-istio-token" pod "d0dff592-d5a1-4832-8104-46cc5fd973c5" (UID: "d0dff592-d5a1-4832-8104-46cc5fd973c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:24:46.422182 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:46.422161    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0dff592-d5a1-4832-8104-46cc5fd973c5-kube-api-access-2tcj4" pod "d0dff592-d5a1-4832-8104-46cc5fd973c5" (UID: "d0dff592-d5a1-4832-8104-46cc5fd973c5"). InnerVolumeSpecName "kube-api-access-2tcj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:24:46.520139 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:46.520099    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tcj4\" (UniqueName: \"kubernetes.io/projected/d0dff592-d5a1-4832-8104-46cc5fd973c5-kube-api-access-2tcj4\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:46.520139 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:46.520132    5713 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0dff592-d5a1-4832-8104-46cc5fd973c5-istio-podinfo\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:46.520139 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:46.520142    5713 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0dff592-d5a1-4832-8104-46cc5fd973c5-istio-envoy\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:46.520384 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:46.520153    5713 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0dff592-d5a1-4832-8104-46cc5fd973c5-istio-token\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:46.520384 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:46.520163    5713 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0dff592-d5a1-4832-8104-46cc5fd973c5-istiod-ca-cert\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:47.100489 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:47.100457    5713 generic.go:356] "Generic (PLEG): container finished" podID="d0dff592-d5a1-4832-8104-46cc5fd973c5" containerID="eb0fe2865c319539aa0fcaba4ec26a0846a4830c675c707d4ee10ebde1983082" exitCode=0
Oct 30 21:24:47.100650 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:47.100496    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg"
Oct 30 21:24:47.100650 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:47.100530    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg" event={"ID":"d0dff592-d5a1-4832-8104-46cc5fd973c5","Type":"ContainerDied","Data":"eb0fe2865c319539aa0fcaba4ec26a0846a4830c675c707d4ee10ebde1983082"}
Oct 30 21:24:47.100650 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:47.100582    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg" event={"ID":"d0dff592-d5a1-4832-8104-46cc5fd973c5","Type":"ContainerDied","Data":"7bf3846ed31e973bed68487e7622297e831b68bbf74eac744cb981a22ff4c18f"}
Oct 30 21:24:47.100650 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:47.100606    5713 scope.go:122] "RemoveContainer" containerID="eb0fe2865c319539aa0fcaba4ec26a0846a4830c675c707d4ee10ebde1983082"
Oct 30 21:24:47.110855 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:47.110839    5713 scope.go:122] "RemoveContainer" containerID="607a33906cd32389d6ea58d710f74b7d5db210e30b2d21241e6e873832713ff8"
Oct 30 21:24:47.123698 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:47.123678    5713 scope.go:122] "RemoveContainer" containerID="eb0fe2865c319539aa0fcaba4ec26a0846a4830c675c707d4ee10ebde1983082"
Oct 30 21:24:47.124019 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:24:47.124000    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb0fe2865c319539aa0fcaba4ec26a0846a4830c675c707d4ee10ebde1983082\": container with ID starting with eb0fe2865c319539aa0fcaba4ec26a0846a4830c675c707d4ee10ebde1983082 not found: ID does not exist" containerID="eb0fe2865c319539aa0fcaba4ec26a0846a4830c675c707d4ee10ebde1983082"
Oct 30 21:24:47.124106 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:47.124031    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb0fe2865c319539aa0fcaba4ec26a0846a4830c675c707d4ee10ebde1983082"} err="failed to get container status \"eb0fe2865c319539aa0fcaba4ec26a0846a4830c675c707d4ee10ebde1983082\": rpc error: code = NotFound desc = could not find container \"eb0fe2865c319539aa0fcaba4ec26a0846a4830c675c707d4ee10ebde1983082\": container with ID starting with eb0fe2865c319539aa0fcaba4ec26a0846a4830c675c707d4ee10ebde1983082 not found: ID does not exist"
Oct 30 21:24:47.124106 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:47.124051    5713 scope.go:122] "RemoveContainer" containerID="607a33906cd32389d6ea58d710f74b7d5db210e30b2d21241e6e873832713ff8"
Oct 30 21:24:47.124331 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:24:47.124317    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"607a33906cd32389d6ea58d710f74b7d5db210e30b2d21241e6e873832713ff8\": container with ID starting with 607a33906cd32389d6ea58d710f74b7d5db210e30b2d21241e6e873832713ff8 not found: ID does not exist" containerID="607a33906cd32389d6ea58d710f74b7d5db210e30b2d21241e6e873832713ff8"
Oct 30 21:24:47.124383 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:47.124334    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"607a33906cd32389d6ea58d710f74b7d5db210e30b2d21241e6e873832713ff8"} err="failed to get container status \"607a33906cd32389d6ea58d710f74b7d5db210e30b2d21241e6e873832713ff8\": rpc error: code = NotFound desc = could not find container \"607a33906cd32389d6ea58d710f74b7d5db210e30b2d21241e6e873832713ff8\": container with ID starting with 607a33906cd32389d6ea58d710f74b7d5db210e30b2d21241e6e873832713ff8 not found: ID does not exist"
Oct 30 21:24:47.266803 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:47.266759    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg"]
Oct 30 21:24:47.316184 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:47.316153    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-f0fe13-openshift-default-54876585b5-zpsxg"]
Oct 30 21:24:48.669015 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:48.668979    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-682999-5489c955b7-dzmt9" podUID="230f6842-289d-43fd-b1b7-849c27c5f97b" containerName="payload-pre-processing" containerID="cri-o://3e751404069b40b372c7beb6b585a4373c732f633e03ae0dea87b9a88fe1a275" gracePeriod=25
Oct 30 21:24:49.062857 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:49.062822    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0dff592-d5a1-4832-8104-46cc5fd973c5" path="/var/lib/kubelet/pods/d0dff592-d5a1-4832-8104-46cc5fd973c5/volumes"
Oct 30 21:24:50.244566 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.244528    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-682999/distinct-2-w3-682999-kserve-7bc687885b-gq5x7" podUID="0dcf0422-081f-46d7-b246-7a49b074c88c" containerName="main" containerID="cri-o://d17cd1119ea5e3005d37fe27c968e8d570ddcd27e133b97288548a2b01fd3b3c" gracePeriod=45
Oct 30 21:24:50.248002 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.247949    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-682999/distinct-w3-682999-kserve-74d49d4955-nndlc" podUID="2a35314b-31ea-42e6-b49a-15ab1305b338" containerName="main" containerID="cri-o://e187cec6c8d8aa1be06bc77fccee4db794c953591f172f7af75e0d8bb68cbbdf" gracePeriod=45
Oct 30 21:24:50.264244 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.264212    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-682999/premium-w3-682999-kserve-5f47cf9b8d-xqnnr" podUID="a54cd33d-feef-42bb-a78b-7480634cb58d" containerName="main" containerID="cri-o://e3eee1e460f9157c14b58bcbd9c0e9f1f91a8d3395ca6267ddcbd0db5540458f" gracePeriod=45
Oct 30 21:24:50.268640 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.268614    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-682999/simulator-w3-682999-kserve-57cfcb6f8f-6mgrg" podUID="53296e7f-c6bd-45f2-ac31-004feddc63a7" containerName="main" containerID="cri-o://d7ee6f5080fa7876e24f0680c9b284c34d7aee714b48cb2ac58d36d8348e318f" gracePeriod=45
Oct 30 21:24:50.545411 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.545383    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-682999/distinct-2-w3-682999-kserve-7bc687885b-gq5x7"
Oct 30 21:24:50.549369 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.549346    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-682999/premium-w3-682999-kserve-5f47cf9b8d-xqnnr"
Oct 30 21:24:50.577092 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.577070    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-682999/simulator-w3-682999-kserve-57cfcb6f8f-6mgrg"
Oct 30 21:24:50.581914 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.581893    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-682999/distinct-w3-682999-kserve-74d49d4955-nndlc"
Oct 30 21:24:50.673890 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.673862    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a54cd33d-feef-42bb-a78b-7480634cb58d-home\" (UniqueName: \"kubernetes.io/empty-dir/a54cd33d-feef-42bb-a78b-7480634cb58d-home\") pod \"a54cd33d-feef-42bb-a78b-7480634cb58d\" (UID: \"a54cd33d-feef-42bb-a78b-7480634cb58d\") "
Oct 30 21:24:50.674038 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.673922    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a54cd33d-feef-42bb-a78b-7480634cb58d-kube-api-access-48ptx\" (UniqueName: \"kubernetes.io/projected/a54cd33d-feef-42bb-a78b-7480634cb58d-kube-api-access-48ptx\") pod \"a54cd33d-feef-42bb-a78b-7480634cb58d\" (UID: \"a54cd33d-feef-42bb-a78b-7480634cb58d\") "
Oct 30 21:24:50.674038 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.673943    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dcf0422-081f-46d7-b246-7a49b074c88c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dcf0422-081f-46d7-b246-7a49b074c88c-dshm\") pod \"0dcf0422-081f-46d7-b246-7a49b074c88c\" (UID: \"0dcf0422-081f-46d7-b246-7a49b074c88c\") "
Oct 30 21:24:50.674038 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.673973    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a35314b-31ea-42e6-b49a-15ab1305b338-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a35314b-31ea-42e6-b49a-15ab1305b338-model-cache\") pod \"2a35314b-31ea-42e6-b49a-15ab1305b338\" (UID: \"2a35314b-31ea-42e6-b49a-15ab1305b338\") "
Oct 30 21:24:50.674038 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.673992    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a35314b-31ea-42e6-b49a-15ab1305b338-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a35314b-31ea-42e6-b49a-15ab1305b338-tmp-dir\") pod \"2a35314b-31ea-42e6-b49a-15ab1305b338\" (UID: \"2a35314b-31ea-42e6-b49a-15ab1305b338\") "
Oct 30 21:24:50.674038 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.674013    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a35314b-31ea-42e6-b49a-15ab1305b338-tls-certs\" (UniqueName: \"kubernetes.io/secret/2a35314b-31ea-42e6-b49a-15ab1305b338-tls-certs\") pod \"2a35314b-31ea-42e6-b49a-15ab1305b338\" (UID: \"2a35314b-31ea-42e6-b49a-15ab1305b338\") "
Oct 30 21:24:50.674355 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.674039    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53296e7f-c6bd-45f2-ac31-004feddc63a7-kube-api-access-2d4bb\" (UniqueName: \"kubernetes.io/projected/53296e7f-c6bd-45f2-ac31-004feddc63a7-kube-api-access-2d4bb\") pod \"53296e7f-c6bd-45f2-ac31-004feddc63a7\" (UID: \"53296e7f-c6bd-45f2-ac31-004feddc63a7\") "
Oct 30 21:24:50.674355 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.674076    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53296e7f-c6bd-45f2-ac31-004feddc63a7-tls-certs\" (UniqueName: \"kubernetes.io/secret/53296e7f-c6bd-45f2-ac31-004feddc63a7-tls-certs\") pod \"53296e7f-c6bd-45f2-ac31-004feddc63a7\" (UID: \"53296e7f-c6bd-45f2-ac31-004feddc63a7\") "
Oct 30 21:24:50.674355 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.674146    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53296e7f-c6bd-45f2-ac31-004feddc63a7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/53296e7f-c6bd-45f2-ac31-004feddc63a7-dshm\") pod \"53296e7f-c6bd-45f2-ac31-004feddc63a7\" (UID: \"53296e7f-c6bd-45f2-ac31-004feddc63a7\") "
Oct 30 21:24:50.674355 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.674158    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a54cd33d-feef-42bb-a78b-7480634cb58d-home" pod "a54cd33d-feef-42bb-a78b-7480634cb58d" (UID: "a54cd33d-feef-42bb-a78b-7480634cb58d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:24:50.674355 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.674176    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dcf0422-081f-46d7-b246-7a49b074c88c-home\" (UniqueName: \"kubernetes.io/empty-dir/0dcf0422-081f-46d7-b246-7a49b074c88c-home\") pod \"0dcf0422-081f-46d7-b246-7a49b074c88c\" (UID: \"0dcf0422-081f-46d7-b246-7a49b074c88c\") "
Oct 30 21:24:50.674355 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.674232    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a35314b-31ea-42e6-b49a-15ab1305b338-model-cache" pod "2a35314b-31ea-42e6-b49a-15ab1305b338" (UID: "2a35314b-31ea-42e6-b49a-15ab1305b338"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:24:50.674355 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.674260    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a35314b-31ea-42e6-b49a-15ab1305b338-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a35314b-31ea-42e6-b49a-15ab1305b338-dshm\") pod \"2a35314b-31ea-42e6-b49a-15ab1305b338\" (UID: \"2a35314b-31ea-42e6-b49a-15ab1305b338\") "
Oct 30 21:24:50.674355 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.674326    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53296e7f-c6bd-45f2-ac31-004feddc63a7-home\" (UniqueName: \"kubernetes.io/empty-dir/53296e7f-c6bd-45f2-ac31-004feddc63a7-home\") pod \"53296e7f-c6bd-45f2-ac31-004feddc63a7\" (UID: \"53296e7f-c6bd-45f2-ac31-004feddc63a7\") "
Oct 30 21:24:50.674958 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.674364    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dcf0422-081f-46d7-b246-7a49b074c88c-home" pod "0dcf0422-081f-46d7-b246-7a49b074c88c" (UID: "0dcf0422-081f-46d7-b246-7a49b074c88c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:24:50.674958 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.674396    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a54cd33d-feef-42bb-a78b-7480634cb58d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a54cd33d-feef-42bb-a78b-7480634cb58d-dshm\") pod \"a54cd33d-feef-42bb-a78b-7480634cb58d\" (UID: \"a54cd33d-feef-42bb-a78b-7480634cb58d\") "
Oct 30 21:24:50.674958 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.674431    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53296e7f-c6bd-45f2-ac31-004feddc63a7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53296e7f-c6bd-45f2-ac31-004feddc63a7-tmp-dir\") pod \"53296e7f-c6bd-45f2-ac31-004feddc63a7\" (UID: \"53296e7f-c6bd-45f2-ac31-004feddc63a7\") "
Oct 30 21:24:50.674958 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.674530    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dcf0422-081f-46d7-b246-7a49b074c88c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dcf0422-081f-46d7-b246-7a49b074c88c-tmp-dir\") pod \"0dcf0422-081f-46d7-b246-7a49b074c88c\" (UID: \"0dcf0422-081f-46d7-b246-7a49b074c88c\") "
Oct 30 21:24:50.674958 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.674607    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53296e7f-c6bd-45f2-ac31-004feddc63a7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53296e7f-c6bd-45f2-ac31-004feddc63a7-model-cache\") pod \"53296e7f-c6bd-45f2-ac31-004feddc63a7\" (UID: \"53296e7f-c6bd-45f2-ac31-004feddc63a7\") "
Oct 30 21:24:50.674958 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.674660    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a54cd33d-feef-42bb-a78b-7480634cb58d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a54cd33d-feef-42bb-a78b-7480634cb58d-tmp-dir\") pod \"a54cd33d-feef-42bb-a78b-7480634cb58d\" (UID: \"a54cd33d-feef-42bb-a78b-7480634cb58d\") "
Oct 30 21:24:50.674958 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.674696    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dcf0422-081f-46d7-b246-7a49b074c88c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dcf0422-081f-46d7-b246-7a49b074c88c-model-cache\") pod \"0dcf0422-081f-46d7-b246-7a49b074c88c\" (UID: \"0dcf0422-081f-46d7-b246-7a49b074c88c\") "
Oct 30 21:24:50.674958 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.674750    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a54cd33d-feef-42bb-a78b-7480634cb58d-tls-certs\" (UniqueName: \"kubernetes.io/secret/a54cd33d-feef-42bb-a78b-7480634cb58d-tls-certs\") pod \"a54cd33d-feef-42bb-a78b-7480634cb58d\" (UID: \"a54cd33d-feef-42bb-a78b-7480634cb58d\") "
Oct 30 21:24:50.674958 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.674775    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a35314b-31ea-42e6-b49a-15ab1305b338-home\" (UniqueName: \"kubernetes.io/empty-dir/2a35314b-31ea-42e6-b49a-15ab1305b338-home\") pod \"2a35314b-31ea-42e6-b49a-15ab1305b338\" (UID: \"2a35314b-31ea-42e6-b49a-15ab1305b338\") "
Oct 30 21:24:50.674958 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.674847    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dcf0422-081f-46d7-b246-7a49b074c88c-kube-api-access-ftnnk\" (UniqueName: \"kubernetes.io/projected/0dcf0422-081f-46d7-b246-7a49b074c88c-kube-api-access-ftnnk\") pod \"0dcf0422-081f-46d7-b246-7a49b074c88c\" (UID: \"0dcf0422-081f-46d7-b246-7a49b074c88c\") "
Oct 30 21:24:50.674958 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.674884    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a35314b-31ea-42e6-b49a-15ab1305b338-kube-api-access-5qb6b\" (UniqueName: \"kubernetes.io/projected/2a35314b-31ea-42e6-b49a-15ab1305b338-kube-api-access-5qb6b\") pod \"2a35314b-31ea-42e6-b49a-15ab1305b338\" (UID: \"2a35314b-31ea-42e6-b49a-15ab1305b338\") "
Oct 30 21:24:50.674958 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.674928    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a54cd33d-feef-42bb-a78b-7480634cb58d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a54cd33d-feef-42bb-a78b-7480634cb58d-model-cache\") pod \"a54cd33d-feef-42bb-a78b-7480634cb58d\" (UID: \"a54cd33d-feef-42bb-a78b-7480634cb58d\") "
Oct 30 21:24:50.675595 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.674967    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0dcf0422-081f-46d7-b246-7a49b074c88c-tls-certs\" (UniqueName: \"kubernetes.io/secret/0dcf0422-081f-46d7-b246-7a49b074c88c-tls-certs\") pod \"0dcf0422-081f-46d7-b246-7a49b074c88c\" (UID: \"0dcf0422-081f-46d7-b246-7a49b074c88c\") "
Oct 30 21:24:50.675873 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.675807    5713 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a54cd33d-feef-42bb-a78b-7480634cb58d-home\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:50.675873 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.675834    5713 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a35314b-31ea-42e6-b49a-15ab1305b338-model-cache\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:50.675873 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.675849    5713 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0dcf0422-081f-46d7-b246-7a49b074c88c-home\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:50.676707 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.674529    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a35314b-31ea-42e6-b49a-15ab1305b338-tmp-dir" pod "2a35314b-31ea-42e6-b49a-15ab1305b338" (UID: "2a35314b-31ea-42e6-b49a-15ab1305b338"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:24:50.676707 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.676499    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dcf0422-081f-46d7-b246-7a49b074c88c-model-cache" pod "0dcf0422-081f-46d7-b246-7a49b074c88c" (UID: "0dcf0422-081f-46d7-b246-7a49b074c88c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:24:50.676707 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.676682    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53296e7f-c6bd-45f2-ac31-004feddc63a7-home" pod "53296e7f-c6bd-45f2-ac31-004feddc63a7" (UID: "53296e7f-c6bd-45f2-ac31-004feddc63a7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:24:50.676707 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.676687    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a54cd33d-feef-42bb-a78b-7480634cb58d-tmp-dir" pod "a54cd33d-feef-42bb-a78b-7480634cb58d" (UID: "a54cd33d-feef-42bb-a78b-7480634cb58d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:24:50.676707 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.676696    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53296e7f-c6bd-45f2-ac31-004feddc63a7-model-cache" pod "53296e7f-c6bd-45f2-ac31-004feddc63a7" (UID: "53296e7f-c6bd-45f2-ac31-004feddc63a7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:24:50.678242 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.677314    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53296e7f-c6bd-45f2-ac31-004feddc63a7-tmp-dir" pod "53296e7f-c6bd-45f2-ac31-004feddc63a7" (UID: "53296e7f-c6bd-45f2-ac31-004feddc63a7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:24:50.678242 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.677818    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a35314b-31ea-42e6-b49a-15ab1305b338-home" pod "2a35314b-31ea-42e6-b49a-15ab1305b338" (UID: "2a35314b-31ea-42e6-b49a-15ab1305b338"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:24:50.678242 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.677883    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dcf0422-081f-46d7-b246-7a49b074c88c-dshm" pod "0dcf0422-081f-46d7-b246-7a49b074c88c" (UID: "0dcf0422-081f-46d7-b246-7a49b074c88c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:24:50.678242 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.677930    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a54cd33d-feef-42bb-a78b-7480634cb58d-model-cache" pod "a54cd33d-feef-42bb-a78b-7480634cb58d" (UID: "a54cd33d-feef-42bb-a78b-7480634cb58d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:24:50.678242 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.678138    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dcf0422-081f-46d7-b246-7a49b074c88c-tmp-dir" pod "0dcf0422-081f-46d7-b246-7a49b074c88c" (UID: "0dcf0422-081f-46d7-b246-7a49b074c88c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:24:50.679166 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.679124    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53296e7f-c6bd-45f2-ac31-004feddc63a7-tls-certs" pod "53296e7f-c6bd-45f2-ac31-004feddc63a7" (UID: "53296e7f-c6bd-45f2-ac31-004feddc63a7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:24:50.679304 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.679284    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a35314b-31ea-42e6-b49a-15ab1305b338-dshm" pod "2a35314b-31ea-42e6-b49a-15ab1305b338" (UID: "2a35314b-31ea-42e6-b49a-15ab1305b338"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:24:50.679934 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.679900    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53296e7f-c6bd-45f2-ac31-004feddc63a7-dshm" pod "53296e7f-c6bd-45f2-ac31-004feddc63a7" (UID: "53296e7f-c6bd-45f2-ac31-004feddc63a7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:24:50.680049 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.679932    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dcf0422-081f-46d7-b246-7a49b074c88c-kube-api-access-ftnnk" pod "0dcf0422-081f-46d7-b246-7a49b074c88c" (UID: "0dcf0422-081f-46d7-b246-7a49b074c88c"). InnerVolumeSpecName "kube-api-access-ftnnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:24:50.680821 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.680756    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53296e7f-c6bd-45f2-ac31-004feddc63a7-kube-api-access-2d4bb" pod "53296e7f-c6bd-45f2-ac31-004feddc63a7" (UID: "53296e7f-c6bd-45f2-ac31-004feddc63a7"). InnerVolumeSpecName "kube-api-access-2d4bb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:24:50.680957 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.680927    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a35314b-31ea-42e6-b49a-15ab1305b338-kube-api-access-5qb6b" pod "2a35314b-31ea-42e6-b49a-15ab1305b338" (UID: "2a35314b-31ea-42e6-b49a-15ab1305b338"). InnerVolumeSpecName "kube-api-access-5qb6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:24:50.681067 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.681006    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a54cd33d-feef-42bb-a78b-7480634cb58d-dshm" pod "a54cd33d-feef-42bb-a78b-7480634cb58d" (UID: "a54cd33d-feef-42bb-a78b-7480634cb58d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:24:50.681151 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.681067    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a54cd33d-feef-42bb-a78b-7480634cb58d-kube-api-access-48ptx" pod "a54cd33d-feef-42bb-a78b-7480634cb58d" (UID: "a54cd33d-feef-42bb-a78b-7480634cb58d"). InnerVolumeSpecName "kube-api-access-48ptx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:24:50.681239 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.681213    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a54cd33d-feef-42bb-a78b-7480634cb58d-tls-certs" pod "a54cd33d-feef-42bb-a78b-7480634cb58d" (UID: "a54cd33d-feef-42bb-a78b-7480634cb58d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:24:50.681938 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.681919    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a35314b-31ea-42e6-b49a-15ab1305b338-tls-certs" pod "2a35314b-31ea-42e6-b49a-15ab1305b338" (UID: "2a35314b-31ea-42e6-b49a-15ab1305b338"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:24:50.688629 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.688605    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dcf0422-081f-46d7-b246-7a49b074c88c-tls-certs" pod "0dcf0422-081f-46d7-b246-7a49b074c88c" (UID: "0dcf0422-081f-46d7-b246-7a49b074c88c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:24:50.777533 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.777464    5713 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a54cd33d-feef-42bb-a78b-7480634cb58d-dshm\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:50.777533 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.777491    5713 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53296e7f-c6bd-45f2-ac31-004feddc63a7-tmp-dir\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:50.777533 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.777500    5713 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dcf0422-081f-46d7-b246-7a49b074c88c-tmp-dir\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:50.777533 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.777509    5713 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53296e7f-c6bd-45f2-ac31-004feddc63a7-model-cache\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:50.777533 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.777516    5713 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a54cd33d-feef-42bb-a78b-7480634cb58d-tmp-dir\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:50.777533 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.777524    5713 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dcf0422-081f-46d7-b246-7a49b074c88c-model-cache\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:50.777533 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.777532    5713 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a54cd33d-feef-42bb-a78b-7480634cb58d-tls-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:50.777916 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.777540    5713 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a35314b-31ea-42e6-b49a-15ab1305b338-home\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:50.777916 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.777549    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftnnk\" (UniqueName: \"kubernetes.io/projected/0dcf0422-081f-46d7-b246-7a49b074c88c-kube-api-access-ftnnk\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:50.777916 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.777561    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qb6b\" (UniqueName: \"kubernetes.io/projected/2a35314b-31ea-42e6-b49a-15ab1305b338-kube-api-access-5qb6b\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:50.777916 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.777573    5713 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a54cd33d-feef-42bb-a78b-7480634cb58d-model-cache\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:50.777916 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.777589    5713 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dcf0422-081f-46d7-b246-7a49b074c88c-tls-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:50.777916 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.777597    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48ptx\" (UniqueName: \"kubernetes.io/projected/a54cd33d-feef-42bb-a78b-7480634cb58d-kube-api-access-48ptx\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:50.777916 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.777608    5713 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dcf0422-081f-46d7-b246-7a49b074c88c-dshm\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:50.777916 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.777616    5713 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a35314b-31ea-42e6-b49a-15ab1305b338-tmp-dir\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:50.777916 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.777625    5713 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a35314b-31ea-42e6-b49a-15ab1305b338-tls-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:50.777916 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.777633    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2d4bb\" (UniqueName: \"kubernetes.io/projected/53296e7f-c6bd-45f2-ac31-004feddc63a7-kube-api-access-2d4bb\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:50.777916 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.777640    5713 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53296e7f-c6bd-45f2-ac31-004feddc63a7-tls-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:50.777916 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.777648    5713 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53296e7f-c6bd-45f2-ac31-004feddc63a7-dshm\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:50.777916 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.777655    5713 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a35314b-31ea-42e6-b49a-15ab1305b338-dshm\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:50.777916 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:50.777662    5713 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53296e7f-c6bd-45f2-ac31-004feddc63a7-home\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:51.130577 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:51.130544    5713 generic.go:356] "Generic (PLEG): container finished" podID="53296e7f-c6bd-45f2-ac31-004feddc63a7" containerID="d7ee6f5080fa7876e24f0680c9b284c34d7aee714b48cb2ac58d36d8348e318f" exitCode=0
Oct 30 21:24:51.130757 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:51.130600    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-682999/simulator-w3-682999-kserve-57cfcb6f8f-6mgrg"
Oct 30 21:24:51.130757 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:51.130625    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-682999/simulator-w3-682999-kserve-57cfcb6f8f-6mgrg" event={"ID":"53296e7f-c6bd-45f2-ac31-004feddc63a7","Type":"ContainerDied","Data":"d7ee6f5080fa7876e24f0680c9b284c34d7aee714b48cb2ac58d36d8348e318f"}
Oct 30 21:24:51.130757 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:51.130678    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-682999/simulator-w3-682999-kserve-57cfcb6f8f-6mgrg" event={"ID":"53296e7f-c6bd-45f2-ac31-004feddc63a7","Type":"ContainerDied","Data":"53b656b9968d2435a783f75a9bfb35f98e8d71e8d47b9eebda422ff927d06919"}
Oct 30 21:24:51.130757 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:51.130700    5713 scope.go:122] "RemoveContainer" containerID="d7ee6f5080fa7876e24f0680c9b284c34d7aee714b48cb2ac58d36d8348e318f"
Oct 30 21:24:51.132466 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:51.132373    5713 generic.go:356] "Generic (PLEG): container finished" podID="2a35314b-31ea-42e6-b49a-15ab1305b338" containerID="e187cec6c8d8aa1be06bc77fccee4db794c953591f172f7af75e0d8bb68cbbdf" exitCode=0
Oct 30 21:24:51.132466 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:51.132445    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-682999/distinct-w3-682999-kserve-74d49d4955-nndlc"
Oct 30 21:24:51.132466 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:51.132474    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-682999/distinct-w3-682999-kserve-74d49d4955-nndlc" event={"ID":"2a35314b-31ea-42e6-b49a-15ab1305b338","Type":"ContainerDied","Data":"e187cec6c8d8aa1be06bc77fccee4db794c953591f172f7af75e0d8bb68cbbdf"}
Oct 30 21:24:51.132836 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:51.132540    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-682999/distinct-w3-682999-kserve-74d49d4955-nndlc" event={"ID":"2a35314b-31ea-42e6-b49a-15ab1305b338","Type":"ContainerDied","Data":"63c003533c2abf821ef17b3a61bd8ce7e0536031572ec2888255a5828da9378f"}
Oct 30 21:24:51.134254 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:51.134230    5713 generic.go:356] "Generic (PLEG): container finished" podID="0dcf0422-081f-46d7-b246-7a49b074c88c" containerID="d17cd1119ea5e3005d37fe27c968e8d570ddcd27e133b97288548a2b01fd3b3c" exitCode=0
Oct 30 21:24:51.134353 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:51.134309    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-682999/distinct-2-w3-682999-kserve-7bc687885b-gq5x7"
Oct 30 21:24:51.134434 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:51.134310    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-682999/distinct-2-w3-682999-kserve-7bc687885b-gq5x7" event={"ID":"0dcf0422-081f-46d7-b246-7a49b074c88c","Type":"ContainerDied","Data":"d17cd1119ea5e3005d37fe27c968e8d570ddcd27e133b97288548a2b01fd3b3c"}
Oct 30 21:24:51.134434 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:51.134412    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-682999/distinct-2-w3-682999-kserve-7bc687885b-gq5x7" event={"ID":"0dcf0422-081f-46d7-b246-7a49b074c88c","Type":"ContainerDied","Data":"c57fed04416968d812c9959d8194394e2c503825dc2f3b716c4713c8ada39581"}
Oct 30 21:24:51.136006 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:51.135981    5713 generic.go:356] "Generic (PLEG): container finished" podID="a54cd33d-feef-42bb-a78b-7480634cb58d" containerID="e3eee1e460f9157c14b58bcbd9c0e9f1f91a8d3395ca6267ddcbd0db5540458f" exitCode=0
Oct 30 21:24:51.136114 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:51.136023    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-682999/premium-w3-682999-kserve-5f47cf9b8d-xqnnr"
Oct 30 21:24:51.136114 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:51.136052    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-682999/premium-w3-682999-kserve-5f47cf9b8d-xqnnr" event={"ID":"a54cd33d-feef-42bb-a78b-7480634cb58d","Type":"ContainerDied","Data":"e3eee1e460f9157c14b58bcbd9c0e9f1f91a8d3395ca6267ddcbd0db5540458f"}
Oct 30 21:24:51.136214 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:51.136123    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-682999/premium-w3-682999-kserve-5f47cf9b8d-xqnnr" event={"ID":"a54cd33d-feef-42bb-a78b-7480634cb58d","Type":"ContainerDied","Data":"4cde660a526396ed02a18238250340456e8c35a229fee6f1ccfa79b48c7de78f"}
Oct 30 21:24:51.142106 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:51.142088    5713 scope.go:122] "RemoveContainer" containerID="d7ee6f5080fa7876e24f0680c9b284c34d7aee714b48cb2ac58d36d8348e318f"
Oct 30 21:24:51.142375 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:24:51.142358    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7ee6f5080fa7876e24f0680c9b284c34d7aee714b48cb2ac58d36d8348e318f\": container with ID starting with d7ee6f5080fa7876e24f0680c9b284c34d7aee714b48cb2ac58d36d8348e318f not found: ID does not exist" containerID="d7ee6f5080fa7876e24f0680c9b284c34d7aee714b48cb2ac58d36d8348e318f"
Oct 30 21:24:51.142423 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:51.142383    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7ee6f5080fa7876e24f0680c9b284c34d7aee714b48cb2ac58d36d8348e318f"} err="failed to get container status \"d7ee6f5080fa7876e24f0680c9b284c34d7aee714b48cb2ac58d36d8348e318f\": rpc error: code = NotFound desc = could not find container \"d7ee6f5080fa7876e24f0680c9b284c34d7aee714b48cb2ac58d36d8348e318f\": container with ID starting with d7ee6f5080fa7876e24f0680c9b284c34d7aee714b48cb2ac58d36d8348e318f not found: ID does not exist"
Oct 30 21:24:51.142423 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:51.142399    5713 scope.go:122] "RemoveContainer" containerID="e187cec6c8d8aa1be06bc77fccee4db794c953591f172f7af75e0d8bb68cbbdf"
Oct 30 21:24:51.151327 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:51.151310    5713 scope.go:122] "RemoveContainer" containerID="e187cec6c8d8aa1be06bc77fccee4db794c953591f172f7af75e0d8bb68cbbdf"
Oct 30 21:24:51.151617 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:24:51.151596    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e187cec6c8d8aa1be06bc77fccee4db794c953591f172f7af75e0d8bb68cbbdf\": container with ID starting with e187cec6c8d8aa1be06bc77fccee4db794c953591f172f7af75e0d8bb68cbbdf not found: ID does not exist" containerID="e187cec6c8d8aa1be06bc77fccee4db794c953591f172f7af75e0d8bb68cbbdf"
Oct 30 21:24:51.151662 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:51.151626    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e187cec6c8d8aa1be06bc77fccee4db794c953591f172f7af75e0d8bb68cbbdf"} err="failed to get container status \"e187cec6c8d8aa1be06bc77fccee4db794c953591f172f7af75e0d8bb68cbbdf\": rpc error: code = NotFound desc = could not find container \"e187cec6c8d8aa1be06bc77fccee4db794c953591f172f7af75e0d8bb68cbbdf\": container with ID starting with e187cec6c8d8aa1be06bc77fccee4db794c953591f172f7af75e0d8bb68cbbdf not found: ID does not exist"
Oct 30 21:24:51.151662 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:51.151643    5713 scope.go:122] "RemoveContainer" containerID="d17cd1119ea5e3005d37fe27c968e8d570ddcd27e133b97288548a2b01fd3b3c"
Oct 30 21:24:51.161662 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:51.161640    5713 scope.go:122] "RemoveContainer" containerID="d17cd1119ea5e3005d37fe27c968e8d570ddcd27e133b97288548a2b01fd3b3c"
Oct 30 21:24:51.161981 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:24:51.161957    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d17cd1119ea5e3005d37fe27c968e8d570ddcd27e133b97288548a2b01fd3b3c\": container with ID starting with d17cd1119ea5e3005d37fe27c968e8d570ddcd27e133b97288548a2b01fd3b3c not found: ID does not exist" containerID="d17cd1119ea5e3005d37fe27c968e8d570ddcd27e133b97288548a2b01fd3b3c"
Oct 30 21:24:51.162055 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:51.161991    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d17cd1119ea5e3005d37fe27c968e8d570ddcd27e133b97288548a2b01fd3b3c"} err="failed to get container status \"d17cd1119ea5e3005d37fe27c968e8d570ddcd27e133b97288548a2b01fd3b3c\": rpc error: code = NotFound desc = could not find container \"d17cd1119ea5e3005d37fe27c968e8d570ddcd27e133b97288548a2b01fd3b3c\": container with ID starting with d17cd1119ea5e3005d37fe27c968e8d570ddcd27e133b97288548a2b01fd3b3c not found: ID does not exist"
Oct 30 21:24:51.162055 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:51.162008    5713 scope.go:122] "RemoveContainer" containerID="e3eee1e460f9157c14b58bcbd9c0e9f1f91a8d3395ca6267ddcbd0db5540458f"
Oct 30 21:24:51.170521 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:51.170507    5713 scope.go:122] "RemoveContainer" containerID="e3eee1e460f9157c14b58bcbd9c0e9f1f91a8d3395ca6267ddcbd0db5540458f"
Oct 30 21:24:51.170835 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:24:51.170817    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3eee1e460f9157c14b58bcbd9c0e9f1f91a8d3395ca6267ddcbd0db5540458f\": container with ID starting with e3eee1e460f9157c14b58bcbd9c0e9f1f91a8d3395ca6267ddcbd0db5540458f not found: ID does not exist" containerID="e3eee1e460f9157c14b58bcbd9c0e9f1f91a8d3395ca6267ddcbd0db5540458f"
Oct 30 21:24:51.170895 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:51.170842    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3eee1e460f9157c14b58bcbd9c0e9f1f91a8d3395ca6267ddcbd0db5540458f"} err="failed to get container status \"e3eee1e460f9157c14b58bcbd9c0e9f1f91a8d3395ca6267ddcbd0db5540458f\": rpc error: code = NotFound desc = could not find container \"e3eee1e460f9157c14b58bcbd9c0e9f1f91a8d3395ca6267ddcbd0db5540458f\": container with ID starting with e3eee1e460f9157c14b58bcbd9c0e9f1f91a8d3395ca6267ddcbd0db5540458f not found: ID does not exist"
Oct 30 21:24:51.294075 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:51.294051    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-682999/simulator-w3-682999-kserve-57cfcb6f8f-6mgrg"]
Oct 30 21:24:51.297387 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:51.297367    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-682999/simulator-w3-682999-kserve-57cfcb6f8f-6mgrg"]
Oct 30 21:24:51.405438 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:51.405361    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-682999/premium-w3-682999-kserve-5f47cf9b8d-xqnnr"]
Oct 30 21:24:51.411348 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:51.411326    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-682999/premium-w3-682999-kserve-5f47cf9b8d-xqnnr"]
Oct 30 21:24:51.518693 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:51.518663    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-682999/distinct-2-w3-682999-kserve-7bc687885b-gq5x7"]
Oct 30 21:24:51.522502 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:51.522475    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-682999/distinct-2-w3-682999-kserve-7bc687885b-gq5x7"]
Oct 30 21:24:51.631464 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:51.631431    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-682999/distinct-w3-682999-kserve-74d49d4955-nndlc"]
Oct 30 21:24:51.635836 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:51.635813    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-682999/distinct-w3-682999-kserve-74d49d4955-nndlc"]
Oct 30 21:24:53.062409 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:53.062376    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dcf0422-081f-46d7-b246-7a49b074c88c" path="/var/lib/kubelet/pods/0dcf0422-081f-46d7-b246-7a49b074c88c/volumes"
Oct 30 21:24:53.062842 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:53.062827    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a35314b-31ea-42e6-b49a-15ab1305b338" path="/var/lib/kubelet/pods/2a35314b-31ea-42e6-b49a-15ab1305b338/volumes"
Oct 30 21:24:53.063218 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:53.063206    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53296e7f-c6bd-45f2-ac31-004feddc63a7" path="/var/lib/kubelet/pods/53296e7f-c6bd-45f2-ac31-004feddc63a7/volumes"
Oct 30 21:24:53.063902 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:53.063884    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a54cd33d-feef-42bb-a78b-7480634cb58d" path="/var/lib/kubelet/pods/a54cd33d-feef-42bb-a78b-7480634cb58d/volumes"
Oct 30 21:24:53.346897 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:53.346817    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-639b07-59547c9664-bqxz8"]
Oct 30 21:24:53.455598 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:53.455564    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-639b07-6dd84859d7-k49qq"]
Oct 30 21:24:54.459394 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:24:54.459353    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:24:55.551674 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:55.551630    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-682999-5489c955b7-dzmt9" podUID="230f6842-289d-43fd-b1b7-849c27c5f97b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:24:58.347226 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:58.347170    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-639b07-59547c9664-bqxz8" podUID="31d0639c-3365-4848-a878-3388c779886c" containerName="payload-processing" containerID="cri-o://e7c196fa1389ed446d6130ebc54a129530a0401aa6cf39f17acec6e539028598" gracePeriod=25
Oct 30 21:24:58.456211 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:58.456009    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-639b07-6dd84859d7-k49qq" podUID="f469bcf6-121f-4e38-a9a1-e0f4237b9242" containerName="payload-pre-processing" containerID="cri-o://7197b609e3127f2f63972f1822cd3c26aa25a79affdd079bd7ed7db11128c1a8" gracePeriod=25
Oct 30 21:24:58.616244 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:58.616176    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-639b07-59547c9664-bqxz8"
Oct 30 21:24:58.665598 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:58.665565    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31d0639c-3365-4848-a878-3388c779886c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31d0639c-3365-4848-a878-3388c779886c-plugins-config-volume\") pod \"31d0639c-3365-4848-a878-3388c779886c\" (UID: \"31d0639c-3365-4848-a878-3388c779886c\") "
Oct 30 21:24:58.665736 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:58.665706    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31d0639c-3365-4848-a878-3388c779886c-kube-api-access-bx2qc\" (UniqueName: \"kubernetes.io/projected/31d0639c-3365-4848-a878-3388c779886c-kube-api-access-bx2qc\") pod \"31d0639c-3365-4848-a878-3388c779886c\" (UID: \"31d0639c-3365-4848-a878-3388c779886c\") "
Oct 30 21:24:58.665950 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:58.665925    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31d0639c-3365-4848-a878-3388c779886c-plugins-config-volume" pod "31d0639c-3365-4848-a878-3388c779886c" (UID: "31d0639c-3365-4848-a878-3388c779886c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:24:58.666089 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:58.666077    5713 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31d0639c-3365-4848-a878-3388c779886c-plugins-config-volume\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:58.668130 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:58.668101    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31d0639c-3365-4848-a878-3388c779886c-kube-api-access-bx2qc" pod "31d0639c-3365-4848-a878-3388c779886c" (UID: "31d0639c-3365-4848-a878-3388c779886c"). InnerVolumeSpecName "kube-api-access-bx2qc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:24:58.716045 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:58.716015    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-639b07-6dd84859d7-k49qq"
Oct 30 21:24:58.767262 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:58.767228    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f469bcf6-121f-4e38-a9a1-e0f4237b9242-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f469bcf6-121f-4e38-a9a1-e0f4237b9242-plugins-config-volume\") pod \"f469bcf6-121f-4e38-a9a1-e0f4237b9242\" (UID: \"f469bcf6-121f-4e38-a9a1-e0f4237b9242\") "
Oct 30 21:24:58.767441 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:58.767309    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f469bcf6-121f-4e38-a9a1-e0f4237b9242-kube-api-access-68jhz\" (UniqueName: \"kubernetes.io/projected/f469bcf6-121f-4e38-a9a1-e0f4237b9242-kube-api-access-68jhz\") pod \"f469bcf6-121f-4e38-a9a1-e0f4237b9242\" (UID: \"f469bcf6-121f-4e38-a9a1-e0f4237b9242\") "
Oct 30 21:24:58.767656 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:58.767632    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f469bcf6-121f-4e38-a9a1-e0f4237b9242-plugins-config-volume" pod "f469bcf6-121f-4e38-a9a1-e0f4237b9242" (UID: "f469bcf6-121f-4e38-a9a1-e0f4237b9242"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:24:58.767820 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:58.767800    5713 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f469bcf6-121f-4e38-a9a1-e0f4237b9242-plugins-config-volume\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:58.767879 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:58.767829    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx2qc\" (UniqueName: \"kubernetes.io/projected/31d0639c-3365-4848-a878-3388c779886c-kube-api-access-bx2qc\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:58.769514 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:58.769492    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f469bcf6-121f-4e38-a9a1-e0f4237b9242-kube-api-access-68jhz" pod "f469bcf6-121f-4e38-a9a1-e0f4237b9242" (UID: "f469bcf6-121f-4e38-a9a1-e0f4237b9242"). InnerVolumeSpecName "kube-api-access-68jhz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:24:58.868664 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:58.868596    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68jhz\" (UniqueName: \"kubernetes.io/projected/f469bcf6-121f-4e38-a9a1-e0f4237b9242-kube-api-access-68jhz\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:24:59.198079 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:59.198044    5713 generic.go:356] "Generic (PLEG): container finished" podID="f469bcf6-121f-4e38-a9a1-e0f4237b9242" containerID="7197b609e3127f2f63972f1822cd3c26aa25a79affdd079bd7ed7db11128c1a8" exitCode=0
Oct 30 21:24:59.198264 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:59.198103    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-639b07-6dd84859d7-k49qq"
Oct 30 21:24:59.198264 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:59.198133    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-639b07-6dd84859d7-k49qq" event={"ID":"f469bcf6-121f-4e38-a9a1-e0f4237b9242","Type":"ContainerDied","Data":"7197b609e3127f2f63972f1822cd3c26aa25a79affdd079bd7ed7db11128c1a8"}
Oct 30 21:24:59.198264 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:59.198217    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-639b07-6dd84859d7-k49qq" event={"ID":"f469bcf6-121f-4e38-a9a1-e0f4237b9242","Type":"ContainerDied","Data":"caf4d205ace57903a1bea9b991c57032d72abb57c97423a275287aafe5adbc9b"}
Oct 30 21:24:59.198264 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:59.198244    5713 scope.go:122] "RemoveContainer" containerID="7197b609e3127f2f63972f1822cd3c26aa25a79affdd079bd7ed7db11128c1a8"
Oct 30 21:24:59.204545 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:59.204515    5713 generic.go:356] "Generic (PLEG): container finished" podID="31d0639c-3365-4848-a878-3388c779886c" containerID="e7c196fa1389ed446d6130ebc54a129530a0401aa6cf39f17acec6e539028598" exitCode=0
Oct 30 21:24:59.204692 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:59.204631    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-639b07-59547c9664-bqxz8" event={"ID":"31d0639c-3365-4848-a878-3388c779886c","Type":"ContainerDied","Data":"e7c196fa1389ed446d6130ebc54a129530a0401aa6cf39f17acec6e539028598"}
Oct 30 21:24:59.204751 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:59.204693    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-639b07-59547c9664-bqxz8"
Oct 30 21:24:59.204751 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:59.204707    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-639b07-59547c9664-bqxz8" event={"ID":"31d0639c-3365-4848-a878-3388c779886c","Type":"ContainerDied","Data":"8c1b53c84601cc4682c81f749258771f3c6d31e2e56adfd2a9b445299fd94534"}
Oct 30 21:24:59.212875 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:59.212853    5713 scope.go:122] "RemoveContainer" containerID="7197b609e3127f2f63972f1822cd3c26aa25a79affdd079bd7ed7db11128c1a8"
Oct 30 21:24:59.213155 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:24:59.213136    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7197b609e3127f2f63972f1822cd3c26aa25a79affdd079bd7ed7db11128c1a8\": container with ID starting with 7197b609e3127f2f63972f1822cd3c26aa25a79affdd079bd7ed7db11128c1a8 not found: ID does not exist" containerID="7197b609e3127f2f63972f1822cd3c26aa25a79affdd079bd7ed7db11128c1a8"
Oct 30 21:24:59.213208 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:59.213165    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7197b609e3127f2f63972f1822cd3c26aa25a79affdd079bd7ed7db11128c1a8"} err="failed to get container status \"7197b609e3127f2f63972f1822cd3c26aa25a79affdd079bd7ed7db11128c1a8\": rpc error: code = NotFound desc = could not find container \"7197b609e3127f2f63972f1822cd3c26aa25a79affdd079bd7ed7db11128c1a8\": container with ID starting with 7197b609e3127f2f63972f1822cd3c26aa25a79affdd079bd7ed7db11128c1a8 not found: ID does not exist"
Oct 30 21:24:59.213208 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:59.213183    5713 scope.go:122] "RemoveContainer" containerID="e7c196fa1389ed446d6130ebc54a129530a0401aa6cf39f17acec6e539028598"
Oct 30 21:24:59.221668 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:59.221636    5713 scope.go:122] "RemoveContainer" containerID="e7c196fa1389ed446d6130ebc54a129530a0401aa6cf39f17acec6e539028598"
Oct 30 21:24:59.221950 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:24:59.221933    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7c196fa1389ed446d6130ebc54a129530a0401aa6cf39f17acec6e539028598\": container with ID starting with e7c196fa1389ed446d6130ebc54a129530a0401aa6cf39f17acec6e539028598 not found: ID does not exist" containerID="e7c196fa1389ed446d6130ebc54a129530a0401aa6cf39f17acec6e539028598"
Oct 30 21:24:59.222016 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:59.221958    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7c196fa1389ed446d6130ebc54a129530a0401aa6cf39f17acec6e539028598"} err="failed to get container status \"e7c196fa1389ed446d6130ebc54a129530a0401aa6cf39f17acec6e539028598\": rpc error: code = NotFound desc = could not find container \"e7c196fa1389ed446d6130ebc54a129530a0401aa6cf39f17acec6e539028598\": container with ID starting with e7c196fa1389ed446d6130ebc54a129530a0401aa6cf39f17acec6e539028598 not found: ID does not exist"
Oct 30 21:24:59.365558 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:59.365529    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-639b07-59547c9664-bqxz8"]
Oct 30 21:24:59.368412 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:59.368390    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-639b07-59547c9664-bqxz8"]
Oct 30 21:24:59.478673 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:59.478584    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-639b07-6dd84859d7-k49qq"]
Oct 30 21:24:59.482316 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:24:59.482290    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-639b07-6dd84859d7-k49qq"]
Oct 30 21:25:01.062980 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:01.062947    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31d0639c-3365-4848-a878-3388c779886c" path="/var/lib/kubelet/pods/31d0639c-3365-4848-a878-3388c779886c/volumes"
Oct 30 21:25:01.063473 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:01.063358    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f469bcf6-121f-4e38-a9a1-e0f4237b9242" path="/var/lib/kubelet/pods/f469bcf6-121f-4e38-a9a1-e0f4237b9242/volumes"
Oct 30 21:25:05.551391 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:05.551354    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-682999-5489c955b7-dzmt9" podUID="230f6842-289d-43fd-b1b7-849c27c5f97b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:25:05.661115 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:05.661084    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7"]
Oct 30 21:25:05.661337 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:05.661308    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7" podUID="7e54eff2-416c-4a97-8ce1-2ea30c7e982a" containerName="istio-proxy" containerID="cri-o://bfb64ae3b71ab72444b59294b30251a88f71ba2e1d44dd5ab46ab655db6c6e9c" gracePeriod=30
Oct 30 21:25:05.663662 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:05.663634    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq"]
Oct 30 21:25:05.663845 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:05.663826    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq" podUID="21edc9eb-88e3-495f-870c-925863b09ce1" containerName="istio-proxy" containerID="cri-o://29e2fc6a89bc0ae81bbdde2d775e4af2599b92161def2347fffd75f647cc22a7" gracePeriod=30
Oct 30 21:25:05.682732 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:05.682701    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh"]
Oct 30 21:25:05.682948 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:05.682922    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh" podUID="4d34434b-cc8a-4ee2-a774-cacdea5465de" containerName="istio-proxy" containerID="cri-o://de9c872d9aa839eacc1977ca29a1b5845f26958755b06df95d519916c82821cd" gracePeriod=30
Oct 30 21:25:06.659006 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:06.658965    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8fa2a7-xdcsk"]
Oct 30 21:25:06.659457 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:06.659444    5713 cpu_manager.go:414] "containerMap: removing container" podUID="53296e7f-c6bd-45f2-ac31-004feddc63a7" containerName="main"
Oct 30 21:25:06.659502 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:06.659459    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53296e7f-c6bd-45f2-ac31-004feddc63a7" containerName="main"
Oct 30 21:25:06.659502 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:06.659473    5713 cpu_manager.go:414] "containerMap: removing container" podUID="f469bcf6-121f-4e38-a9a1-e0f4237b9242" containerName="payload-pre-processing"
Oct 30 21:25:06.659502 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:06.659478    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f469bcf6-121f-4e38-a9a1-e0f4237b9242" containerName="payload-pre-processing"
Oct 30 21:25:06.659502 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:06.659488    5713 cpu_manager.go:414] "containerMap: removing container" podUID="d0dff592-d5a1-4832-8104-46cc5fd973c5" containerName="istio-proxy"
Oct 30 21:25:06.659502 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:06.659498    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0dff592-d5a1-4832-8104-46cc5fd973c5" containerName="istio-proxy"
Oct 30 21:25:06.659663 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:06.659520    5713 cpu_manager.go:414] "containerMap: removing container" podUID="d0dff592-d5a1-4832-8104-46cc5fd973c5" containerName="istio-proxy"
Oct 30 21:25:06.659663 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:06.659524    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0dff592-d5a1-4832-8104-46cc5fd973c5" containerName="istio-proxy"
Oct 30 21:25:06.659663 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:06.659533    5713 cpu_manager.go:414] "containerMap: removing container" podUID="b559e33d-9df3-49f7-a9f3-b97c1c79c0e9" containerName="revoke-keys"
Oct 30 21:25:06.659663 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:06.659537    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b559e33d-9df3-49f7-a9f3-b97c1c79c0e9" containerName="revoke-keys"
Oct 30 21:25:06.659663 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:06.659544    5713 cpu_manager.go:414] "containerMap: removing container" podUID="a54cd33d-feef-42bb-a78b-7480634cb58d" containerName="main"
Oct 30 21:25:06.659663 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:06.659549    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a54cd33d-feef-42bb-a78b-7480634cb58d" containerName="main"
Oct 30 21:25:06.659663 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:06.659561    5713 cpu_manager.go:414] "containerMap: removing container" podUID="2a35314b-31ea-42e6-b49a-15ab1305b338" containerName="main"
Oct 30 21:25:06.659663 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:06.659566    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a35314b-31ea-42e6-b49a-15ab1305b338" containerName="main"
Oct 30 21:25:06.659663 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:06.659578    5713 cpu_manager.go:414] "containerMap: removing container" podUID="0dcf0422-081f-46d7-b246-7a49b074c88c" containerName="main"
Oct 30 21:25:06.659663 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:06.659583    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dcf0422-081f-46d7-b246-7a49b074c88c" containerName="main"
Oct 30 21:25:06.659663 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:06.659598    5713 cpu_manager.go:414] "containerMap: removing container" podUID="31d0639c-3365-4848-a878-3388c779886c" containerName="payload-processing"
Oct 30 21:25:06.659663 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:06.659603    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31d0639c-3365-4848-a878-3388c779886c" containerName="payload-processing"
Oct 30 21:25:06.660067 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:06.659677    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbb39bfd-2716-4ded-9e69-6131383afdb9" containerName="istio-proxy"
Oct 30 21:25:06.660067 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:06.659687    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0dff592-d5a1-4832-8104-46cc5fd973c5" containerName="istio-proxy"
Oct 30 21:25:06.660067 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:06.659694    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="b559e33d-9df3-49f7-a9f3-b97c1c79c0e9" containerName="revoke-keys"
Oct 30 21:25:06.660067 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:06.659700    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="a54cd33d-feef-42bb-a78b-7480634cb58d" containerName="main"
Oct 30 21:25:06.660067 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:06.659709    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a35314b-31ea-42e6-b49a-15ab1305b338" containerName="main"
Oct 30 21:25:06.660067 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:06.659715    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dcf0422-081f-46d7-b246-7a49b074c88c" containerName="main"
Oct 30 21:25:06.660067 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:06.659724    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="31d0639c-3365-4848-a878-3388c779886c" containerName="payload-processing"
Oct 30 21:25:06.660067 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:06.659733    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="53296e7f-c6bd-45f2-ac31-004feddc63a7" containerName="main"
Oct 30 21:25:06.660067 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:06.659739    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="f469bcf6-121f-4e38-a9a1-e0f4237b9242" containerName="payload-pre-processing"
Oct 30 21:25:06.663043 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:06.663024    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8fa2a7-xdcsk"
Oct 30 21:25:06.715773 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:06.715735    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:25:06.716020 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:06.715774    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:25:06.716020 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:06.715964    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:25:06.716183 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:06.716122    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:25:06.719691 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:06.719664    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8fa2a7-xdcsk"]
Oct 30 21:25:06.763264 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:06.763229    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2lv4\" (UniqueName: \"kubernetes.io/projected/b3dc86ed-d40b-43f4-b403-7c7840462545-kube-api-access-r2lv4\") pod \"maas-api-revoke-keys-e2e-worker-w6-8fa2a7-xdcsk\" (UID: \"b3dc86ed-d40b-43f4-b403-7c7840462545\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8fa2a7-xdcsk"
Oct 30 21:25:06.763444 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:06.763291    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3dc86ed-d40b-43f4-b403-7c7840462545-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-8fa2a7-xdcsk\" (UID: \"b3dc86ed-d40b-43f4-b403-7c7840462545\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8fa2a7-xdcsk"
Oct 30 21:25:06.763444 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:06.763434    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3dc86ed-d40b-43f4-b403-7c7840462545-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-8fa2a7-xdcsk\" (UID: \"b3dc86ed-d40b-43f4-b403-7c7840462545\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8fa2a7-xdcsk"
Oct 30 21:25:06.864997 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:06.864961    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3dc86ed-d40b-43f4-b403-7c7840462545-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-8fa2a7-xdcsk\" (UID: \"b3dc86ed-d40b-43f4-b403-7c7840462545\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8fa2a7-xdcsk"
Oct 30 21:25:06.865182 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:06.865048    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2lv4\" (UniqueName: \"kubernetes.io/projected/b3dc86ed-d40b-43f4-b403-7c7840462545-kube-api-access-r2lv4\") pod \"maas-api-revoke-keys-e2e-worker-w6-8fa2a7-xdcsk\" (UID: \"b3dc86ed-d40b-43f4-b403-7c7840462545\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8fa2a7-xdcsk"
Oct 30 21:25:06.865182 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:06.865088    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3dc86ed-d40b-43f4-b403-7c7840462545-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-8fa2a7-xdcsk\" (UID: \"b3dc86ed-d40b-43f4-b403-7c7840462545\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8fa2a7-xdcsk"
Oct 30 21:25:06.865775 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:06.865747    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3dc86ed-d40b-43f4-b403-7c7840462545-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-8fa2a7-xdcsk\" (UID: \"b3dc86ed-d40b-43f4-b403-7c7840462545\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8fa2a7-xdcsk"
Oct 30 21:25:06.867501 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:06.867480    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3dc86ed-d40b-43f4-b403-7c7840462545-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-8fa2a7-xdcsk\" (UID: \"b3dc86ed-d40b-43f4-b403-7c7840462545\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8fa2a7-xdcsk"
Oct 30 21:25:06.922339 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:06.922266    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2lv4\" (UniqueName: \"kubernetes.io/projected/b3dc86ed-d40b-43f4-b403-7c7840462545-kube-api-access-r2lv4\") pod \"maas-api-revoke-keys-e2e-worker-w6-8fa2a7-xdcsk\" (UID: \"b3dc86ed-d40b-43f4-b403-7c7840462545\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8fa2a7-xdcsk"
Oct 30 21:25:06.976854 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:06.976780    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8fa2a7-xdcsk"
Oct 30 21:25:07.351764 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:07.351731    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8fa2a7-xdcsk"]
Oct 30 21:25:07.354131 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:25:07.354101    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3dc86ed_d40b_43f4_b403_7c7840462545.slice/crio-4c23b4bba8cfa7aa6fae51ee32066948e004327968f3ceca76b60498e3a5c096 WatchSource:0}: Error finding container 4c23b4bba8cfa7aa6fae51ee32066948e004327968f3ceca76b60498e3a5c096: Status 404 returned error can't find the container with id 4c23b4bba8cfa7aa6fae51ee32066948e004327968f3ceca76b60498e3a5c096
Oct 30 21:25:08.265199 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:08.265171    5713 generic.go:356] "Generic (PLEG): container finished" podID="b3dc86ed-d40b-43f4-b403-7c7840462545" containerID="75fded7c3ce819ecfabf758745be1881ce8d661cf8e8d5f94cfbaad34a0ba08f" exitCode=0
Oct 30 21:25:08.265531 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:08.265229    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8fa2a7-xdcsk" event={"ID":"b3dc86ed-d40b-43f4-b403-7c7840462545","Type":"ContainerDied","Data":"75fded7c3ce819ecfabf758745be1881ce8d661cf8e8d5f94cfbaad34a0ba08f"}
Oct 30 21:25:08.265531 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:08.265281    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8fa2a7-xdcsk" event={"ID":"b3dc86ed-d40b-43f4-b403-7c7840462545","Type":"ContainerStarted","Data":"4c23b4bba8cfa7aa6fae51ee32066948e004327968f3ceca76b60498e3a5c096"}
Oct 30 21:25:08.939143 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:08.939119    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-682999-5489c955b7-dzmt9"
Oct 30 21:25:09.093296 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:09.093259    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/230f6842-289d-43fd-b1b7-849c27c5f97b-kube-api-access-9ckjv\" (UniqueName: \"kubernetes.io/projected/230f6842-289d-43fd-b1b7-849c27c5f97b-kube-api-access-9ckjv\") pod \"230f6842-289d-43fd-b1b7-849c27c5f97b\" (UID: \"230f6842-289d-43fd-b1b7-849c27c5f97b\") "
Oct 30 21:25:09.093499 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:09.093436    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/230f6842-289d-43fd-b1b7-849c27c5f97b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/230f6842-289d-43fd-b1b7-849c27c5f97b-plugins-config-volume\") pod \"230f6842-289d-43fd-b1b7-849c27c5f97b\" (UID: \"230f6842-289d-43fd-b1b7-849c27c5f97b\") "
Oct 30 21:25:09.093812 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:09.093768    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/230f6842-289d-43fd-b1b7-849c27c5f97b-plugins-config-volume" pod "230f6842-289d-43fd-b1b7-849c27c5f97b" (UID: "230f6842-289d-43fd-b1b7-849c27c5f97b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:25:09.095599 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:09.095575    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/230f6842-289d-43fd-b1b7-849c27c5f97b-kube-api-access-9ckjv" pod "230f6842-289d-43fd-b1b7-849c27c5f97b" (UID: "230f6842-289d-43fd-b1b7-849c27c5f97b"). InnerVolumeSpecName "kube-api-access-9ckjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:09.195416 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:09.195342    5713 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/230f6842-289d-43fd-b1b7-849c27c5f97b-plugins-config-volume\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:25:09.195416 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:09.195379    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ckjv\" (UniqueName: \"kubernetes.io/projected/230f6842-289d-43fd-b1b7-849c27c5f97b-kube-api-access-9ckjv\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:25:09.273338 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:09.273308    5713 generic.go:356] "Generic (PLEG): container finished" podID="230f6842-289d-43fd-b1b7-849c27c5f97b" containerID="3e751404069b40b372c7beb6b585a4373c732f633e03ae0dea87b9a88fe1a275" exitCode=0
Oct 30 21:25:09.273716 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:09.273421    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-682999-5489c955b7-dzmt9"
Oct 30 21:25:09.273716 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:09.273453    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-682999-5489c955b7-dzmt9" event={"ID":"230f6842-289d-43fd-b1b7-849c27c5f97b","Type":"ContainerDied","Data":"3e751404069b40b372c7beb6b585a4373c732f633e03ae0dea87b9a88fe1a275"}
Oct 30 21:25:09.273716 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:09.273521    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-682999-5489c955b7-dzmt9" event={"ID":"230f6842-289d-43fd-b1b7-849c27c5f97b","Type":"ContainerDied","Data":"a4c6f32d21873699db7942bab67d56310fac306376aa3f171828c65c46c3aabe"}
Oct 30 21:25:09.273716 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:09.273549    5713 scope.go:122] "RemoveContainer" containerID="3e751404069b40b372c7beb6b585a4373c732f633e03ae0dea87b9a88fe1a275"
Oct 30 21:25:09.284476 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:09.284431    5713 scope.go:122] "RemoveContainer" containerID="3e751404069b40b372c7beb6b585a4373c732f633e03ae0dea87b9a88fe1a275"
Oct 30 21:25:09.284759 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:25:09.284736    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e751404069b40b372c7beb6b585a4373c732f633e03ae0dea87b9a88fe1a275\": container with ID starting with 3e751404069b40b372c7beb6b585a4373c732f633e03ae0dea87b9a88fe1a275 not found: ID does not exist" containerID="3e751404069b40b372c7beb6b585a4373c732f633e03ae0dea87b9a88fe1a275"
Oct 30 21:25:09.284873 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:09.284773    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e751404069b40b372c7beb6b585a4373c732f633e03ae0dea87b9a88fe1a275"} err="failed to get container status \"3e751404069b40b372c7beb6b585a4373c732f633e03ae0dea87b9a88fe1a275\": rpc error: code = NotFound desc = could not find container \"3e751404069b40b372c7beb6b585a4373c732f633e03ae0dea87b9a88fe1a275\": container with ID starting with 3e751404069b40b372c7beb6b585a4373c732f633e03ae0dea87b9a88fe1a275 not found: ID does not exist"
Oct 30 21:25:09.418706 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:09.418680    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8fa2a7-xdcsk"
Oct 30 21:25:09.439522 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:09.439486    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-682999-5489c955b7-dzmt9"]
Oct 30 21:25:09.442696 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:09.442663    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-682999-5489c955b7-dzmt9"]
Oct 30 21:25:09.498383 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:09.498311    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3dc86ed-d40b-43f4-b403-7c7840462545-kube-api-access-r2lv4\" (UniqueName: \"kubernetes.io/projected/b3dc86ed-d40b-43f4-b403-7c7840462545-kube-api-access-r2lv4\") pod \"b3dc86ed-d40b-43f4-b403-7c7840462545\" (UID: \"b3dc86ed-d40b-43f4-b403-7c7840462545\") "
Oct 30 21:25:09.498535 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:09.498452    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3dc86ed-d40b-43f4-b403-7c7840462545-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3dc86ed-d40b-43f4-b403-7c7840462545-maas-api-openshift-ca\") pod \"b3dc86ed-d40b-43f4-b403-7c7840462545\" (UID: \"b3dc86ed-d40b-43f4-b403-7c7840462545\") "
Oct 30 21:25:09.498535 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:09.498498    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3dc86ed-d40b-43f4-b403-7c7840462545-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3dc86ed-d40b-43f4-b403-7c7840462545-maas-api-xks-ca\") pod \"b3dc86ed-d40b-43f4-b403-7c7840462545\" (UID: \"b3dc86ed-d40b-43f4-b403-7c7840462545\") "
Oct 30 21:25:09.498843 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:09.498820    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3dc86ed-d40b-43f4-b403-7c7840462545-maas-api-openshift-ca" pod "b3dc86ed-d40b-43f4-b403-7c7840462545" (UID: "b3dc86ed-d40b-43f4-b403-7c7840462545"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:25:09.498940 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:09.498927    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3dc86ed-d40b-43f4-b403-7c7840462545-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:25:09.500570 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:09.500548    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3dc86ed-d40b-43f4-b403-7c7840462545-kube-api-access-r2lv4" pod "b3dc86ed-d40b-43f4-b403-7c7840462545" (UID: "b3dc86ed-d40b-43f4-b403-7c7840462545"). InnerVolumeSpecName "kube-api-access-r2lv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:09.500684 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:09.500666    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3dc86ed-d40b-43f4-b403-7c7840462545-maas-api-xks-ca" pod "b3dc86ed-d40b-43f4-b403-7c7840462545" (UID: "b3dc86ed-d40b-43f4-b403-7c7840462545"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:25:09.600094 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:09.600055    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3dc86ed-d40b-43f4-b403-7c7840462545-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:25:09.600094 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:09.600091    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2lv4\" (UniqueName: \"kubernetes.io/projected/b3dc86ed-d40b-43f4-b403-7c7840462545-kube-api-access-r2lv4\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:25:10.281688 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:10.281658    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8fa2a7-xdcsk"
Oct 30 21:25:10.282159 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:10.281686    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8fa2a7-xdcsk" event={"ID":"b3dc86ed-d40b-43f4-b403-7c7840462545","Type":"ContainerDied","Data":"4c23b4bba8cfa7aa6fae51ee32066948e004327968f3ceca76b60498e3a5c096"}
Oct 30 21:25:10.282159 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:10.281742    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c23b4bba8cfa7aa6fae51ee32066948e004327968f3ceca76b60498e3a5c096"
Oct 30 21:25:10.957485 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:10.957456    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq"
Oct 30 21:25:11.009144 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.009116    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7"
Oct 30 21:25:11.011158 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.011141    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh"
Oct 30 21:25:11.061881 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.061852    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="230f6842-289d-43fd-b1b7-849c27c5f97b" path="/var/lib/kubelet/pods/230f6842-289d-43fd-b1b7-849c27c5f97b/volumes"
Oct 30 21:25:11.118180 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.118145    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e54eff2-416c-4a97-8ce1-2ea30c7e982a-kube-api-access-wjs7z\" (UniqueName: \"kubernetes.io/projected/7e54eff2-416c-4a97-8ce1-2ea30c7e982a-kube-api-access-wjs7z\") pod \"7e54eff2-416c-4a97-8ce1-2ea30c7e982a\" (UID: \"7e54eff2-416c-4a97-8ce1-2ea30c7e982a\") "
Oct 30 21:25:11.118384 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.118197    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21edc9eb-88e3-495f-870c-925863b09ce1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21edc9eb-88e3-495f-870c-925863b09ce1-istio-envoy\") pod \"21edc9eb-88e3-495f-870c-925863b09ce1\" (UID: \"21edc9eb-88e3-495f-870c-925863b09ce1\") "
Oct 30 21:25:11.118384 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.118229    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7e54eff2-416c-4a97-8ce1-2ea30c7e982a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e54eff2-416c-4a97-8ce1-2ea30c7e982a-istio-podinfo\") pod \"7e54eff2-416c-4a97-8ce1-2ea30c7e982a\" (UID: \"7e54eff2-416c-4a97-8ce1-2ea30c7e982a\") "
Oct 30 21:25:11.118384 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.118266    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e54eff2-416c-4a97-8ce1-2ea30c7e982a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e54eff2-416c-4a97-8ce1-2ea30c7e982a-istio-envoy\") pod \"7e54eff2-416c-4a97-8ce1-2ea30c7e982a\" (UID: \"7e54eff2-416c-4a97-8ce1-2ea30c7e982a\") "
Oct 30 21:25:11.118384 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.118294    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e54eff2-416c-4a97-8ce1-2ea30c7e982a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e54eff2-416c-4a97-8ce1-2ea30c7e982a-workload-socket\") pod \"7e54eff2-416c-4a97-8ce1-2ea30c7e982a\" (UID: \"7e54eff2-416c-4a97-8ce1-2ea30c7e982a\") "
Oct 30 21:25:11.118384 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.118325    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e54eff2-416c-4a97-8ce1-2ea30c7e982a-istio-token\" (UniqueName: \"kubernetes.io/projected/7e54eff2-416c-4a97-8ce1-2ea30c7e982a-istio-token\") pod \"7e54eff2-416c-4a97-8ce1-2ea30c7e982a\" (UID: \"7e54eff2-416c-4a97-8ce1-2ea30c7e982a\") "
Oct 30 21:25:11.118384 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.118382    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21edc9eb-88e3-495f-870c-925863b09ce1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21edc9eb-88e3-495f-870c-925863b09ce1-istio-data\") pod \"21edc9eb-88e3-495f-870c-925863b09ce1\" (UID: \"21edc9eb-88e3-495f-870c-925863b09ce1\") "
Oct 30 21:25:11.118735 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.118422    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4d34434b-cc8a-4ee2-a774-cacdea5465de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d34434b-cc8a-4ee2-a774-cacdea5465de-istio-podinfo\") pod \"4d34434b-cc8a-4ee2-a774-cacdea5465de\" (UID: \"4d34434b-cc8a-4ee2-a774-cacdea5465de\") "
Oct 30 21:25:11.118735 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.118468    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/21edc9eb-88e3-495f-870c-925863b09ce1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21edc9eb-88e3-495f-870c-925863b09ce1-istio-podinfo\") pod \"21edc9eb-88e3-495f-870c-925863b09ce1\" (UID: \"21edc9eb-88e3-495f-870c-925863b09ce1\") "
Oct 30 21:25:11.118735 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.118501    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21edc9eb-88e3-495f-870c-925863b09ce1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21edc9eb-88e3-495f-870c-925863b09ce1-istiod-ca-cert\") pod \"21edc9eb-88e3-495f-870c-925863b09ce1\" (UID: \"21edc9eb-88e3-495f-870c-925863b09ce1\") "
Oct 30 21:25:11.118735 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.118533    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21edc9eb-88e3-495f-870c-925863b09ce1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21edc9eb-88e3-495f-870c-925863b09ce1-workload-socket\") pod \"21edc9eb-88e3-495f-870c-925863b09ce1\" (UID: \"21edc9eb-88e3-495f-870c-925863b09ce1\") "
Oct 30 21:25:11.118735 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.118566    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21edc9eb-88e3-495f-870c-925863b09ce1-kube-api-access-vqvqb\" (UniqueName: \"kubernetes.io/projected/21edc9eb-88e3-495f-870c-925863b09ce1-kube-api-access-vqvqb\") pod \"21edc9eb-88e3-495f-870c-925863b09ce1\" (UID: \"21edc9eb-88e3-495f-870c-925863b09ce1\") "
Oct 30 21:25:11.118735 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.118629    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d34434b-cc8a-4ee2-a774-cacdea5465de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d34434b-cc8a-4ee2-a774-cacdea5465de-istio-envoy\") pod \"4d34434b-cc8a-4ee2-a774-cacdea5465de\" (UID: \"4d34434b-cc8a-4ee2-a774-cacdea5465de\") "
Oct 30 21:25:11.118735 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.118632    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e54eff2-416c-4a97-8ce1-2ea30c7e982a-workload-socket" pod "7e54eff2-416c-4a97-8ce1-2ea30c7e982a" (UID: "7e54eff2-416c-4a97-8ce1-2ea30c7e982a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:25:11.118735 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.118671    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e54eff2-416c-4a97-8ce1-2ea30c7e982a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e54eff2-416c-4a97-8ce1-2ea30c7e982a-istio-data\") pod \"7e54eff2-416c-4a97-8ce1-2ea30c7e982a\" (UID: \"7e54eff2-416c-4a97-8ce1-2ea30c7e982a\") "
Oct 30 21:25:11.118735 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.118709    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d34434b-cc8a-4ee2-a774-cacdea5465de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d34434b-cc8a-4ee2-a774-cacdea5465de-credential-socket\") pod \"4d34434b-cc8a-4ee2-a774-cacdea5465de\" (UID: \"4d34434b-cc8a-4ee2-a774-cacdea5465de\") "
Oct 30 21:25:11.118735 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.118740    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21edc9eb-88e3-495f-870c-925863b09ce1-istio-token\" (UniqueName: \"kubernetes.io/projected/21edc9eb-88e3-495f-870c-925863b09ce1-istio-token\") pod \"21edc9eb-88e3-495f-870c-925863b09ce1\" (UID: \"21edc9eb-88e3-495f-870c-925863b09ce1\") "
Oct 30 21:25:11.119460 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.118778    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21edc9eb-88e3-495f-870c-925863b09ce1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21edc9eb-88e3-495f-870c-925863b09ce1-credential-socket\") pod \"21edc9eb-88e3-495f-870c-925863b09ce1\" (UID: \"21edc9eb-88e3-495f-870c-925863b09ce1\") "
Oct 30 21:25:11.119460 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.118854    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21edc9eb-88e3-495f-870c-925863b09ce1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21edc9eb-88e3-495f-870c-925863b09ce1-workload-certs\") pod \"21edc9eb-88e3-495f-870c-925863b09ce1\" (UID: \"21edc9eb-88e3-495f-870c-925863b09ce1\") "
Oct 30 21:25:11.119460 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.118903    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e54eff2-416c-4a97-8ce1-2ea30c7e982a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e54eff2-416c-4a97-8ce1-2ea30c7e982a-credential-socket\") pod \"7e54eff2-416c-4a97-8ce1-2ea30c7e982a\" (UID: \"7e54eff2-416c-4a97-8ce1-2ea30c7e982a\") "
Oct 30 21:25:11.119460 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.118935    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e54eff2-416c-4a97-8ce1-2ea30c7e982a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e54eff2-416c-4a97-8ce1-2ea30c7e982a-istiod-ca-cert\") pod \"7e54eff2-416c-4a97-8ce1-2ea30c7e982a\" (UID: \"7e54eff2-416c-4a97-8ce1-2ea30c7e982a\") "
Oct 30 21:25:11.119460 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.118960    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d34434b-cc8a-4ee2-a774-cacdea5465de-istio-token\" (UniqueName: \"kubernetes.io/projected/4d34434b-cc8a-4ee2-a774-cacdea5465de-istio-token\") pod \"4d34434b-cc8a-4ee2-a774-cacdea5465de\" (UID: \"4d34434b-cc8a-4ee2-a774-cacdea5465de\") "
Oct 30 21:25:11.119460 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.118990    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d34434b-cc8a-4ee2-a774-cacdea5465de-kube-api-access-jblvf\" (UniqueName: \"kubernetes.io/projected/4d34434b-cc8a-4ee2-a774-cacdea5465de-kube-api-access-jblvf\") pod \"4d34434b-cc8a-4ee2-a774-cacdea5465de\" (UID: \"4d34434b-cc8a-4ee2-a774-cacdea5465de\") "
Oct 30 21:25:11.119460 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.119053    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e54eff2-416c-4a97-8ce1-2ea30c7e982a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e54eff2-416c-4a97-8ce1-2ea30c7e982a-workload-certs\") pod \"7e54eff2-416c-4a97-8ce1-2ea30c7e982a\" (UID: \"7e54eff2-416c-4a97-8ce1-2ea30c7e982a\") "
Oct 30 21:25:11.119460 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.119213    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21edc9eb-88e3-495f-870c-925863b09ce1-istiod-ca-cert" pod "21edc9eb-88e3-495f-870c-925863b09ce1" (UID: "21edc9eb-88e3-495f-870c-925863b09ce1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:25:11.119460 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.119245    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d34434b-cc8a-4ee2-a774-cacdea5465de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d34434b-cc8a-4ee2-a774-cacdea5465de-workload-socket\") pod \"4d34434b-cc8a-4ee2-a774-cacdea5465de\" (UID: \"4d34434b-cc8a-4ee2-a774-cacdea5465de\") "
Oct 30 21:25:11.119460 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.119289    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d34434b-cc8a-4ee2-a774-cacdea5465de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d34434b-cc8a-4ee2-a774-cacdea5465de-istio-data\") pod \"4d34434b-cc8a-4ee2-a774-cacdea5465de\" (UID: \"4d34434b-cc8a-4ee2-a774-cacdea5465de\") "
Oct 30 21:25:11.119460 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.119330    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d34434b-cc8a-4ee2-a774-cacdea5465de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d34434b-cc8a-4ee2-a774-cacdea5465de-workload-certs\") pod \"4d34434b-cc8a-4ee2-a774-cacdea5465de\" (UID: \"4d34434b-cc8a-4ee2-a774-cacdea5465de\") "
Oct 30 21:25:11.119460 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.119391    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d34434b-cc8a-4ee2-a774-cacdea5465de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d34434b-cc8a-4ee2-a774-cacdea5465de-istiod-ca-cert\") pod \"4d34434b-cc8a-4ee2-a774-cacdea5465de\" (UID: \"4d34434b-cc8a-4ee2-a774-cacdea5465de\") "
Oct 30 21:25:11.119460 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.119456    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21edc9eb-88e3-495f-870c-925863b09ce1-istio-data" pod "21edc9eb-88e3-495f-870c-925863b09ce1" (UID: "21edc9eb-88e3-495f-870c-925863b09ce1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:25:11.120249 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.119890    5713 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e54eff2-416c-4a97-8ce1-2ea30c7e982a-workload-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:25:11.120249 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.119910    5713 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21edc9eb-88e3-495f-870c-925863b09ce1-istio-data\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:25:11.120249 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.119926    5713 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21edc9eb-88e3-495f-870c-925863b09ce1-istiod-ca-cert\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:25:11.120413 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.120286    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d34434b-cc8a-4ee2-a774-cacdea5465de-istiod-ca-cert" pod "4d34434b-cc8a-4ee2-a774-cacdea5465de" (UID: "4d34434b-cc8a-4ee2-a774-cacdea5465de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:25:11.120560 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.120535    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21edc9eb-88e3-495f-870c-925863b09ce1-workload-socket" pod "21edc9eb-88e3-495f-870c-925863b09ce1" (UID: "21edc9eb-88e3-495f-870c-925863b09ce1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:25:11.121424 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.121192    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21edc9eb-88e3-495f-870c-925863b09ce1-istio-envoy" pod "21edc9eb-88e3-495f-870c-925863b09ce1" (UID: "21edc9eb-88e3-495f-870c-925863b09ce1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:25:11.121424 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.121279    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e54eff2-416c-4a97-8ce1-2ea30c7e982a-istio-token" pod "7e54eff2-416c-4a97-8ce1-2ea30c7e982a" (UID: "7e54eff2-416c-4a97-8ce1-2ea30c7e982a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:11.121632 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.121440    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e54eff2-416c-4a97-8ce1-2ea30c7e982a-credential-socket" pod "7e54eff2-416c-4a97-8ce1-2ea30c7e982a" (UID: "7e54eff2-416c-4a97-8ce1-2ea30c7e982a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:25:11.121632 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.121443    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e54eff2-416c-4a97-8ce1-2ea30c7e982a-kube-api-access-wjs7z" pod "7e54eff2-416c-4a97-8ce1-2ea30c7e982a" (UID: "7e54eff2-416c-4a97-8ce1-2ea30c7e982a"). InnerVolumeSpecName "kube-api-access-wjs7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:11.121632 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.121465    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e54eff2-416c-4a97-8ce1-2ea30c7e982a-istio-podinfo" pod "7e54eff2-416c-4a97-8ce1-2ea30c7e982a" (UID: "7e54eff2-416c-4a97-8ce1-2ea30c7e982a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:25:11.121632 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.121535    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d34434b-cc8a-4ee2-a774-cacdea5465de-credential-socket" pod "4d34434b-cc8a-4ee2-a774-cacdea5465de" (UID: "4d34434b-cc8a-4ee2-a774-cacdea5465de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:25:11.121903 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.121693    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e54eff2-416c-4a97-8ce1-2ea30c7e982a-workload-certs" pod "7e54eff2-416c-4a97-8ce1-2ea30c7e982a" (UID: "7e54eff2-416c-4a97-8ce1-2ea30c7e982a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:25:11.122264 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.122228    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e54eff2-416c-4a97-8ce1-2ea30c7e982a-istio-envoy" pod "7e54eff2-416c-4a97-8ce1-2ea30c7e982a" (UID: "7e54eff2-416c-4a97-8ce1-2ea30c7e982a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:25:11.122488 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.122463    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d34434b-cc8a-4ee2-a774-cacdea5465de-istio-data" pod "4d34434b-cc8a-4ee2-a774-cacdea5465de" (UID: "4d34434b-cc8a-4ee2-a774-cacdea5465de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:25:11.122870 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.122637    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d34434b-cc8a-4ee2-a774-cacdea5465de-istio-podinfo" pod "4d34434b-cc8a-4ee2-a774-cacdea5465de" (UID: "4d34434b-cc8a-4ee2-a774-cacdea5465de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:25:11.122870 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.122652    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d34434b-cc8a-4ee2-a774-cacdea5465de-workload-socket" pod "4d34434b-cc8a-4ee2-a774-cacdea5465de" (UID: "4d34434b-cc8a-4ee2-a774-cacdea5465de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:25:11.123029 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.122877    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d34434b-cc8a-4ee2-a774-cacdea5465de-workload-certs" pod "4d34434b-cc8a-4ee2-a774-cacdea5465de" (UID: "4d34434b-cc8a-4ee2-a774-cacdea5465de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:25:11.123029 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.122905    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21edc9eb-88e3-495f-870c-925863b09ce1-credential-socket" pod "21edc9eb-88e3-495f-870c-925863b09ce1" (UID: "21edc9eb-88e3-495f-870c-925863b09ce1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:25:11.123164 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.123102    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e54eff2-416c-4a97-8ce1-2ea30c7e982a-istio-data" pod "7e54eff2-416c-4a97-8ce1-2ea30c7e982a" (UID: "7e54eff2-416c-4a97-8ce1-2ea30c7e982a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:25:11.123248 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.123220    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21edc9eb-88e3-495f-870c-925863b09ce1-kube-api-access-vqvqb" pod "21edc9eb-88e3-495f-870c-925863b09ce1" (UID: "21edc9eb-88e3-495f-870c-925863b09ce1"). InnerVolumeSpecName "kube-api-access-vqvqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:11.123333 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.123304    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21edc9eb-88e3-495f-870c-925863b09ce1-workload-certs" pod "21edc9eb-88e3-495f-870c-925863b09ce1" (UID: "21edc9eb-88e3-495f-870c-925863b09ce1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:25:11.123395 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.123232    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e54eff2-416c-4a97-8ce1-2ea30c7e982a-istiod-ca-cert" pod "7e54eff2-416c-4a97-8ce1-2ea30c7e982a" (UID: "7e54eff2-416c-4a97-8ce1-2ea30c7e982a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:25:11.123686 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.123665    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21edc9eb-88e3-495f-870c-925863b09ce1-istio-token" pod "21edc9eb-88e3-495f-870c-925863b09ce1" (UID: "21edc9eb-88e3-495f-870c-925863b09ce1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:11.123770 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.123720    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21edc9eb-88e3-495f-870c-925863b09ce1-istio-podinfo" pod "21edc9eb-88e3-495f-870c-925863b09ce1" (UID: "21edc9eb-88e3-495f-870c-925863b09ce1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:25:11.123770 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.123746    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d34434b-cc8a-4ee2-a774-cacdea5465de-istio-envoy" pod "4d34434b-cc8a-4ee2-a774-cacdea5465de" (UID: "4d34434b-cc8a-4ee2-a774-cacdea5465de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:25:11.123956 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.123939    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d34434b-cc8a-4ee2-a774-cacdea5465de-kube-api-access-jblvf" pod "4d34434b-cc8a-4ee2-a774-cacdea5465de" (UID: "4d34434b-cc8a-4ee2-a774-cacdea5465de"). InnerVolumeSpecName "kube-api-access-jblvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:11.124356 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.124333    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d34434b-cc8a-4ee2-a774-cacdea5465de-istio-token" pod "4d34434b-cc8a-4ee2-a774-cacdea5465de" (UID: "4d34434b-cc8a-4ee2-a774-cacdea5465de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:11.134108 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.134077    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-8fa2a7/distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm"]
Oct 30 21:25:11.139110 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.139084    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-8fa2a7/distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7"]
Oct 30 21:25:11.221155 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.221117    5713 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e54eff2-416c-4a97-8ce1-2ea30c7e982a-istio-envoy\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:25:11.221155 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.221146    5713 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e54eff2-416c-4a97-8ce1-2ea30c7e982a-istio-token\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:25:11.221155 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.221156    5713 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d34434b-cc8a-4ee2-a774-cacdea5465de-istio-podinfo\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:25:11.221155 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.221165    5713 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21edc9eb-88e3-495f-870c-925863b09ce1-istio-podinfo\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:25:11.221496 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.221175    5713 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21edc9eb-88e3-495f-870c-925863b09ce1-workload-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:25:11.221496 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.221184    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqvqb\" (UniqueName: \"kubernetes.io/projected/21edc9eb-88e3-495f-870c-925863b09ce1-kube-api-access-vqvqb\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:25:11.221496 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.221193    5713 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d34434b-cc8a-4ee2-a774-cacdea5465de-istio-envoy\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:25:11.221496 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.221201    5713 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e54eff2-416c-4a97-8ce1-2ea30c7e982a-istio-data\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:25:11.221496 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.221210    5713 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d34434b-cc8a-4ee2-a774-cacdea5465de-credential-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:25:11.221496 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.221219    5713 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21edc9eb-88e3-495f-870c-925863b09ce1-istio-token\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:25:11.221496 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.221228    5713 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21edc9eb-88e3-495f-870c-925863b09ce1-credential-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:25:11.221496 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.221238    5713 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21edc9eb-88e3-495f-870c-925863b09ce1-workload-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:25:11.221496 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.221247    5713 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e54eff2-416c-4a97-8ce1-2ea30c7e982a-credential-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:25:11.221496 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.221255    5713 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e54eff2-416c-4a97-8ce1-2ea30c7e982a-istiod-ca-cert\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:25:11.221496 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.221263    5713 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d34434b-cc8a-4ee2-a774-cacdea5465de-istio-token\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:25:11.221496 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.221271    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jblvf\" (UniqueName: \"kubernetes.io/projected/4d34434b-cc8a-4ee2-a774-cacdea5465de-kube-api-access-jblvf\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:25:11.221496 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.221279    5713 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e54eff2-416c-4a97-8ce1-2ea30c7e982a-workload-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:25:11.221496 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.221289    5713 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d34434b-cc8a-4ee2-a774-cacdea5465de-workload-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:25:11.221496 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.221297    5713 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d34434b-cc8a-4ee2-a774-cacdea5465de-istio-data\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:25:11.221496 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.221305    5713 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d34434b-cc8a-4ee2-a774-cacdea5465de-workload-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:25:11.221496 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.221313    5713 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d34434b-cc8a-4ee2-a774-cacdea5465de-istiod-ca-cert\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:25:11.221496 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.221321    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjs7z\" (UniqueName: \"kubernetes.io/projected/7e54eff2-416c-4a97-8ce1-2ea30c7e982a-kube-api-access-wjs7z\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:25:11.221496 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.221329    5713 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21edc9eb-88e3-495f-870c-925863b09ce1-istio-envoy\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:25:11.221496 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.221339    5713 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e54eff2-416c-4a97-8ce1-2ea30c7e982a-istio-podinfo\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:25:11.303889 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.303842    5713 generic.go:356] "Generic (PLEG): container finished" podID="4d34434b-cc8a-4ee2-a774-cacdea5465de" containerID="de9c872d9aa839eacc1977ca29a1b5845f26958755b06df95d519916c82821cd" exitCode=0
Oct 30 21:25:11.304392 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.303898    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh" event={"ID":"4d34434b-cc8a-4ee2-a774-cacdea5465de","Type":"ContainerDied","Data":"de9c872d9aa839eacc1977ca29a1b5845f26958755b06df95d519916c82821cd"}
Oct 30 21:25:11.304392 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.303949    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh"
Oct 30 21:25:11.304392 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.304013    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh" event={"ID":"4d34434b-cc8a-4ee2-a774-cacdea5465de","Type":"ContainerDied","Data":"65372081de886ca60ef75c1d96951e6c51810c9a657090b96dd954ead10acefb"}
Oct 30 21:25:11.304392 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.304049    5713 scope.go:122] "RemoveContainer" containerID="de9c872d9aa839eacc1977ca29a1b5845f26958755b06df95d519916c82821cd"
Oct 30 21:25:11.307851 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.305849    5713 generic.go:356] "Generic (PLEG): container finished" podID="7e54eff2-416c-4a97-8ce1-2ea30c7e982a" containerID="bfb64ae3b71ab72444b59294b30251a88f71ba2e1d44dd5ab46ab655db6c6e9c" exitCode=0
Oct 30 21:25:11.307851 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.305913    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7"
Oct 30 21:25:11.307851 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.305987    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7" event={"ID":"7e54eff2-416c-4a97-8ce1-2ea30c7e982a","Type":"ContainerDied","Data":"bfb64ae3b71ab72444b59294b30251a88f71ba2e1d44dd5ab46ab655db6c6e9c"}
Oct 30 21:25:11.307851 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.306041    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7" event={"ID":"7e54eff2-416c-4a97-8ce1-2ea30c7e982a","Type":"ContainerDied","Data":"5733e2f6025d8c019885cbf46efceb972a845510653f6f577f166c2e7da3161f"}
Oct 30 21:25:11.307851 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.307840    5713 generic.go:356] "Generic (PLEG): container finished" podID="21edc9eb-88e3-495f-870c-925863b09ce1" containerID="29e2fc6a89bc0ae81bbdde2d775e4af2599b92161def2347fffd75f647cc22a7" exitCode=0
Oct 30 21:25:11.308174 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.307878    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq" event={"ID":"21edc9eb-88e3-495f-870c-925863b09ce1","Type":"ContainerDied","Data":"29e2fc6a89bc0ae81bbdde2d775e4af2599b92161def2347fffd75f647cc22a7"}
Oct 30 21:25:11.308174 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.307910    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq"
Oct 30 21:25:11.308174 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.307941    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq" event={"ID":"21edc9eb-88e3-495f-870c-925863b09ce1","Type":"ContainerDied","Data":"6f231fd8564319bc84e7683bde5056a73d2c540d0a1dcd6744804176fec6ced1"}
Oct 30 21:25:11.319661 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.319622    5713 scope.go:122] "RemoveContainer" containerID="e00836866e297848f37cc5fd9f64d6e203b7203567da64847fd01764c6023368"
Oct 30 21:25:11.332537 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.332514    5713 scope.go:122] "RemoveContainer" containerID="de9c872d9aa839eacc1977ca29a1b5845f26958755b06df95d519916c82821cd"
Oct 30 21:25:11.332900 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:25:11.332845    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de9c872d9aa839eacc1977ca29a1b5845f26958755b06df95d519916c82821cd\": container with ID starting with de9c872d9aa839eacc1977ca29a1b5845f26958755b06df95d519916c82821cd not found: ID does not exist" containerID="de9c872d9aa839eacc1977ca29a1b5845f26958755b06df95d519916c82821cd"
Oct 30 21:25:11.332989 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.332894    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de9c872d9aa839eacc1977ca29a1b5845f26958755b06df95d519916c82821cd"} err="failed to get container status \"de9c872d9aa839eacc1977ca29a1b5845f26958755b06df95d519916c82821cd\": rpc error: code = NotFound desc = could not find container \"de9c872d9aa839eacc1977ca29a1b5845f26958755b06df95d519916c82821cd\": container with ID starting with de9c872d9aa839eacc1977ca29a1b5845f26958755b06df95d519916c82821cd not found: ID does not exist"
Oct 30 21:25:11.332989 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.332922    5713 scope.go:122] "RemoveContainer" containerID="e00836866e297848f37cc5fd9f64d6e203b7203567da64847fd01764c6023368"
Oct 30 21:25:11.333248 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:25:11.333227    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e00836866e297848f37cc5fd9f64d6e203b7203567da64847fd01764c6023368\": container with ID starting with e00836866e297848f37cc5fd9f64d6e203b7203567da64847fd01764c6023368 not found: ID does not exist" containerID="e00836866e297848f37cc5fd9f64d6e203b7203567da64847fd01764c6023368"
Oct 30 21:25:11.333300 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.333254    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e00836866e297848f37cc5fd9f64d6e203b7203567da64847fd01764c6023368"} err="failed to get container status \"e00836866e297848f37cc5fd9f64d6e203b7203567da64847fd01764c6023368\": rpc error: code = NotFound desc = could not find container \"e00836866e297848f37cc5fd9f64d6e203b7203567da64847fd01764c6023368\": container with ID starting with e00836866e297848f37cc5fd9f64d6e203b7203567da64847fd01764c6023368 not found: ID does not exist"
Oct 30 21:25:11.333300 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.333271    5713 scope.go:122] "RemoveContainer" containerID="bfb64ae3b71ab72444b59294b30251a88f71ba2e1d44dd5ab46ab655db6c6e9c"
Oct 30 21:25:11.342820 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.342770    5713 scope.go:122] "RemoveContainer" containerID="bfb64ae3b71ab72444b59294b30251a88f71ba2e1d44dd5ab46ab655db6c6e9c"
Oct 30 21:25:11.343151 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:25:11.343127    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfb64ae3b71ab72444b59294b30251a88f71ba2e1d44dd5ab46ab655db6c6e9c\": container with ID starting with bfb64ae3b71ab72444b59294b30251a88f71ba2e1d44dd5ab46ab655db6c6e9c not found: ID does not exist" containerID="bfb64ae3b71ab72444b59294b30251a88f71ba2e1d44dd5ab46ab655db6c6e9c"
Oct 30 21:25:11.343227 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.343159    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfb64ae3b71ab72444b59294b30251a88f71ba2e1d44dd5ab46ab655db6c6e9c"} err="failed to get container status \"bfb64ae3b71ab72444b59294b30251a88f71ba2e1d44dd5ab46ab655db6c6e9c\": rpc error: code = NotFound desc = could not find container \"bfb64ae3b71ab72444b59294b30251a88f71ba2e1d44dd5ab46ab655db6c6e9c\": container with ID starting with bfb64ae3b71ab72444b59294b30251a88f71ba2e1d44dd5ab46ab655db6c6e9c not found: ID does not exist"
Oct 30 21:25:11.343227 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.343176    5713 scope.go:122] "RemoveContainer" containerID="29e2fc6a89bc0ae81bbdde2d775e4af2599b92161def2347fffd75f647cc22a7"
Oct 30 21:25:11.352735 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.352714    5713 scope.go:122] "RemoveContainer" containerID="29e2fc6a89bc0ae81bbdde2d775e4af2599b92161def2347fffd75f647cc22a7"
Oct 30 21:25:11.353059 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:25:11.353035    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29e2fc6a89bc0ae81bbdde2d775e4af2599b92161def2347fffd75f647cc22a7\": container with ID starting with 29e2fc6a89bc0ae81bbdde2d775e4af2599b92161def2347fffd75f647cc22a7 not found: ID does not exist" containerID="29e2fc6a89bc0ae81bbdde2d775e4af2599b92161def2347fffd75f647cc22a7"
Oct 30 21:25:11.353158 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.353064    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29e2fc6a89bc0ae81bbdde2d775e4af2599b92161def2347fffd75f647cc22a7"} err="failed to get container status \"29e2fc6a89bc0ae81bbdde2d775e4af2599b92161def2347fffd75f647cc22a7\": rpc error: code = NotFound desc = could not find container \"29e2fc6a89bc0ae81bbdde2d775e4af2599b92161def2347fffd75f647cc22a7\": container with ID starting with 29e2fc6a89bc0ae81bbdde2d775e4af2599b92161def2347fffd75f647cc22a7 not found: ID does not exist"
Oct 30 21:25:11.859894 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.859864    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh"]
Oct 30 21:25:11.863954 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.863927    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-zqdbh"]
Oct 30 21:25:11.972123 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.972078    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7"]
Oct 30 21:25:11.975761 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:11.975735    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-n9bv7"]
Oct 30 21:25:12.085437 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:12.085377    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq"]
Oct 30 21:25:12.089974 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:12.089938    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d1b889bd-openshift-default-566bd56c85-lpmxq"]
Oct 30 21:25:13.061810 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:13.061759    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21edc9eb-88e3-495f-870c-925863b09ce1" path="/var/lib/kubelet/pods/21edc9eb-88e3-495f-870c-925863b09ce1/volumes"
Oct 30 21:25:13.062209 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:13.062195    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d34434b-cc8a-4ee2-a774-cacdea5465de" path="/var/lib/kubelet/pods/4d34434b-cc8a-4ee2-a774-cacdea5465de/volumes"
Oct 30 21:25:13.062608 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:13.062595    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e54eff2-416c-4a97-8ce1-2ea30c7e982a" path="/var/lib/kubelet/pods/7e54eff2-416c-4a97-8ce1-2ea30c7e982a/volumes"
Oct 30 21:25:13.874848 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:13.874816    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1be5211-l9qt4"]
Oct 30 21:25:13.875351 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:13.875329    5713 cpu_manager.go:414] "containerMap: removing container" podUID="7e54eff2-416c-4a97-8ce1-2ea30c7e982a" containerName="istio-proxy"
Oct 30 21:25:13.875460 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:13.875353    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e54eff2-416c-4a97-8ce1-2ea30c7e982a" containerName="istio-proxy"
Oct 30 21:25:13.875460 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:13.875378    5713 cpu_manager.go:414] "containerMap: removing container" podUID="21edc9eb-88e3-495f-870c-925863b09ce1" containerName="istio-proxy"
Oct 30 21:25:13.875460 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:13.875385    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21edc9eb-88e3-495f-870c-925863b09ce1" containerName="istio-proxy"
Oct 30 21:25:13.875460 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:13.875401    5713 cpu_manager.go:414] "containerMap: removing container" podUID="4d34434b-cc8a-4ee2-a774-cacdea5465de" containerName="istio-proxy"
Oct 30 21:25:13.875460 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:13.875409    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d34434b-cc8a-4ee2-a774-cacdea5465de" containerName="istio-proxy"
Oct 30 21:25:13.875460 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:13.875423    5713 cpu_manager.go:414] "containerMap: removing container" podUID="4d34434b-cc8a-4ee2-a774-cacdea5465de" containerName="istio-proxy"
Oct 30 21:25:13.875460 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:13.875448    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d34434b-cc8a-4ee2-a774-cacdea5465de" containerName="istio-proxy"
Oct 30 21:25:13.875884 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:13.875484    5713 cpu_manager.go:414] "containerMap: removing container" podUID="230f6842-289d-43fd-b1b7-849c27c5f97b" containerName="payload-pre-processing"
Oct 30 21:25:13.875884 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:13.875493    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="230f6842-289d-43fd-b1b7-849c27c5f97b" containerName="payload-pre-processing"
Oct 30 21:25:13.875884 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:13.875508    5713 cpu_manager.go:414] "containerMap: removing container" podUID="b3dc86ed-d40b-43f4-b403-7c7840462545" containerName="revoke-keys"
Oct 30 21:25:13.875884 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:13.875516    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3dc86ed-d40b-43f4-b403-7c7840462545" containerName="revoke-keys"
Oct 30 21:25:13.875884 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:13.875643    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e54eff2-416c-4a97-8ce1-2ea30c7e982a" containerName="istio-proxy"
Oct 30 21:25:13.875884 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:13.875661    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="21edc9eb-88e3-495f-870c-925863b09ce1" containerName="istio-proxy"
Oct 30 21:25:13.875884 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:13.875670    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d34434b-cc8a-4ee2-a774-cacdea5465de" containerName="istio-proxy"
Oct 30 21:25:13.875884 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:13.875680    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d34434b-cc8a-4ee2-a774-cacdea5465de" containerName="istio-proxy"
Oct 30 21:25:13.875884 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:13.875693    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="230f6842-289d-43fd-b1b7-849c27c5f97b" containerName="payload-pre-processing"
Oct 30 21:25:13.875884 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:13.875703    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0dff592-d5a1-4832-8104-46cc5fd973c5" containerName="istio-proxy"
Oct 30 21:25:13.875884 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:13.875729    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3dc86ed-d40b-43f4-b403-7c7840462545" containerName="revoke-keys"
Oct 30 21:25:13.881174 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:13.881153    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1be5211-l9qt4"
Oct 30 21:25:13.919645 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:13.919611    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-664cc788-rdbnn"]
Oct 30 21:25:13.928444 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:13.928412    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-664cc788-rdbnn"
Oct 30 21:25:13.935170 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:13.935141    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1be5211-l9qt4"]
Oct 30 21:25:13.967018 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:13.966985    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:25:13.967018 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:13.966987    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:25:13.967240 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:13.967031    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:25:13.980985 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:13.980955    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:25:13.984504 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:13.984476    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-664cc788-rdbnn"]
Oct 30 21:25:14.051291 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:14.051260    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/403b83a9-0bc8-4dc5-8053-139663aea455-maas-api-xks-ca\") pod \"maas-api-revoke-sub-664cc788-rdbnn\" (UID: \"403b83a9-0bc8-4dc5-8053-139663aea455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-664cc788-rdbnn"
Oct 30 21:25:14.051481 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:14.051303    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl8m4\" (UniqueName: \"kubernetes.io/projected/4e5ad572-be75-4519-a23b-32004eeab2b7-kube-api-access-wl8m4\") pod \"maas-api-revoke-sub-d1be5211-l9qt4\" (UID: \"4e5ad572-be75-4519-a23b-32004eeab2b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1be5211-l9qt4"
Oct 30 21:25:14.051481 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:14.051338    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/403b83a9-0bc8-4dc5-8053-139663aea455-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-664cc788-rdbnn\" (UID: \"403b83a9-0bc8-4dc5-8053-139663aea455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-664cc788-rdbnn"
Oct 30 21:25:14.051481 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:14.051473    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e5ad572-be75-4519-a23b-32004eeab2b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1be5211-l9qt4\" (UID: \"4e5ad572-be75-4519-a23b-32004eeab2b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1be5211-l9qt4"
Oct 30 21:25:14.051584 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:14.051523    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7t59\" (UniqueName: \"kubernetes.io/projected/403b83a9-0bc8-4dc5-8053-139663aea455-kube-api-access-p7t59\") pod \"maas-api-revoke-sub-664cc788-rdbnn\" (UID: \"403b83a9-0bc8-4dc5-8053-139663aea455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-664cc788-rdbnn"
Oct 30 21:25:14.051638 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:14.051614    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e5ad572-be75-4519-a23b-32004eeab2b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1be5211-l9qt4\" (UID: \"4e5ad572-be75-4519-a23b-32004eeab2b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1be5211-l9qt4"
Oct 30 21:25:14.152532 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:14.152438    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e5ad572-be75-4519-a23b-32004eeab2b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1be5211-l9qt4\" (UID: \"4e5ad572-be75-4519-a23b-32004eeab2b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1be5211-l9qt4"
Oct 30 21:25:14.152532 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:14.152495    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7t59\" (UniqueName: \"kubernetes.io/projected/403b83a9-0bc8-4dc5-8053-139663aea455-kube-api-access-p7t59\") pod \"maas-api-revoke-sub-664cc788-rdbnn\" (UID: \"403b83a9-0bc8-4dc5-8053-139663aea455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-664cc788-rdbnn"
Oct 30 21:25:14.152532 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:14.152532    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e5ad572-be75-4519-a23b-32004eeab2b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1be5211-l9qt4\" (UID: \"4e5ad572-be75-4519-a23b-32004eeab2b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1be5211-l9qt4"
Oct 30 21:25:14.153230 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:14.152586    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/403b83a9-0bc8-4dc5-8053-139663aea455-maas-api-xks-ca\") pod \"maas-api-revoke-sub-664cc788-rdbnn\" (UID: \"403b83a9-0bc8-4dc5-8053-139663aea455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-664cc788-rdbnn"
Oct 30 21:25:14.153230 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:14.152626    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl8m4\" (UniqueName: \"kubernetes.io/projected/4e5ad572-be75-4519-a23b-32004eeab2b7-kube-api-access-wl8m4\") pod \"maas-api-revoke-sub-d1be5211-l9qt4\" (UID: \"4e5ad572-be75-4519-a23b-32004eeab2b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1be5211-l9qt4"
Oct 30 21:25:14.153230 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:14.152676    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/403b83a9-0bc8-4dc5-8053-139663aea455-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-664cc788-rdbnn\" (UID: \"403b83a9-0bc8-4dc5-8053-139663aea455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-664cc788-rdbnn"
Oct 30 21:25:14.153573 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:14.153256    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e5ad572-be75-4519-a23b-32004eeab2b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1be5211-l9qt4\" (UID: \"4e5ad572-be75-4519-a23b-32004eeab2b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1be5211-l9qt4"
Oct 30 21:25:14.153573 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:14.153340    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/403b83a9-0bc8-4dc5-8053-139663aea455-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-664cc788-rdbnn\" (UID: \"403b83a9-0bc8-4dc5-8053-139663aea455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-664cc788-rdbnn"
Oct 30 21:25:14.155282 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:14.155259    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e5ad572-be75-4519-a23b-32004eeab2b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1be5211-l9qt4\" (UID: \"4e5ad572-be75-4519-a23b-32004eeab2b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1be5211-l9qt4"
Oct 30 21:25:14.155423 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:14.155396    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/403b83a9-0bc8-4dc5-8053-139663aea455-maas-api-xks-ca\") pod \"maas-api-revoke-sub-664cc788-rdbnn\" (UID: \"403b83a9-0bc8-4dc5-8053-139663aea455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-664cc788-rdbnn"
Oct 30 21:25:14.209259 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:14.209220    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7t59\" (UniqueName: \"kubernetes.io/projected/403b83a9-0bc8-4dc5-8053-139663aea455-kube-api-access-p7t59\") pod \"maas-api-revoke-sub-664cc788-rdbnn\" (UID: \"403b83a9-0bc8-4dc5-8053-139663aea455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-664cc788-rdbnn"
Oct 30 21:25:14.209421 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:14.209323    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl8m4\" (UniqueName: \"kubernetes.io/projected/4e5ad572-be75-4519-a23b-32004eeab2b7-kube-api-access-wl8m4\") pod \"maas-api-revoke-sub-d1be5211-l9qt4\" (UID: \"4e5ad572-be75-4519-a23b-32004eeab2b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1be5211-l9qt4"
Oct 30 21:25:14.248270 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:14.248234    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-664cc788-rdbnn"
Oct 30 21:25:14.495894 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:14.495856    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1be5211-l9qt4"
Oct 30 21:25:14.626471 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:14.626440    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-664cc788-rdbnn"]
Oct 30 21:25:14.628164 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:25:14.628131    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod403b83a9_0bc8_4dc5_8053_139663aea455.slice/crio-3f8f58115f85b4671e545471e12d1bcf71414bd0d88a7316b33eafcd865866ff WatchSource:0}: Error finding container 3f8f58115f85b4671e545471e12d1bcf71414bd0d88a7316b33eafcd865866ff: Status 404 returned error can't find the container with id 3f8f58115f85b4671e545471e12d1bcf71414bd0d88a7316b33eafcd865866ff
Oct 30 21:25:14.872902 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:14.872875    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1be5211-l9qt4"]
Oct 30 21:25:14.936779 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:25:14.936738    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e5ad572_be75_4519_a23b_32004eeab2b7.slice/crio-027f7a822c1b8dbe4f1d7105343fe98009541f4ca24e4c3314f7f3168fb08881 WatchSource:0}: Error finding container 027f7a822c1b8dbe4f1d7105343fe98009541f4ca24e4c3314f7f3168fb08881: Status 404 returned error can't find the container with id 027f7a822c1b8dbe4f1d7105343fe98009541f4ca24e4c3314f7f3168fb08881
Oct 30 21:25:15.339334 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:15.339291    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1be5211-l9qt4" event={"ID":"4e5ad572-be75-4519-a23b-32004eeab2b7","Type":"ContainerStarted","Data":"264ea12b3678a7471e360a66aa272a2307e5545291b99e20dd12eef5cb93dffd"}
Oct 30 21:25:15.339844 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:15.339354    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1be5211-l9qt4" event={"ID":"4e5ad572-be75-4519-a23b-32004eeab2b7","Type":"ContainerStarted","Data":"027f7a822c1b8dbe4f1d7105343fe98009541f4ca24e4c3314f7f3168fb08881"}
Oct 30 21:25:15.341155 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:15.341125    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-664cc788-rdbnn" event={"ID":"403b83a9-0bc8-4dc5-8053-139663aea455","Type":"ContainerStarted","Data":"2a4295411daee5452740a8de8bdf52430577ba59f56c54be015cac966f71c334"}
Oct 30 21:25:15.341273 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:15.341175    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-664cc788-rdbnn" event={"ID":"403b83a9-0bc8-4dc5-8053-139663aea455","Type":"ContainerStarted","Data":"3f8f58115f85b4671e545471e12d1bcf71414bd0d88a7316b33eafcd865866ff"}
Oct 30 21:25:15.452821 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:15.452740    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1be5211-l9qt4" podStartSLOduration=2.452726598 podStartE2EDuration="2.452726598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:25:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:25:15.452193334 +0000 UTC m=+2798.516969734" watchObservedRunningTime="2026-10-30 21:25:15.452726598 +0000 UTC m=+2798.517502995"
Oct 30 21:25:15.566672 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:15.566601    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-664cc788-rdbnn" podStartSLOduration=2.566586323 podStartE2EDuration="2.566586323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:25:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:25:15.563619744 +0000 UTC m=+2798.628396139" watchObservedRunningTime="2026-10-30 21:25:15.566586323 +0000 UTC m=+2798.631362733"
Oct 30 21:25:16.220302 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:16.220261    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-639b07-8zrvs"]
Oct 30 21:25:16.224251 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:16.224226    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-639b07-8zrvs"]
Oct 30 21:25:16.352759 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:16.352720    5713 generic.go:356] "Generic (PLEG): container finished" podID="4e5ad572-be75-4519-a23b-32004eeab2b7" containerID="264ea12b3678a7471e360a66aa272a2307e5545291b99e20dd12eef5cb93dffd" exitCode=0
Oct 30 21:25:16.353248 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:16.352833    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1be5211-l9qt4" event={"ID":"4e5ad572-be75-4519-a23b-32004eeab2b7","Type":"ContainerDied","Data":"264ea12b3678a7471e360a66aa272a2307e5545291b99e20dd12eef5cb93dffd"}
Oct 30 21:25:16.354335 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:16.354307    5713 generic.go:356] "Generic (PLEG): container finished" podID="403b83a9-0bc8-4dc5-8053-139663aea455" containerID="2a4295411daee5452740a8de8bdf52430577ba59f56c54be015cac966f71c334" exitCode=0
Oct 30 21:25:16.354465 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:16.354385    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-664cc788-rdbnn" event={"ID":"403b83a9-0bc8-4dc5-8053-139663aea455","Type":"ContainerDied","Data":"2a4295411daee5452740a8de8bdf52430577ba59f56c54be015cac966f71c334"}
Oct 30 21:25:17.062942 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:17.062908    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b559e33d-9df3-49f7-a9f3-b97c1c79c0e9" path="/var/lib/kubelet/pods/b559e33d-9df3-49f7-a9f3-b97c1c79c0e9/volumes"
Oct 30 21:25:17.504456 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:17.504421    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-664cc788-rdbnn"
Oct 30 21:25:17.544855 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:17.544828    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1be5211-l9qt4"
Oct 30 21:25:17.692387 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:17.692356    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/403b83a9-0bc8-4dc5-8053-139663aea455-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/403b83a9-0bc8-4dc5-8053-139663aea455-maas-api-openshift-ca\") pod \"403b83a9-0bc8-4dc5-8053-139663aea455\" (UID: \"403b83a9-0bc8-4dc5-8053-139663aea455\") "
Oct 30 21:25:17.692609 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:17.692410    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e5ad572-be75-4519-a23b-32004eeab2b7-kube-api-access-wl8m4\" (UniqueName: \"kubernetes.io/projected/4e5ad572-be75-4519-a23b-32004eeab2b7-kube-api-access-wl8m4\") pod \"4e5ad572-be75-4519-a23b-32004eeab2b7\" (UID: \"4e5ad572-be75-4519-a23b-32004eeab2b7\") "
Oct 30 21:25:17.692609 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:17.692438    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e5ad572-be75-4519-a23b-32004eeab2b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e5ad572-be75-4519-a23b-32004eeab2b7-maas-api-xks-ca\") pod \"4e5ad572-be75-4519-a23b-32004eeab2b7\" (UID: \"4e5ad572-be75-4519-a23b-32004eeab2b7\") "
Oct 30 21:25:17.692609 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:17.692473    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/403b83a9-0bc8-4dc5-8053-139663aea455-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/403b83a9-0bc8-4dc5-8053-139663aea455-maas-api-xks-ca\") pod \"403b83a9-0bc8-4dc5-8053-139663aea455\" (UID: \"403b83a9-0bc8-4dc5-8053-139663aea455\") "
Oct 30 21:25:17.692609 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:17.692516    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e5ad572-be75-4519-a23b-32004eeab2b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e5ad572-be75-4519-a23b-32004eeab2b7-maas-api-openshift-ca\") pod \"4e5ad572-be75-4519-a23b-32004eeab2b7\" (UID: \"4e5ad572-be75-4519-a23b-32004eeab2b7\") "
Oct 30 21:25:17.692609 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:17.692601    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/403b83a9-0bc8-4dc5-8053-139663aea455-kube-api-access-p7t59\" (UniqueName: \"kubernetes.io/projected/403b83a9-0bc8-4dc5-8053-139663aea455-kube-api-access-p7t59\") pod \"403b83a9-0bc8-4dc5-8053-139663aea455\" (UID: \"403b83a9-0bc8-4dc5-8053-139663aea455\") "
Oct 30 21:25:17.692954 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:17.692917    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/403b83a9-0bc8-4dc5-8053-139663aea455-maas-api-openshift-ca" pod "403b83a9-0bc8-4dc5-8053-139663aea455" (UID: "403b83a9-0bc8-4dc5-8053-139663aea455"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:25:17.692954 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:17.692945    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e5ad572-be75-4519-a23b-32004eeab2b7-maas-api-openshift-ca" pod "4e5ad572-be75-4519-a23b-32004eeab2b7" (UID: "4e5ad572-be75-4519-a23b-32004eeab2b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:25:17.693121 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:17.693100    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/403b83a9-0bc8-4dc5-8053-139663aea455-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:25:17.693197 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:17.693123    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e5ad572-be75-4519-a23b-32004eeab2b7-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:25:17.694989 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:17.694964    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e5ad572-be75-4519-a23b-32004eeab2b7-maas-api-xks-ca" pod "4e5ad572-be75-4519-a23b-32004eeab2b7" (UID: "4e5ad572-be75-4519-a23b-32004eeab2b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:25:17.695115 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:17.695020    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e5ad572-be75-4519-a23b-32004eeab2b7-kube-api-access-wl8m4" pod "4e5ad572-be75-4519-a23b-32004eeab2b7" (UID: "4e5ad572-be75-4519-a23b-32004eeab2b7"). InnerVolumeSpecName "kube-api-access-wl8m4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:17.695291 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:17.695273    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/403b83a9-0bc8-4dc5-8053-139663aea455-kube-api-access-p7t59" pod "403b83a9-0bc8-4dc5-8053-139663aea455" (UID: "403b83a9-0bc8-4dc5-8053-139663aea455"). InnerVolumeSpecName "kube-api-access-p7t59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:17.695333 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:17.695293    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/403b83a9-0bc8-4dc5-8053-139663aea455-maas-api-xks-ca" pod "403b83a9-0bc8-4dc5-8053-139663aea455" (UID: "403b83a9-0bc8-4dc5-8053-139663aea455"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:25:17.793737 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:17.793696    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7t59\" (UniqueName: \"kubernetes.io/projected/403b83a9-0bc8-4dc5-8053-139663aea455-kube-api-access-p7t59\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:25:17.793737 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:17.793729    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wl8m4\" (UniqueName: \"kubernetes.io/projected/4e5ad572-be75-4519-a23b-32004eeab2b7-kube-api-access-wl8m4\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:25:17.793737 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:17.793740    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e5ad572-be75-4519-a23b-32004eeab2b7-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:25:17.793737 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:17.793749    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/403b83a9-0bc8-4dc5-8053-139663aea455-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:25:18.334422 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:18.334386    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-8fa2a7-977ff4795-mgd5h"]
Oct 30 21:25:18.369717 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:18.369689    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1be5211-l9qt4"
Oct 30 21:25:18.369910 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:18.369683    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1be5211-l9qt4" event={"ID":"4e5ad572-be75-4519-a23b-32004eeab2b7","Type":"ContainerDied","Data":"027f7a822c1b8dbe4f1d7105343fe98009541f4ca24e4c3314f7f3168fb08881"}
Oct 30 21:25:18.369910 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:18.369848    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="027f7a822c1b8dbe4f1d7105343fe98009541f4ca24e4c3314f7f3168fb08881"
Oct 30 21:25:18.371219 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:18.371190    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-664cc788-rdbnn" event={"ID":"403b83a9-0bc8-4dc5-8053-139663aea455","Type":"ContainerDied","Data":"3f8f58115f85b4671e545471e12d1bcf71414bd0d88a7316b33eafcd865866ff"}
Oct 30 21:25:18.371353 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:18.371244    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f8f58115f85b4671e545471e12d1bcf71414bd0d88a7316b33eafcd865866ff"
Oct 30 21:25:18.371353 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:18.371197    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-664cc788-rdbnn"
Oct 30 21:25:23.335333 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:23.335263    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-8fa2a7-977ff4795-mgd5h" podUID="0fa5be36-c8de-4cdc-87d9-ebf11115f0c3" containerName="payload-pre-processing" containerID="cri-o://c8860b676c5420e7791c6545f94fc26e9a5ebcd3b4cad569dcfc5d92bd9b6d4d" gracePeriod=25
Oct 30 21:25:23.768866 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:23.768827    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-5c45e5-bfsnk"]
Oct 30 21:25:23.769306 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:23.769292    5713 cpu_manager.go:414] "containerMap: removing container" podUID="403b83a9-0bc8-4dc5-8053-139663aea455" containerName="revoke-keys"
Oct 30 21:25:23.769364 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:23.769308    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="403b83a9-0bc8-4dc5-8053-139663aea455" containerName="revoke-keys"
Oct 30 21:25:23.769364 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:23.769345    5713 cpu_manager.go:414] "containerMap: removing container" podUID="4e5ad572-be75-4519-a23b-32004eeab2b7" containerName="revoke-keys"
Oct 30 21:25:23.769364 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:23.769351    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e5ad572-be75-4519-a23b-32004eeab2b7" containerName="revoke-keys"
Oct 30 21:25:23.769462 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:23.769420    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e5ad572-be75-4519-a23b-32004eeab2b7" containerName="revoke-keys"
Oct 30 21:25:23.769462 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:23.769434    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="403b83a9-0bc8-4dc5-8053-139663aea455" containerName="revoke-keys"
Oct 30 21:25:23.772941 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:23.772921    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-5c45e5-bfsnk"
Oct 30 21:25:23.830295 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:23.830265    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-5c45e5-bfsnk"]
Oct 30 21:25:23.869028 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:23.868994    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dd683f0-d100-4c2b-b47c-84f211e6f239-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-5c45e5-bfsnk\" (UID: \"3dd683f0-d100-4c2b-b47c-84f211e6f239\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-5c45e5-bfsnk"
Oct 30 21:25:23.869258 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:23.869043    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dd683f0-d100-4c2b-b47c-84f211e6f239-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-5c45e5-bfsnk\" (UID: \"3dd683f0-d100-4c2b-b47c-84f211e6f239\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-5c45e5-bfsnk"
Oct 30 21:25:23.869258 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:23.869145    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgggn\" (UniqueName: \"kubernetes.io/projected/3dd683f0-d100-4c2b-b47c-84f211e6f239-kube-api-access-kgggn\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-5c45e5-bfsnk\" (UID: \"3dd683f0-d100-4c2b-b47c-84f211e6f239\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-5c45e5-bfsnk"
Oct 30 21:25:23.876153 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:23.876125    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:25:23.876153 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:23.876137    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:25:23.876315 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:23.876165    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:25:23.876315 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:23.876200    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:25:23.969900 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:23.969870    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgggn\" (UniqueName: \"kubernetes.io/projected/3dd683f0-d100-4c2b-b47c-84f211e6f239-kube-api-access-kgggn\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-5c45e5-bfsnk\" (UID: \"3dd683f0-d100-4c2b-b47c-84f211e6f239\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-5c45e5-bfsnk"
Oct 30 21:25:23.970081 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:23.970012    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dd683f0-d100-4c2b-b47c-84f211e6f239-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-5c45e5-bfsnk\" (UID: \"3dd683f0-d100-4c2b-b47c-84f211e6f239\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-5c45e5-bfsnk"
Oct 30 21:25:23.970081 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:23.970062    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dd683f0-d100-4c2b-b47c-84f211e6f239-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-5c45e5-bfsnk\" (UID: \"3dd683f0-d100-4c2b-b47c-84f211e6f239\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-5c45e5-bfsnk"
Oct 30 21:25:23.970679 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:23.970654    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dd683f0-d100-4c2b-b47c-84f211e6f239-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-5c45e5-bfsnk\" (UID: \"3dd683f0-d100-4c2b-b47c-84f211e6f239\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-5c45e5-bfsnk"
Oct 30 21:25:23.972494 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:23.972475    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dd683f0-d100-4c2b-b47c-84f211e6f239-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-5c45e5-bfsnk\" (UID: \"3dd683f0-d100-4c2b-b47c-84f211e6f239\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-5c45e5-bfsnk"
Oct 30 21:25:24.027592 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:24.027525    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgggn\" (UniqueName: \"kubernetes.io/projected/3dd683f0-d100-4c2b-b47c-84f211e6f239-kube-api-access-kgggn\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-5c45e5-bfsnk\" (UID: \"3dd683f0-d100-4c2b-b47c-84f211e6f239\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-5c45e5-bfsnk"
Oct 30 21:25:24.088401 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:24.088373    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-5c45e5-bfsnk"
Oct 30 21:25:24.459527 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:25:24.459463    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:25:24.463172 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:24.463145    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-5c45e5-bfsnk"]
Oct 30 21:25:24.464257 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:25:24.464233    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dd683f0_d100_4c2b_b47c_84f211e6f239.slice/crio-a7c0de057cac02ec713d73038a920250670972a479857440188b18d982bd10f5 WatchSource:0}: Error finding container a7c0de057cac02ec713d73038a920250670972a479857440188b18d982bd10f5: Status 404 returned error can't find the container with id a7c0de057cac02ec713d73038a920250670972a479857440188b18d982bd10f5
Oct 30 21:25:25.418419 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:25.418380    5713 generic.go:356] "Generic (PLEG): container finished" podID="3dd683f0-d100-4c2b-b47c-84f211e6f239" containerID="6bb0dabed4de25bc5ca696d1fb589828ca4f6bf9a2f8558ff04c0247a34d94da" exitCode=0
Oct 30 21:25:25.418570 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:25.418470    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-5c45e5-bfsnk" event={"ID":"3dd683f0-d100-4c2b-b47c-84f211e6f239","Type":"ContainerDied","Data":"6bb0dabed4de25bc5ca696d1fb589828ca4f6bf9a2f8558ff04c0247a34d94da"}
Oct 30 21:25:25.418570 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:25.418525    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-5c45e5-bfsnk" event={"ID":"3dd683f0-d100-4c2b-b47c-84f211e6f239","Type":"ContainerStarted","Data":"a7c0de057cac02ec713d73038a920250670972a479857440188b18d982bd10f5"}
Oct 30 21:25:26.158850 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:26.158807    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm" podUID="cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776" containerName="main" containerID="cri-o://317fcab0854178e48daa348f091f4a7abaf1653ccf67bfae69aa14db368026ad" gracePeriod=45
Oct 30 21:25:26.159335 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:26.159075    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7" podUID="b2193213-ebae-498a-911a-6056a8e94312" containerName="main" containerID="cri-o://248aa9b531f637cfc83c8a4e0738ad785d6d0518394e7efbe5223b0eee488774" gracePeriod=45
Oct 30 21:25:26.429573 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:26.429406    5713 generic.go:356] "Generic (PLEG): container finished" podID="b2193213-ebae-498a-911a-6056a8e94312" containerID="248aa9b531f637cfc83c8a4e0738ad785d6d0518394e7efbe5223b0eee488774" exitCode=0
Oct 30 21:25:26.429573 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:26.429458    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7" event={"ID":"b2193213-ebae-498a-911a-6056a8e94312","Type":"ContainerDied","Data":"248aa9b531f637cfc83c8a4e0738ad785d6d0518394e7efbe5223b0eee488774"}
Oct 30 21:25:26.429573 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:26.429533    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7" event={"ID":"b2193213-ebae-498a-911a-6056a8e94312","Type":"ContainerDied","Data":"5ebde4ea2bb0553729b57d8e72f5fcc2bdb6243621d3523a48b108d76e542dba"}
Oct 30 21:25:26.429573 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:26.429547    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ebde4ea2bb0553729b57d8e72f5fcc2bdb6243621d3523a48b108d76e542dba"
Oct 30 21:25:26.431314 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:26.431288    5713 generic.go:356] "Generic (PLEG): container finished" podID="cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776" containerID="317fcab0854178e48daa348f091f4a7abaf1653ccf67bfae69aa14db368026ad" exitCode=0
Oct 30 21:25:26.431431 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:26.431355    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm" event={"ID":"cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776","Type":"ContainerDied","Data":"317fcab0854178e48daa348f091f4a7abaf1653ccf67bfae69aa14db368026ad"}
Oct 30 21:25:26.431431 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:26.431392    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm" event={"ID":"cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776","Type":"ContainerDied","Data":"61b16e2cd5ab53974a9be6a718148498fe26781fe2280858aa3c6f0886f82af1"}
Oct 30 21:25:26.431431 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:26.431405    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61b16e2cd5ab53974a9be6a718148498fe26781fe2280858aa3c6f0886f82af1"
Oct 30 21:25:26.433401 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:26.433378    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm"
Oct 30 21:25:26.437246 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:26.437226    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7"
Oct 30 21:25:26.498293 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:26.498263    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776-tls-certs\" (UniqueName: \"kubernetes.io/secret/cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776-tls-certs\") pod \"cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776\" (UID: \"cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776\") "
Oct 30 21:25:26.498492 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:26.498314    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2193213-ebae-498a-911a-6056a8e94312-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2193213-ebae-498a-911a-6056a8e94312-tmp-dir\") pod \"b2193213-ebae-498a-911a-6056a8e94312\" (UID: \"b2193213-ebae-498a-911a-6056a8e94312\") "
Oct 30 21:25:26.498492 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:26.498350    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2193213-ebae-498a-911a-6056a8e94312-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2193213-ebae-498a-911a-6056a8e94312-model-cache\") pod \"b2193213-ebae-498a-911a-6056a8e94312\" (UID: \"b2193213-ebae-498a-911a-6056a8e94312\") "
Oct 30 21:25:26.498492 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:26.498377    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776-dshm\") pod \"cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776\" (UID: \"cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776\") "
Oct 30 21:25:26.498492 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:26.498409    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776-kube-api-access-r7j6k\" (UniqueName: \"kubernetes.io/projected/cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776-kube-api-access-r7j6k\") pod \"cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776\" (UID: \"cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776\") "
Oct 30 21:25:26.498492 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:26.498432    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2193213-ebae-498a-911a-6056a8e94312-home\" (UniqueName: \"kubernetes.io/empty-dir/b2193213-ebae-498a-911a-6056a8e94312-home\") pod \"b2193213-ebae-498a-911a-6056a8e94312\" (UID: \"b2193213-ebae-498a-911a-6056a8e94312\") "
Oct 30 21:25:26.498492 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:26.498453    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2193213-ebae-498a-911a-6056a8e94312-tls-certs\" (UniqueName: \"kubernetes.io/secret/b2193213-ebae-498a-911a-6056a8e94312-tls-certs\") pod \"b2193213-ebae-498a-911a-6056a8e94312\" (UID: \"b2193213-ebae-498a-911a-6056a8e94312\") "
Oct 30 21:25:26.498850 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:26.498500    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776-model-cache\") pod \"cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776\" (UID: \"cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776\") "
Oct 30 21:25:26.498850 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:26.498524    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776-tmp-dir\") pod \"cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776\" (UID: \"cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776\") "
Oct 30 21:25:26.498850 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:26.498569    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2193213-ebae-498a-911a-6056a8e94312-kube-api-access-qlncq\" (UniqueName: \"kubernetes.io/projected/b2193213-ebae-498a-911a-6056a8e94312-kube-api-access-qlncq\") pod \"b2193213-ebae-498a-911a-6056a8e94312\" (UID: \"b2193213-ebae-498a-911a-6056a8e94312\") "
Oct 30 21:25:26.498850 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:26.498619    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2193213-ebae-498a-911a-6056a8e94312-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2193213-ebae-498a-911a-6056a8e94312-dshm\") pod \"b2193213-ebae-498a-911a-6056a8e94312\" (UID: \"b2193213-ebae-498a-911a-6056a8e94312\") "
Oct 30 21:25:26.498850 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:26.498717    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776-home\" (UniqueName: \"kubernetes.io/empty-dir/cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776-home\") pod \"cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776\" (UID: \"cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776\") "
Oct 30 21:25:26.499282 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:26.498915    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776-model-cache" pod "cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776" (UID: "cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:25:26.499282 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:26.499229    5713 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776-model-cache\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:25:26.499400 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:26.499290    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2193213-ebae-498a-911a-6056a8e94312-home" pod "b2193213-ebae-498a-911a-6056a8e94312" (UID: "b2193213-ebae-498a-911a-6056a8e94312"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:25:26.499459 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:26.499428    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2193213-ebae-498a-911a-6056a8e94312-model-cache" pod "b2193213-ebae-498a-911a-6056a8e94312" (UID: "b2193213-ebae-498a-911a-6056a8e94312"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:25:26.499459 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:26.499441    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776-home" pod "cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776" (UID: "cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:25:26.499906 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:26.499867    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776-tmp-dir" pod "cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776" (UID: "cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:25:26.500181 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:26.500151    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2193213-ebae-498a-911a-6056a8e94312-tmp-dir" pod "b2193213-ebae-498a-911a-6056a8e94312" (UID: "b2193213-ebae-498a-911a-6056a8e94312"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:25:26.502389 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:26.502187    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776-tls-certs" pod "cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776" (UID: "cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:25:26.502389 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:26.502192    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2193213-ebae-498a-911a-6056a8e94312-tls-certs" pod "b2193213-ebae-498a-911a-6056a8e94312" (UID: "b2193213-ebae-498a-911a-6056a8e94312"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:25:26.502389 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:26.502263    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776-dshm" pod "cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776" (UID: "cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:25:26.502389 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:26.502359    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776-kube-api-access-r7j6k" pod "cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776" (UID: "cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776"). InnerVolumeSpecName "kube-api-access-r7j6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:26.502683 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:26.502472    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2193213-ebae-498a-911a-6056a8e94312-dshm" pod "b2193213-ebae-498a-911a-6056a8e94312" (UID: "b2193213-ebae-498a-911a-6056a8e94312"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:25:26.503670 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:26.503642    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2193213-ebae-498a-911a-6056a8e94312-kube-api-access-qlncq" pod "b2193213-ebae-498a-911a-6056a8e94312" (UID: "b2193213-ebae-498a-911a-6056a8e94312"). InnerVolumeSpecName "kube-api-access-qlncq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:26.578530 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:26.578504    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-5c45e5-bfsnk"
Oct 30 21:25:26.599831 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:26.599782    5713 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776-home\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:25:26.599831 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:26.599829    5713 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776-tls-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:25:26.600098 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:26.599841    5713 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2193213-ebae-498a-911a-6056a8e94312-tmp-dir\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:25:26.600098 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:26.599857    5713 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2193213-ebae-498a-911a-6056a8e94312-model-cache\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:25:26.600098 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:26.599867    5713 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776-dshm\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:25:26.600098 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:26.599878    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7j6k\" (UniqueName: \"kubernetes.io/projected/cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776-kube-api-access-r7j6k\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:25:26.600098 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:26.599891    5713 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2193213-ebae-498a-911a-6056a8e94312-home\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:25:26.600098 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:26.599899    5713 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2193213-ebae-498a-911a-6056a8e94312-tls-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:25:26.600098 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:26.599907    5713 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776-tmp-dir\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:25:26.600098 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:26.599915    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlncq\" (UniqueName: \"kubernetes.io/projected/b2193213-ebae-498a-911a-6056a8e94312-kube-api-access-qlncq\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:25:26.600098 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:26.599927    5713 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2193213-ebae-498a-911a-6056a8e94312-dshm\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:25:26.700310 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:26.700274    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3dd683f0-d100-4c2b-b47c-84f211e6f239-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dd683f0-d100-4c2b-b47c-84f211e6f239-maas-api-xks-ca\") pod \"3dd683f0-d100-4c2b-b47c-84f211e6f239\" (UID: \"3dd683f0-d100-4c2b-b47c-84f211e6f239\") "
Oct 30 21:25:26.700523 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:26.700411    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dd683f0-d100-4c2b-b47c-84f211e6f239-kube-api-access-kgggn\" (UniqueName: \"kubernetes.io/projected/3dd683f0-d100-4c2b-b47c-84f211e6f239-kube-api-access-kgggn\") pod \"3dd683f0-d100-4c2b-b47c-84f211e6f239\" (UID: \"3dd683f0-d100-4c2b-b47c-84f211e6f239\") "
Oct 30 21:25:26.700523 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:26.700435    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dd683f0-d100-4c2b-b47c-84f211e6f239-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dd683f0-d100-4c2b-b47c-84f211e6f239-maas-api-openshift-ca\") pod \"3dd683f0-d100-4c2b-b47c-84f211e6f239\" (UID: \"3dd683f0-d100-4c2b-b47c-84f211e6f239\") "
Oct 30 21:25:26.700959 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:26.700924    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dd683f0-d100-4c2b-b47c-84f211e6f239-maas-api-openshift-ca" pod "3dd683f0-d100-4c2b-b47c-84f211e6f239" (UID: "3dd683f0-d100-4c2b-b47c-84f211e6f239"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:25:26.702646 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:26.702619    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dd683f0-d100-4c2b-b47c-84f211e6f239-maas-api-xks-ca" pod "3dd683f0-d100-4c2b-b47c-84f211e6f239" (UID: "3dd683f0-d100-4c2b-b47c-84f211e6f239"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:25:26.702721 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:26.702687    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dd683f0-d100-4c2b-b47c-84f211e6f239-kube-api-access-kgggn" pod "3dd683f0-d100-4c2b-b47c-84f211e6f239" (UID: "3dd683f0-d100-4c2b-b47c-84f211e6f239"). InnerVolumeSpecName "kube-api-access-kgggn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:26.801535 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:26.801501    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgggn\" (UniqueName: \"kubernetes.io/projected/3dd683f0-d100-4c2b-b47c-84f211e6f239-kube-api-access-kgggn\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:25:26.801535 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:26.801532    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dd683f0-d100-4c2b-b47c-84f211e6f239-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:25:26.801535 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:26.801542    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dd683f0-d100-4c2b-b47c-84f211e6f239-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:25:27.439582 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:27.439550    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-5c45e5-bfsnk"
Oct 30 21:25:27.440001 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:27.439576    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-5c45e5-bfsnk" event={"ID":"3dd683f0-d100-4c2b-b47c-84f211e6f239","Type":"ContainerDied","Data":"a7c0de057cac02ec713d73038a920250670972a479857440188b18d982bd10f5"}
Oct 30 21:25:27.440001 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:27.439634    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7c0de057cac02ec713d73038a920250670972a479857440188b18d982bd10f5"
Oct 30 21:25:27.440001 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:27.439713    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7"
Oct 30 21:25:27.440001 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:27.439718    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-8fa2a7/distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm"
Oct 30 21:25:27.656731 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:27.656693    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-8fa2a7/distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm"]
Oct 30 21:25:27.706341 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:27.706256    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-8fa2a7/distinct-2-w6-8fa2a7-kserve-759556bb77-99dsm"]
Oct 30 21:25:27.821912 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:27.821881    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-8fa2a7/distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7"]
Oct 30 21:25:27.825373 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:27.825345    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-8fa2a7/distinct-w6-8fa2a7-kserve-75d68f458b-bh4n7"]
Oct 30 21:25:29.061889 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:29.061854    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2193213-ebae-498a-911a-6056a8e94312" path="/var/lib/kubelet/pods/b2193213-ebae-498a-911a-6056a8e94312/volumes"
Oct 30 21:25:29.062299 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:29.062273    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776" path="/var/lib/kubelet/pods/cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776/volumes"
Oct 30 21:25:32.593881 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:32.593848    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-8fa2a7-977ff4795-mgd5h" podUID="0fa5be36-c8de-4cdc-87d9-ebf11115f0c3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:25:35.745613 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:35.745579    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-430f8459-rc6bx"]
Oct 30 21:25:35.746057 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:35.746040    5713 cpu_manager.go:414] "containerMap: removing container" podUID="cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776" containerName="main"
Oct 30 21:25:35.746057 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:35.746052    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776" containerName="main"
Oct 30 21:25:35.746138 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:35.746064    5713 cpu_manager.go:414] "containerMap: removing container" podUID="3dd683f0-d100-4c2b-b47c-84f211e6f239" containerName="revoke-keys"
Oct 30 21:25:35.746138 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:35.746070    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dd683f0-d100-4c2b-b47c-84f211e6f239" containerName="revoke-keys"
Oct 30 21:25:35.746138 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:35.746087    5713 cpu_manager.go:414] "containerMap: removing container" podUID="b2193213-ebae-498a-911a-6056a8e94312" containerName="main"
Oct 30 21:25:35.746138 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:35.746092    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2193213-ebae-498a-911a-6056a8e94312" containerName="main"
Oct 30 21:25:35.746272 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:35.746187    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd7dd1ea-85b8-4601-bb8e-0ceaa24f5776" containerName="main"
Oct 30 21:25:35.746272 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:35.746199    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dd683f0-d100-4c2b-b47c-84f211e6f239" containerName="revoke-keys"
Oct 30 21:25:35.746272 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:35.746205    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2193213-ebae-498a-911a-6056a8e94312" containerName="main"
Oct 30 21:25:35.749482 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:35.749465    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-430f8459-rc6bx"
Oct 30 21:25:35.801914 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:35.801881    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47b7c7b6-7fmvd"]
Oct 30 21:25:35.802905 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:35.802875    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:25:35.803230 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:35.803208    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:25:35.806018 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:35.805995    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-430f8459-rc6bx"]
Oct 30 21:25:35.806165 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:35.806090    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b7c7b6-7fmvd"
Oct 30 21:25:35.848858 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:35.848823    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:25:35.851799 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:35.851768    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:25:35.872075 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:35.872028    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47b7c7b6-7fmvd"]
Oct 30 21:25:35.896737 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:35.896705    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37c106d7-0120-4409-acc2-ef0db4a8b003-maas-api-xks-ca\") pod \"maas-api-revoke-sub-430f8459-rc6bx\" (UID: \"37c106d7-0120-4409-acc2-ef0db4a8b003\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-430f8459-rc6bx"
Oct 30 21:25:35.896915 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:35.896887    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgbhs\" (UniqueName: \"kubernetes.io/projected/37663328-2eda-4b83-bc43-c60883c8694c-kube-api-access-kgbhs\") pod \"maas-api-revoke-sub-47b7c7b6-7fmvd\" (UID: \"37663328-2eda-4b83-bc43-c60883c8694c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b7c7b6-7fmvd"
Oct 30 21:25:35.896985 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:35.896931    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37663328-2eda-4b83-bc43-c60883c8694c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47b7c7b6-7fmvd\" (UID: \"37663328-2eda-4b83-bc43-c60883c8694c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b7c7b6-7fmvd"
Oct 30 21:25:35.896985 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:35.896979    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37663328-2eda-4b83-bc43-c60883c8694c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47b7c7b6-7fmvd\" (UID: \"37663328-2eda-4b83-bc43-c60883c8694c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b7c7b6-7fmvd"
Oct 30 21:25:35.897114 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:35.897096    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37c106d7-0120-4409-acc2-ef0db4a8b003-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-430f8459-rc6bx\" (UID: \"37c106d7-0120-4409-acc2-ef0db4a8b003\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-430f8459-rc6bx"
Oct 30 21:25:35.897226 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:35.897202    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7z6q\" (UniqueName: \"kubernetes.io/projected/37c106d7-0120-4409-acc2-ef0db4a8b003-kube-api-access-k7z6q\") pod \"maas-api-revoke-sub-430f8459-rc6bx\" (UID: \"37c106d7-0120-4409-acc2-ef0db4a8b003\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-430f8459-rc6bx"
Oct 30 21:25:35.998476 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:35.998374    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7z6q\" (UniqueName: \"kubernetes.io/projected/37c106d7-0120-4409-acc2-ef0db4a8b003-kube-api-access-k7z6q\") pod \"maas-api-revoke-sub-430f8459-rc6bx\" (UID: \"37c106d7-0120-4409-acc2-ef0db4a8b003\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-430f8459-rc6bx"
Oct 30 21:25:35.998476 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:35.998421    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37c106d7-0120-4409-acc2-ef0db4a8b003-maas-api-xks-ca\") pod \"maas-api-revoke-sub-430f8459-rc6bx\" (UID: \"37c106d7-0120-4409-acc2-ef0db4a8b003\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-430f8459-rc6bx"
Oct 30 21:25:35.998666 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:35.998603    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgbhs\" (UniqueName: \"kubernetes.io/projected/37663328-2eda-4b83-bc43-c60883c8694c-kube-api-access-kgbhs\") pod \"maas-api-revoke-sub-47b7c7b6-7fmvd\" (UID: \"37663328-2eda-4b83-bc43-c60883c8694c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b7c7b6-7fmvd"
Oct 30 21:25:35.998666 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:35.998650    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37663328-2eda-4b83-bc43-c60883c8694c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47b7c7b6-7fmvd\" (UID: \"37663328-2eda-4b83-bc43-c60883c8694c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b7c7b6-7fmvd"
Oct 30 21:25:35.998761 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:35.998710    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37663328-2eda-4b83-bc43-c60883c8694c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47b7c7b6-7fmvd\" (UID: \"37663328-2eda-4b83-bc43-c60883c8694c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b7c7b6-7fmvd"
Oct 30 21:25:35.998840 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:35.998773    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37c106d7-0120-4409-acc2-ef0db4a8b003-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-430f8459-rc6bx\" (UID: \"37c106d7-0120-4409-acc2-ef0db4a8b003\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-430f8459-rc6bx"
Oct 30 21:25:35.999529 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:35.999467    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37663328-2eda-4b83-bc43-c60883c8694c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47b7c7b6-7fmvd\" (UID: \"37663328-2eda-4b83-bc43-c60883c8694c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b7c7b6-7fmvd"
Oct 30 21:25:35.999529 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:35.999470    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37c106d7-0120-4409-acc2-ef0db4a8b003-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-430f8459-rc6bx\" (UID: \"37c106d7-0120-4409-acc2-ef0db4a8b003\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-430f8459-rc6bx"
Oct 30 21:25:36.001048 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:36.001017    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37c106d7-0120-4409-acc2-ef0db4a8b003-maas-api-xks-ca\") pod \"maas-api-revoke-sub-430f8459-rc6bx\" (UID: \"37c106d7-0120-4409-acc2-ef0db4a8b003\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-430f8459-rc6bx"
Oct 30 21:25:36.001175 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:36.001125    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37663328-2eda-4b83-bc43-c60883c8694c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47b7c7b6-7fmvd\" (UID: \"37663328-2eda-4b83-bc43-c60883c8694c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b7c7b6-7fmvd"
Oct 30 21:25:36.055869 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:36.055834    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7z6q\" (UniqueName: \"kubernetes.io/projected/37c106d7-0120-4409-acc2-ef0db4a8b003-kube-api-access-k7z6q\") pod \"maas-api-revoke-sub-430f8459-rc6bx\" (UID: \"37c106d7-0120-4409-acc2-ef0db4a8b003\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-430f8459-rc6bx"
Oct 30 21:25:36.056034 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:36.056012    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgbhs\" (UniqueName: \"kubernetes.io/projected/37663328-2eda-4b83-bc43-c60883c8694c-kube-api-access-kgbhs\") pod \"maas-api-revoke-sub-47b7c7b6-7fmvd\" (UID: \"37663328-2eda-4b83-bc43-c60883c8694c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b7c7b6-7fmvd"
Oct 30 21:25:36.062115 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:36.062095    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-430f8459-rc6bx"
Oct 30 21:25:36.124443 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:36.122731    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b7c7b6-7fmvd"
Oct 30 21:25:36.441402 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:36.441378    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-430f8459-rc6bx"]
Oct 30 21:25:36.442926 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:25:36.442898    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37c106d7_0120_4409_acc2_ef0db4a8b003.slice/crio-4e6e2be47bfdade751dcc740d4e89958b0261acf4909b042fed80e3cdac6d273 WatchSource:0}: Error finding container 4e6e2be47bfdade751dcc740d4e89958b0261acf4909b042fed80e3cdac6d273: Status 404 returned error can't find the container with id 4e6e2be47bfdade751dcc740d4e89958b0261acf4909b042fed80e3cdac6d273
Oct 30 21:25:36.497477 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:36.497451    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-430f8459-rc6bx" event={"ID":"37c106d7-0120-4409-acc2-ef0db4a8b003","Type":"ContainerStarted","Data":"4e6e2be47bfdade751dcc740d4e89958b0261acf4909b042fed80e3cdac6d273"}
Oct 30 21:25:36.541605 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:36.541578    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47b7c7b6-7fmvd"]
Oct 30 21:25:36.542896 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:25:36.542871    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37663328_2eda_4b83_bc43_c60883c8694c.slice/crio-9f063bd53389eb2fc3b4cd038878c12b4458e47a76ed0e36e1eb9df091486cbf WatchSource:0}: Error finding container 9f063bd53389eb2fc3b4cd038878c12b4458e47a76ed0e36e1eb9df091486cbf: Status 404 returned error can't find the container with id 9f063bd53389eb2fc3b4cd038878c12b4458e47a76ed0e36e1eb9df091486cbf
Oct 30 21:25:37.505991 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:37.505959    5713 generic.go:356] "Generic (PLEG): container finished" podID="37663328-2eda-4b83-bc43-c60883c8694c" containerID="183ad8e0a51a68b2f3c329e7d6c44d8914dd91a92fa5cf4a4fb612ccaf0fdfcd" exitCode=0
Oct 30 21:25:37.506431 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:37.506049    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b7c7b6-7fmvd" event={"ID":"37663328-2eda-4b83-bc43-c60883c8694c","Type":"ContainerDied","Data":"183ad8e0a51a68b2f3c329e7d6c44d8914dd91a92fa5cf4a4fb612ccaf0fdfcd"}
Oct 30 21:25:37.506431 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:37.506101    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b7c7b6-7fmvd" event={"ID":"37663328-2eda-4b83-bc43-c60883c8694c","Type":"ContainerStarted","Data":"9f063bd53389eb2fc3b4cd038878c12b4458e47a76ed0e36e1eb9df091486cbf"}
Oct 30 21:25:37.507855 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:37.507816    5713 generic.go:356] "Generic (PLEG): container finished" podID="37c106d7-0120-4409-acc2-ef0db4a8b003" containerID="1c9006288c5cf8c256fbfb4d85deae72113b9d1eff97dd9f9b1981f8931bc763" exitCode=0
Oct 30 21:25:37.507994 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:37.507883    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-430f8459-rc6bx" event={"ID":"37c106d7-0120-4409-acc2-ef0db4a8b003","Type":"ContainerDied","Data":"1c9006288c5cf8c256fbfb4d85deae72113b9d1eff97dd9f9b1981f8931bc763"}
Oct 30 21:25:38.037298 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:38.037262    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-682999-gkt8n"]
Oct 30 21:25:38.039709 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:38.039680    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-682999-gkt8n"]
Oct 30 21:25:38.655367 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:38.655343    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-430f8459-rc6bx"
Oct 30 21:25:38.686766 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:38.686742    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b7c7b6-7fmvd"
Oct 30 21:25:38.728314 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:38.728284    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37c106d7-0120-4409-acc2-ef0db4a8b003-kube-api-access-k7z6q\" (UniqueName: \"kubernetes.io/projected/37c106d7-0120-4409-acc2-ef0db4a8b003-kube-api-access-k7z6q\") pod \"37c106d7-0120-4409-acc2-ef0db4a8b003\" (UID: \"37c106d7-0120-4409-acc2-ef0db4a8b003\") "
Oct 30 21:25:38.728523 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:38.728396    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37c106d7-0120-4409-acc2-ef0db4a8b003-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37c106d7-0120-4409-acc2-ef0db4a8b003-maas-api-openshift-ca\") pod \"37c106d7-0120-4409-acc2-ef0db4a8b003\" (UID: \"37c106d7-0120-4409-acc2-ef0db4a8b003\") "
Oct 30 21:25:38.728523 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:38.728420    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37c106d7-0120-4409-acc2-ef0db4a8b003-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37c106d7-0120-4409-acc2-ef0db4a8b003-maas-api-xks-ca\") pod \"37c106d7-0120-4409-acc2-ef0db4a8b003\" (UID: \"37c106d7-0120-4409-acc2-ef0db4a8b003\") "
Oct 30 21:25:38.728753 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:38.728728    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37c106d7-0120-4409-acc2-ef0db4a8b003-maas-api-openshift-ca" pod "37c106d7-0120-4409-acc2-ef0db4a8b003" (UID: "37c106d7-0120-4409-acc2-ef0db4a8b003"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:25:38.730563 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:38.730541    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37c106d7-0120-4409-acc2-ef0db4a8b003-maas-api-xks-ca" pod "37c106d7-0120-4409-acc2-ef0db4a8b003" (UID: "37c106d7-0120-4409-acc2-ef0db4a8b003"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:25:38.730647 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:38.730617    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37c106d7-0120-4409-acc2-ef0db4a8b003-kube-api-access-k7z6q" pod "37c106d7-0120-4409-acc2-ef0db4a8b003" (UID: "37c106d7-0120-4409-acc2-ef0db4a8b003"). InnerVolumeSpecName "kube-api-access-k7z6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:38.829297 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:38.829212    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37663328-2eda-4b83-bc43-c60883c8694c-kube-api-access-kgbhs\" (UniqueName: \"kubernetes.io/projected/37663328-2eda-4b83-bc43-c60883c8694c-kube-api-access-kgbhs\") pod \"37663328-2eda-4b83-bc43-c60883c8694c\" (UID: \"37663328-2eda-4b83-bc43-c60883c8694c\") "
Oct 30 21:25:38.829464 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:38.829336    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37663328-2eda-4b83-bc43-c60883c8694c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37663328-2eda-4b83-bc43-c60883c8694c-maas-api-xks-ca\") pod \"37663328-2eda-4b83-bc43-c60883c8694c\" (UID: \"37663328-2eda-4b83-bc43-c60883c8694c\") "
Oct 30 21:25:38.829464 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:38.829364    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37663328-2eda-4b83-bc43-c60883c8694c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37663328-2eda-4b83-bc43-c60883c8694c-maas-api-openshift-ca\") pod \"37663328-2eda-4b83-bc43-c60883c8694c\" (UID: \"37663328-2eda-4b83-bc43-c60883c8694c\") "
Oct 30 21:25:38.829574 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:38.829560    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37c106d7-0120-4409-acc2-ef0db4a8b003-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:25:38.829639 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:38.829575    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7z6q\" (UniqueName: \"kubernetes.io/projected/37c106d7-0120-4409-acc2-ef0db4a8b003-kube-api-access-k7z6q\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:25:38.829639 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:38.829585    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37c106d7-0120-4409-acc2-ef0db4a8b003-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:25:38.829857 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:38.829831    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37663328-2eda-4b83-bc43-c60883c8694c-maas-api-openshift-ca" pod "37663328-2eda-4b83-bc43-c60883c8694c" (UID: "37663328-2eda-4b83-bc43-c60883c8694c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:25:38.831496 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:38.831475    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37663328-2eda-4b83-bc43-c60883c8694c-kube-api-access-kgbhs" pod "37663328-2eda-4b83-bc43-c60883c8694c" (UID: "37663328-2eda-4b83-bc43-c60883c8694c"). InnerVolumeSpecName "kube-api-access-kgbhs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:38.831570 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:38.831532    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37663328-2eda-4b83-bc43-c60883c8694c-maas-api-xks-ca" pod "37663328-2eda-4b83-bc43-c60883c8694c" (UID: "37663328-2eda-4b83-bc43-c60883c8694c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:25:38.930219 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:38.930176    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37663328-2eda-4b83-bc43-c60883c8694c-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:25:38.930219 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:38.930212    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgbhs\" (UniqueName: \"kubernetes.io/projected/37663328-2eda-4b83-bc43-c60883c8694c-kube-api-access-kgbhs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:25:38.930219 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:38.930224    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37663328-2eda-4b83-bc43-c60883c8694c-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:25:39.062366 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:39.062322    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d0b55cf-0697-4ecb-aaa2-479b968fbd2f" path="/var/lib/kubelet/pods/5d0b55cf-0697-4ecb-aaa2-479b968fbd2f/volumes"
Oct 30 21:25:39.521426 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:39.521389    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-430f8459-rc6bx"
Oct 30 21:25:39.521622 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:39.521420    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-430f8459-rc6bx" event={"ID":"37c106d7-0120-4409-acc2-ef0db4a8b003","Type":"ContainerDied","Data":"4e6e2be47bfdade751dcc740d4e89958b0261acf4909b042fed80e3cdac6d273"}
Oct 30 21:25:39.521622 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:39.521504    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e6e2be47bfdade751dcc740d4e89958b0261acf4909b042fed80e3cdac6d273"
Oct 30 21:25:39.522846 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:39.522822    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b7c7b6-7fmvd"
Oct 30 21:25:39.522967 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:39.522817    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b7c7b6-7fmvd" event={"ID":"37663328-2eda-4b83-bc43-c60883c8694c","Type":"ContainerDied","Data":"9f063bd53389eb2fc3b4cd038878c12b4458e47a76ed0e36e1eb9df091486cbf"}
Oct 30 21:25:39.522967 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:39.522936    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f063bd53389eb2fc3b4cd038878c12b4458e47a76ed0e36e1eb9df091486cbf"
Oct 30 21:25:39.761230 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:39.761199    5713 scope.go:122] "RemoveContainer" containerID="248aa9b531f637cfc83c8a4e0738ad785d6d0518394e7efbe5223b0eee488774"
Oct 30 21:25:39.771157 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:39.771135    5713 scope.go:122] "RemoveContainer" containerID="317fcab0854178e48daa348f091f4a7abaf1653ccf67bfae69aa14db368026ad"
Oct 30 21:25:42.593213 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:42.593173    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-8fa2a7-977ff4795-mgd5h" podUID="0fa5be36-c8de-4cdc-87d9-ebf11115f0c3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:25:43.550620 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:43.550586    5713 generic.go:356] "Generic (PLEG): container finished" podID="0fa5be36-c8de-4cdc-87d9-ebf11115f0c3" containerID="c8860b676c5420e7791c6545f94fc26e9a5ebcd3b4cad569dcfc5d92bd9b6d4d" exitCode=0
Oct 30 21:25:43.550827 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:43.550660    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-8fa2a7-977ff4795-mgd5h" event={"ID":"0fa5be36-c8de-4cdc-87d9-ebf11115f0c3","Type":"ContainerDied","Data":"c8860b676c5420e7791c6545f94fc26e9a5ebcd3b4cad569dcfc5d92bd9b6d4d"}
Oct 30 21:25:43.590944 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:43.590918    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-8fa2a7-977ff4795-mgd5h"
Oct 30 21:25:43.680475 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:43.680447    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fa5be36-c8de-4cdc-87d9-ebf11115f0c3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fa5be36-c8de-4cdc-87d9-ebf11115f0c3-plugins-config-volume\") pod \"0fa5be36-c8de-4cdc-87d9-ebf11115f0c3\" (UID: \"0fa5be36-c8de-4cdc-87d9-ebf11115f0c3\") "
Oct 30 21:25:43.680917 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:43.680553    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fa5be36-c8de-4cdc-87d9-ebf11115f0c3-kube-api-access-h9h2j\" (UniqueName: \"kubernetes.io/projected/0fa5be36-c8de-4cdc-87d9-ebf11115f0c3-kube-api-access-h9h2j\") pod \"0fa5be36-c8de-4cdc-87d9-ebf11115f0c3\" (UID: \"0fa5be36-c8de-4cdc-87d9-ebf11115f0c3\") "
Oct 30 21:25:43.680917 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:43.680818    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fa5be36-c8de-4cdc-87d9-ebf11115f0c3-plugins-config-volume" pod "0fa5be36-c8de-4cdc-87d9-ebf11115f0c3" (UID: "0fa5be36-c8de-4cdc-87d9-ebf11115f0c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:25:43.682877 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:43.682854    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fa5be36-c8de-4cdc-87d9-ebf11115f0c3-kube-api-access-h9h2j" pod "0fa5be36-c8de-4cdc-87d9-ebf11115f0c3" (UID: "0fa5be36-c8de-4cdc-87d9-ebf11115f0c3"). InnerVolumeSpecName "kube-api-access-h9h2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:43.781688 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:43.781655    5713 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fa5be36-c8de-4cdc-87d9-ebf11115f0c3-plugins-config-volume\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:25:43.781688 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:43.781681    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9h2j\" (UniqueName: \"kubernetes.io/projected/0fa5be36-c8de-4cdc-87d9-ebf11115f0c3-kube-api-access-h9h2j\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:25:44.352689 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:44.352652    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-5c45e5-bfsnk"]
Oct 30 21:25:44.402324 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:44.402289    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-5c45e5-bfsnk"]
Oct 30 21:25:44.558141 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:44.558113    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-8fa2a7-977ff4795-mgd5h"
Oct 30 21:25:44.558141 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:44.558117    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-8fa2a7-977ff4795-mgd5h" event={"ID":"0fa5be36-c8de-4cdc-87d9-ebf11115f0c3","Type":"ContainerDied","Data":"c64919b63d4c86ae9639fc914d7004712633e653bd0aa3caf70cec217a181ca2"}
Oct 30 21:25:44.558383 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:44.558219    5713 scope.go:122] "RemoveContainer" containerID="c8860b676c5420e7791c6545f94fc26e9a5ebcd3b4cad569dcfc5d92bd9b6d4d"
Oct 30 21:25:44.731313 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:44.731279    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-8fa2a7-977ff4795-mgd5h"]
Oct 30 21:25:44.742567 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:44.742539    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-8fa2a7-977ff4795-mgd5h"]
Oct 30 21:25:45.062682 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:45.062598    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fa5be36-c8de-4cdc-87d9-ebf11115f0c3" path="/var/lib/kubelet/pods/0fa5be36-c8de-4cdc-87d9-ebf11115f0c3/volumes"
Oct 30 21:25:45.063108 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:45.063089    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dd683f0-d100-4c2b-b47c-84f211e6f239" path="/var/lib/kubelet/pods/3dd683f0-d100-4c2b-b47c-84f211e6f239/volumes"
Oct 30 21:25:50.046266 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:50.046227    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fa33f00-hcbpf"]
Oct 30 21:25:50.096041 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:50.096006    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fa33f00-hcbpf"]
Oct 30 21:25:51.062818 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:51.062759    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c6b30eb-8d16-4cbb-a7b7-d833f5e5e843" path="/var/lib/kubelet/pods/0c6b30eb-8d16-4cbb-a7b7-d833f5e5e843/volumes"
Oct 30 21:25:52.020252 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:52.020210    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-88da25-cg9x4"]
Oct 30 21:25:52.020714 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:52.020694    5713 cpu_manager.go:414] "containerMap: removing container" podUID="0fa5be36-c8de-4cdc-87d9-ebf11115f0c3" containerName="payload-pre-processing"
Oct 30 21:25:52.020849 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:52.020716    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fa5be36-c8de-4cdc-87d9-ebf11115f0c3" containerName="payload-pre-processing"
Oct 30 21:25:52.020849 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:52.020740    5713 cpu_manager.go:414] "containerMap: removing container" podUID="37c106d7-0120-4409-acc2-ef0db4a8b003" containerName="revoke-keys"
Oct 30 21:25:52.020849 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:52.020747    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37c106d7-0120-4409-acc2-ef0db4a8b003" containerName="revoke-keys"
Oct 30 21:25:52.020849 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:52.020815    5713 cpu_manager.go:414] "containerMap: removing container" podUID="37663328-2eda-4b83-bc43-c60883c8694c" containerName="revoke-keys"
Oct 30 21:25:52.020849 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:52.020824    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37663328-2eda-4b83-bc43-c60883c8694c" containerName="revoke-keys"
Oct 30 21:25:52.021121 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:52.020935    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fa5be36-c8de-4cdc-87d9-ebf11115f0c3" containerName="payload-pre-processing"
Oct 30 21:25:52.021121 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:52.020951    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="37c106d7-0120-4409-acc2-ef0db4a8b003" containerName="revoke-keys"
Oct 30 21:25:52.021121 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:52.020971    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="37663328-2eda-4b83-bc43-c60883c8694c" containerName="revoke-keys"
Oct 30 21:25:52.025634 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:52.025611    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-88da25-cg9x4"
Oct 30 21:25:52.066464 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:52.066433    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a08bcbb4-ecdd-4062-93fb-6744af95db60-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-88da25-cg9x4\" (UID: \"a08bcbb4-ecdd-4062-93fb-6744af95db60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-88da25-cg9x4"
Oct 30 21:25:52.066858 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:52.066606    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw4b4\" (UniqueName: \"kubernetes.io/projected/a08bcbb4-ecdd-4062-93fb-6744af95db60-kube-api-access-jw4b4\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-88da25-cg9x4\" (UID: \"a08bcbb4-ecdd-4062-93fb-6744af95db60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-88da25-cg9x4"
Oct 30 21:25:52.066858 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:52.066639    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a08bcbb4-ecdd-4062-93fb-6744af95db60-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-88da25-cg9x4\" (UID: \"a08bcbb4-ecdd-4062-93fb-6744af95db60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-88da25-cg9x4"
Oct 30 21:25:52.081616 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:52.081587    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-88da25-cg9x4"]
Oct 30 21:25:52.128432 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:52.128401    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:25:52.128432 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:52.128425    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:25:52.128608 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:52.128411    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:25:52.128608 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:52.128446    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:25:52.167930 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:52.167899    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw4b4\" (UniqueName: \"kubernetes.io/projected/a08bcbb4-ecdd-4062-93fb-6744af95db60-kube-api-access-jw4b4\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-88da25-cg9x4\" (UID: \"a08bcbb4-ecdd-4062-93fb-6744af95db60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-88da25-cg9x4"
Oct 30 21:25:52.168052 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:52.167936    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a08bcbb4-ecdd-4062-93fb-6744af95db60-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-88da25-cg9x4\" (UID: \"a08bcbb4-ecdd-4062-93fb-6744af95db60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-88da25-cg9x4"
Oct 30 21:25:52.168052 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:52.167985    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a08bcbb4-ecdd-4062-93fb-6744af95db60-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-88da25-cg9x4\" (UID: \"a08bcbb4-ecdd-4062-93fb-6744af95db60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-88da25-cg9x4"
Oct 30 21:25:52.168644 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:52.168621    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a08bcbb4-ecdd-4062-93fb-6744af95db60-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-88da25-cg9x4\" (UID: \"a08bcbb4-ecdd-4062-93fb-6744af95db60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-88da25-cg9x4"
Oct 30 21:25:52.170561 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:52.170541    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a08bcbb4-ecdd-4062-93fb-6744af95db60-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-88da25-cg9x4\" (UID: \"a08bcbb4-ecdd-4062-93fb-6744af95db60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-88da25-cg9x4"
Oct 30 21:25:52.224980 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:52.224952    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw4b4\" (UniqueName: \"kubernetes.io/projected/a08bcbb4-ecdd-4062-93fb-6744af95db60-kube-api-access-jw4b4\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-88da25-cg9x4\" (UID: \"a08bcbb4-ecdd-4062-93fb-6744af95db60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-88da25-cg9x4"
Oct 30 21:25:52.339344 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:52.339271    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-88da25-cg9x4"
Oct 30 21:25:52.714348 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:52.714321    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-88da25-cg9x4"]
Oct 30 21:25:52.716282 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:25:52.716250    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda08bcbb4_ecdd_4062_93fb_6744af95db60.slice/crio-2879ebbfc123d76da6c66fb424266d9ea3bd6d7cf0120eaac34649a5ab9cad1e WatchSource:0}: Error finding container 2879ebbfc123d76da6c66fb424266d9ea3bd6d7cf0120eaac34649a5ab9cad1e: Status 404 returned error can't find the container with id 2879ebbfc123d76da6c66fb424266d9ea3bd6d7cf0120eaac34649a5ab9cad1e
Oct 30 21:25:53.625687 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:53.625659    5713 generic.go:356] "Generic (PLEG): container finished" podID="a08bcbb4-ecdd-4062-93fb-6744af95db60" containerID="be9e42f3f1c03db862a405b5f2d0907e947d6003a191517fe270c87cfc496fa2" exitCode=0
Oct 30 21:25:53.626015 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:53.625714    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-88da25-cg9x4" event={"ID":"a08bcbb4-ecdd-4062-93fb-6744af95db60","Type":"ContainerDied","Data":"be9e42f3f1c03db862a405b5f2d0907e947d6003a191517fe270c87cfc496fa2"}
Oct 30 21:25:53.626015 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:53.625782    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-88da25-cg9x4" event={"ID":"a08bcbb4-ecdd-4062-93fb-6744af95db60","Type":"ContainerStarted","Data":"2879ebbfc123d76da6c66fb424266d9ea3bd6d7cf0120eaac34649a5ab9cad1e"}
Oct 30 21:25:54.458725 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:25:54.458690    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:25:54.780735 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:54.780708    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-88da25-cg9x4"
Oct 30 21:25:54.787969 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:54.787949    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a08bcbb4-ecdd-4062-93fb-6744af95db60-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a08bcbb4-ecdd-4062-93fb-6744af95db60-maas-api-openshift-ca\") pod \"a08bcbb4-ecdd-4062-93fb-6744af95db60\" (UID: \"a08bcbb4-ecdd-4062-93fb-6744af95db60\") "
Oct 30 21:25:54.788038 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:54.787996    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a08bcbb4-ecdd-4062-93fb-6744af95db60-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a08bcbb4-ecdd-4062-93fb-6744af95db60-maas-api-xks-ca\") pod \"a08bcbb4-ecdd-4062-93fb-6744af95db60\" (UID: \"a08bcbb4-ecdd-4062-93fb-6744af95db60\") "
Oct 30 21:25:54.788087 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:54.788054    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a08bcbb4-ecdd-4062-93fb-6744af95db60-kube-api-access-jw4b4\" (UniqueName: \"kubernetes.io/projected/a08bcbb4-ecdd-4062-93fb-6744af95db60-kube-api-access-jw4b4\") pod \"a08bcbb4-ecdd-4062-93fb-6744af95db60\" (UID: \"a08bcbb4-ecdd-4062-93fb-6744af95db60\") "
Oct 30 21:25:54.788336 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:54.788311    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a08bcbb4-ecdd-4062-93fb-6744af95db60-maas-api-openshift-ca" pod "a08bcbb4-ecdd-4062-93fb-6744af95db60" (UID: "a08bcbb4-ecdd-4062-93fb-6744af95db60"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:25:54.788431 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:54.788416    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a08bcbb4-ecdd-4062-93fb-6744af95db60-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:25:54.790266 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:54.790241    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a08bcbb4-ecdd-4062-93fb-6744af95db60-maas-api-xks-ca" pod "a08bcbb4-ecdd-4062-93fb-6744af95db60" (UID: "a08bcbb4-ecdd-4062-93fb-6744af95db60"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:25:54.790346 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:54.790299    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a08bcbb4-ecdd-4062-93fb-6744af95db60-kube-api-access-jw4b4" pod "a08bcbb4-ecdd-4062-93fb-6744af95db60" (UID: "a08bcbb4-ecdd-4062-93fb-6744af95db60"). InnerVolumeSpecName "kube-api-access-jw4b4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:54.889848 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:54.889815    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a08bcbb4-ecdd-4062-93fb-6744af95db60-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:25:54.889848 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:54.889843    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jw4b4\" (UniqueName: \"kubernetes.io/projected/a08bcbb4-ecdd-4062-93fb-6744af95db60-kube-api-access-jw4b4\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:25:55.640807 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:55.640759    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-88da25-cg9x4"
Oct 30 21:25:55.641021 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:55.640808    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-88da25-cg9x4" event={"ID":"a08bcbb4-ecdd-4062-93fb-6744af95db60","Type":"ContainerDied","Data":"2879ebbfc123d76da6c66fb424266d9ea3bd6d7cf0120eaac34649a5ab9cad1e"}
Oct 30 21:25:55.641021 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:55.640858    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2879ebbfc123d76da6c66fb424266d9ea3bd6d7cf0120eaac34649a5ab9cad1e"
Oct 30 21:25:56.024627 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:56.024598    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8fa2a7-xdcsk"]
Oct 30 21:25:56.027653 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:56.027628    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8fa2a7-xdcsk"]
Oct 30 21:25:57.062019 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:25:57.061985    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3dc86ed-d40b-43f4-b403-7c7840462545" path="/var/lib/kubelet/pods/b3dc86ed-d40b-43f4-b403-7c7840462545/volumes"
Oct 30 21:26:02.611288 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:02.611255    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs"]
Oct 30 21:26:02.611757 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:02.611454    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs" podUID="7907175f-e439-48d7-8754-04591bde77cd" containerName="istio-proxy" containerID="cri-o://0e42e9d485f3930f8a16e3310171c6c31f47bbcdfb4268da004df78e284e2b98" gracePeriod=30
Oct 30 21:26:04.331895 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:04.331771    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-88da25-5689b65574-4grv2"]
Oct 30 21:26:07.863677 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:07.863651    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs"
Oct 30 21:26:07.920043 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:07.919968    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7907175f-e439-48d7-8754-04591bde77cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7907175f-e439-48d7-8754-04591bde77cd-istio-data\") pod \"7907175f-e439-48d7-8754-04591bde77cd\" (UID: \"7907175f-e439-48d7-8754-04591bde77cd\") "
Oct 30 21:26:07.920043 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:07.920013    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7907175f-e439-48d7-8754-04591bde77cd-istio-token\" (UniqueName: \"kubernetes.io/projected/7907175f-e439-48d7-8754-04591bde77cd-istio-token\") pod \"7907175f-e439-48d7-8754-04591bde77cd\" (UID: \"7907175f-e439-48d7-8754-04591bde77cd\") "
Oct 30 21:26:07.920043 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:07.920035    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7907175f-e439-48d7-8754-04591bde77cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7907175f-e439-48d7-8754-04591bde77cd-workload-certs\") pod \"7907175f-e439-48d7-8754-04591bde77cd\" (UID: \"7907175f-e439-48d7-8754-04591bde77cd\") "
Oct 30 21:26:07.920307 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:07.920071    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7907175f-e439-48d7-8754-04591bde77cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7907175f-e439-48d7-8754-04591bde77cd-credential-socket\") pod \"7907175f-e439-48d7-8754-04591bde77cd\" (UID: \"7907175f-e439-48d7-8754-04591bde77cd\") "
Oct 30 21:26:07.920307 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:07.920102    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7907175f-e439-48d7-8754-04591bde77cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7907175f-e439-48d7-8754-04591bde77cd-istio-envoy\") pod \"7907175f-e439-48d7-8754-04591bde77cd\" (UID: \"7907175f-e439-48d7-8754-04591bde77cd\") "
Oct 30 21:26:07.920307 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:07.920168    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7907175f-e439-48d7-8754-04591bde77cd-kube-api-access-2st6r\" (UniqueName: \"kubernetes.io/projected/7907175f-e439-48d7-8754-04591bde77cd-kube-api-access-2st6r\") pod \"7907175f-e439-48d7-8754-04591bde77cd\" (UID: \"7907175f-e439-48d7-8754-04591bde77cd\") "
Oct 30 21:26:07.920307 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:07.920202    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7907175f-e439-48d7-8754-04591bde77cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7907175f-e439-48d7-8754-04591bde77cd-workload-socket\") pod \"7907175f-e439-48d7-8754-04591bde77cd\" (UID: \"7907175f-e439-48d7-8754-04591bde77cd\") "
Oct 30 21:26:07.920307 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:07.920248    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7907175f-e439-48d7-8754-04591bde77cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7907175f-e439-48d7-8754-04591bde77cd-istiod-ca-cert\") pod \"7907175f-e439-48d7-8754-04591bde77cd\" (UID: \"7907175f-e439-48d7-8754-04591bde77cd\") "
Oct 30 21:26:07.920307 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:07.920266    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7907175f-e439-48d7-8754-04591bde77cd-istio-data" pod "7907175f-e439-48d7-8754-04591bde77cd" (UID: "7907175f-e439-48d7-8754-04591bde77cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:26:07.920625 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:07.920366    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7907175f-e439-48d7-8754-04591bde77cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7907175f-e439-48d7-8754-04591bde77cd-istio-podinfo\") pod \"7907175f-e439-48d7-8754-04591bde77cd\" (UID: \"7907175f-e439-48d7-8754-04591bde77cd\") "
Oct 30 21:26:07.920625 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:07.920464    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7907175f-e439-48d7-8754-04591bde77cd-credential-socket" pod "7907175f-e439-48d7-8754-04591bde77cd" (UID: "7907175f-e439-48d7-8754-04591bde77cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:26:07.920625 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:07.920482    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7907175f-e439-48d7-8754-04591bde77cd-workload-certs" pod "7907175f-e439-48d7-8754-04591bde77cd" (UID: "7907175f-e439-48d7-8754-04591bde77cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:26:07.920625 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:07.920549    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7907175f-e439-48d7-8754-04591bde77cd-workload-socket" pod "7907175f-e439-48d7-8754-04591bde77cd" (UID: "7907175f-e439-48d7-8754-04591bde77cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:26:07.920838 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:07.920816    5713 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7907175f-e439-48d7-8754-04591bde77cd-istio-data\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:26:07.920948 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:07.920887    5713 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7907175f-e439-48d7-8754-04591bde77cd-workload-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:26:07.921012 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:07.920962    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7907175f-e439-48d7-8754-04591bde77cd-istiod-ca-cert" pod "7907175f-e439-48d7-8754-04591bde77cd" (UID: "7907175f-e439-48d7-8754-04591bde77cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:26:07.921052 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:07.921003    5713 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7907175f-e439-48d7-8754-04591bde77cd-credential-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:26:07.921052 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:07.921033    5713 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7907175f-e439-48d7-8754-04591bde77cd-workload-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:26:07.922495 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:07.922473    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7907175f-e439-48d7-8754-04591bde77cd-istio-token" pod "7907175f-e439-48d7-8754-04591bde77cd" (UID: "7907175f-e439-48d7-8754-04591bde77cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:26:07.922614 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:07.922500    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7907175f-e439-48d7-8754-04591bde77cd-kube-api-access-2st6r" pod "7907175f-e439-48d7-8754-04591bde77cd" (UID: "7907175f-e439-48d7-8754-04591bde77cd"). InnerVolumeSpecName "kube-api-access-2st6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:26:07.922666 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:07.922610    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7907175f-e439-48d7-8754-04591bde77cd-istio-podinfo" pod "7907175f-e439-48d7-8754-04591bde77cd" (UID: "7907175f-e439-48d7-8754-04591bde77cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:26:07.922877 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:07.922857    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7907175f-e439-48d7-8754-04591bde77cd-istio-envoy" pod "7907175f-e439-48d7-8754-04591bde77cd" (UID: "7907175f-e439-48d7-8754-04591bde77cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:26:08.021512 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:08.021485    5713 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7907175f-e439-48d7-8754-04591bde77cd-istiod-ca-cert\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:26:08.021512 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:08.021514    5713 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7907175f-e439-48d7-8754-04591bde77cd-istio-podinfo\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:26:08.021717 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:08.021524    5713 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7907175f-e439-48d7-8754-04591bde77cd-istio-token\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:26:08.021717 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:08.021534    5713 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7907175f-e439-48d7-8754-04591bde77cd-istio-envoy\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:26:08.021717 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:08.021543    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2st6r\" (UniqueName: \"kubernetes.io/projected/7907175f-e439-48d7-8754-04591bde77cd-kube-api-access-2st6r\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:26:08.722224 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:08.722188    5713 generic.go:356] "Generic (PLEG): container finished" podID="7907175f-e439-48d7-8754-04591bde77cd" containerID="0e42e9d485f3930f8a16e3310171c6c31f47bbcdfb4268da004df78e284e2b98" exitCode=0
Oct 30 21:26:08.722400 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:08.722233    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs"
Oct 30 21:26:08.722400 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:08.722276    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs" event={"ID":"7907175f-e439-48d7-8754-04591bde77cd","Type":"ContainerDied","Data":"0e42e9d485f3930f8a16e3310171c6c31f47bbcdfb4268da004df78e284e2b98"}
Oct 30 21:26:08.722400 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:08.722326    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs" event={"ID":"7907175f-e439-48d7-8754-04591bde77cd","Type":"ContainerDied","Data":"daa0282ee07ef17c6bbf349ea0f28f1a29e691536b5a97556530366d50b648bb"}
Oct 30 21:26:08.722400 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:08.722350    5713 scope.go:122] "RemoveContainer" containerID="0e42e9d485f3930f8a16e3310171c6c31f47bbcdfb4268da004df78e284e2b98"
Oct 30 21:26:08.734184 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:08.734013    5713 scope.go:122] "RemoveContainer" containerID="819041f4dd6f4abf2ee3ffddd4269f5574b61ea26270d5292d338e4bbf72ce51"
Oct 30 21:26:08.744525 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:08.744505    5713 scope.go:122] "RemoveContainer" containerID="0e42e9d485f3930f8a16e3310171c6c31f47bbcdfb4268da004df78e284e2b98"
Oct 30 21:26:08.744780 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:26:08.744763    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e42e9d485f3930f8a16e3310171c6c31f47bbcdfb4268da004df78e284e2b98\": container with ID starting with 0e42e9d485f3930f8a16e3310171c6c31f47bbcdfb4268da004df78e284e2b98 not found: ID does not exist" containerID="0e42e9d485f3930f8a16e3310171c6c31f47bbcdfb4268da004df78e284e2b98"
Oct 30 21:26:08.744869 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:08.744817    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e42e9d485f3930f8a16e3310171c6c31f47bbcdfb4268da004df78e284e2b98"} err="failed to get container status \"0e42e9d485f3930f8a16e3310171c6c31f47bbcdfb4268da004df78e284e2b98\": rpc error: code = NotFound desc = could not find container \"0e42e9d485f3930f8a16e3310171c6c31f47bbcdfb4268da004df78e284e2b98\": container with ID starting with 0e42e9d485f3930f8a16e3310171c6c31f47bbcdfb4268da004df78e284e2b98 not found: ID does not exist"
Oct 30 21:26:08.744869 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:08.744844    5713 scope.go:122] "RemoveContainer" containerID="819041f4dd6f4abf2ee3ffddd4269f5574b61ea26270d5292d338e4bbf72ce51"
Oct 30 21:26:08.745072 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:26:08.745052    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"819041f4dd6f4abf2ee3ffddd4269f5574b61ea26270d5292d338e4bbf72ce51\": container with ID starting with 819041f4dd6f4abf2ee3ffddd4269f5574b61ea26270d5292d338e4bbf72ce51 not found: ID does not exist" containerID="819041f4dd6f4abf2ee3ffddd4269f5574b61ea26270d5292d338e4bbf72ce51"
Oct 30 21:26:08.745111 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:08.745082    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"819041f4dd6f4abf2ee3ffddd4269f5574b61ea26270d5292d338e4bbf72ce51"} err="failed to get container status \"819041f4dd6f4abf2ee3ffddd4269f5574b61ea26270d5292d338e4bbf72ce51\": rpc error: code = NotFound desc = could not find container \"819041f4dd6f4abf2ee3ffddd4269f5574b61ea26270d5292d338e4bbf72ce51\": container with ID starting with 819041f4dd6f4abf2ee3ffddd4269f5574b61ea26270d5292d338e4bbf72ce51 not found: ID does not exist"
Oct 30 21:26:08.936547 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:08.936516    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs"]
Oct 30 21:26:08.986281 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:08.986209    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-8fa2a7-openshift-default-69876bd499-hl2vs"]
Oct 30 21:26:09.070723 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:09.070681    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7907175f-e439-48d7-8754-04591bde77cd" path="/var/lib/kubelet/pods/7907175f-e439-48d7-8754-04591bde77cd/volumes"
Oct 30 21:26:09.333227 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:09.333130    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-88da25-5689b65574-4grv2" podUID="2b53c958-a28c-44d6-bf7b-76b9891f67e5" containerName="payload-processing" containerID="cri-o://330e193080a93bbed8e503754669b4abb9783e65b94fe3e27c38f650059ae247" gracePeriod=25
Oct 30 21:26:09.594525 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:09.594450    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-88da25-5689b65574-4grv2"
Oct 30 21:26:09.636676 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:09.636633    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b53c958-a28c-44d6-bf7b-76b9891f67e5-kube-api-access-rbqt5\" (UniqueName: \"kubernetes.io/projected/2b53c958-a28c-44d6-bf7b-76b9891f67e5-kube-api-access-rbqt5\") pod \"2b53c958-a28c-44d6-bf7b-76b9891f67e5\" (UID: \"2b53c958-a28c-44d6-bf7b-76b9891f67e5\") "
Oct 30 21:26:09.636913 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:09.636834    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b53c958-a28c-44d6-bf7b-76b9891f67e5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b53c958-a28c-44d6-bf7b-76b9891f67e5-plugins-config-volume\") pod \"2b53c958-a28c-44d6-bf7b-76b9891f67e5\" (UID: \"2b53c958-a28c-44d6-bf7b-76b9891f67e5\") "
Oct 30 21:26:09.637202 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:09.637179    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b53c958-a28c-44d6-bf7b-76b9891f67e5-plugins-config-volume" pod "2b53c958-a28c-44d6-bf7b-76b9891f67e5" (UID: "2b53c958-a28c-44d6-bf7b-76b9891f67e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:26:09.638999 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:09.638977    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b53c958-a28c-44d6-bf7b-76b9891f67e5-kube-api-access-rbqt5" pod "2b53c958-a28c-44d6-bf7b-76b9891f67e5" (UID: "2b53c958-a28c-44d6-bf7b-76b9891f67e5"). InnerVolumeSpecName "kube-api-access-rbqt5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:26:09.731842 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:09.731802    5713 generic.go:356] "Generic (PLEG): container finished" podID="2b53c958-a28c-44d6-bf7b-76b9891f67e5" containerID="330e193080a93bbed8e503754669b4abb9783e65b94fe3e27c38f650059ae247" exitCode=0
Oct 30 21:26:09.732034 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:09.731868    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-88da25-5689b65574-4grv2"
Oct 30 21:26:09.732034 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:09.731909    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-88da25-5689b65574-4grv2" event={"ID":"2b53c958-a28c-44d6-bf7b-76b9891f67e5","Type":"ContainerDied","Data":"330e193080a93bbed8e503754669b4abb9783e65b94fe3e27c38f650059ae247"}
Oct 30 21:26:09.732034 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:09.731969    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-88da25-5689b65574-4grv2" event={"ID":"2b53c958-a28c-44d6-bf7b-76b9891f67e5","Type":"ContainerDied","Data":"be86d7437c78698de40d965cee0d2651f10be2ab1b76f9f0114838777f249cfd"}
Oct 30 21:26:09.732034 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:09.731993    5713 scope.go:122] "RemoveContainer" containerID="330e193080a93bbed8e503754669b4abb9783e65b94fe3e27c38f650059ae247"
Oct 30 21:26:09.738448 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:09.738415    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbqt5\" (UniqueName: \"kubernetes.io/projected/2b53c958-a28c-44d6-bf7b-76b9891f67e5-kube-api-access-rbqt5\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:26:09.738600 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:09.738453    5713 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b53c958-a28c-44d6-bf7b-76b9891f67e5-plugins-config-volume\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:26:09.744616 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:09.744494    5713 scope.go:122] "RemoveContainer" containerID="330e193080a93bbed8e503754669b4abb9783e65b94fe3e27c38f650059ae247"
Oct 30 21:26:09.744868 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:26:09.744846    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"330e193080a93bbed8e503754669b4abb9783e65b94fe3e27c38f650059ae247\": container with ID starting with 330e193080a93bbed8e503754669b4abb9783e65b94fe3e27c38f650059ae247 not found: ID does not exist" containerID="330e193080a93bbed8e503754669b4abb9783e65b94fe3e27c38f650059ae247"
Oct 30 21:26:09.744925 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:09.744877    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"330e193080a93bbed8e503754669b4abb9783e65b94fe3e27c38f650059ae247"} err="failed to get container status \"330e193080a93bbed8e503754669b4abb9783e65b94fe3e27c38f650059ae247\": rpc error: code = NotFound desc = could not find container \"330e193080a93bbed8e503754669b4abb9783e65b94fe3e27c38f650059ae247\": container with ID starting with 330e193080a93bbed8e503754669b4abb9783e65b94fe3e27c38f650059ae247 not found: ID does not exist"
Oct 30 21:26:09.895902 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:09.895783    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-88da25-5689b65574-4grv2"]
Oct 30 21:26:09.900881 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:09.900848    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-88da25-5689b65574-4grv2"]
Oct 30 21:26:11.062606 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:11.062564    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b53c958-a28c-44d6-bf7b-76b9891f67e5" path="/var/lib/kubelet/pods/2b53c958-a28c-44d6-bf7b-76b9891f67e5/volumes"
Oct 30 21:26:11.854049 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:11.854007    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-88da25-cg9x4"]
Oct 30 21:26:11.903580 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:11.903548    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-88da25-cg9x4"]
Oct 30 21:26:13.061454 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:13.061422    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a08bcbb4-ecdd-4062-93fb-6744af95db60" path="/var/lib/kubelet/pods/a08bcbb4-ecdd-4062-93fb-6744af95db60/volumes"
Oct 30 21:26:19.290156 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:19.290122    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1fdaa7-m9jds"]
Oct 30 21:26:19.290543 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:19.290530    5713 cpu_manager.go:414] "containerMap: removing container" podUID="7907175f-e439-48d7-8754-04591bde77cd" containerName="istio-proxy"
Oct 30 21:26:19.290603 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:19.290545    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7907175f-e439-48d7-8754-04591bde77cd" containerName="istio-proxy"
Oct 30 21:26:19.290603 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:19.290553    5713 cpu_manager.go:414] "containerMap: removing container" podUID="a08bcbb4-ecdd-4062-93fb-6744af95db60" containerName="revoke-keys"
Oct 30 21:26:19.290603 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:19.290559    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a08bcbb4-ecdd-4062-93fb-6744af95db60" containerName="revoke-keys"
Oct 30 21:26:19.290603 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:19.290579    5713 cpu_manager.go:414] "containerMap: removing container" podUID="7907175f-e439-48d7-8754-04591bde77cd" containerName="istio-proxy"
Oct 30 21:26:19.290603 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:19.290584    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7907175f-e439-48d7-8754-04591bde77cd" containerName="istio-proxy"
Oct 30 21:26:19.290767 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:19.290607    5713 cpu_manager.go:414] "containerMap: removing container" podUID="2b53c958-a28c-44d6-bf7b-76b9891f67e5" containerName="payload-processing"
Oct 30 21:26:19.290767 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:19.290612    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b53c958-a28c-44d6-bf7b-76b9891f67e5" containerName="payload-processing"
Oct 30 21:26:19.290767 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:19.290672    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b53c958-a28c-44d6-bf7b-76b9891f67e5" containerName="payload-processing"
Oct 30 21:26:19.290767 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:19.290683    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="7907175f-e439-48d7-8754-04591bde77cd" containerName="istio-proxy"
Oct 30 21:26:19.290767 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:19.290689    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="a08bcbb4-ecdd-4062-93fb-6744af95db60" containerName="revoke-keys"
Oct 30 21:26:19.290767 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:19.290695    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="7907175f-e439-48d7-8754-04591bde77cd" containerName="istio-proxy"
Oct 30 21:26:19.295095 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:19.295075    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1fdaa7-m9jds"
Oct 30 21:26:19.347631 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:19.347604    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:26:19.350140 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:19.350113    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1fdaa7-m9jds"]
Oct 30 21:26:19.396438 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:19.396414    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:26:19.396438 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:19.396421    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:26:19.396641 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:19.396460    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:26:19.435286 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:19.435256    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/547b9a53-5c3d-487b-ad2c-eabcb6c71256-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-1fdaa7-m9jds\" (UID: \"547b9a53-5c3d-487b-ad2c-eabcb6c71256\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1fdaa7-m9jds"
Oct 30 21:26:19.435286 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:19.435287    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjcrk\" (UniqueName: \"kubernetes.io/projected/547b9a53-5c3d-487b-ad2c-eabcb6c71256-kube-api-access-vjcrk\") pod \"maas-api-revoke-keys-e2e-worker-w4-1fdaa7-m9jds\" (UID: \"547b9a53-5c3d-487b-ad2c-eabcb6c71256\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1fdaa7-m9jds"
Oct 30 21:26:19.435464 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:19.435313    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/547b9a53-5c3d-487b-ad2c-eabcb6c71256-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-1fdaa7-m9jds\" (UID: \"547b9a53-5c3d-487b-ad2c-eabcb6c71256\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1fdaa7-m9jds"
Oct 30 21:26:19.536068 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:19.536026    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/547b9a53-5c3d-487b-ad2c-eabcb6c71256-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-1fdaa7-m9jds\" (UID: \"547b9a53-5c3d-487b-ad2c-eabcb6c71256\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1fdaa7-m9jds"
Oct 30 21:26:19.536068 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:19.536071    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjcrk\" (UniqueName: \"kubernetes.io/projected/547b9a53-5c3d-487b-ad2c-eabcb6c71256-kube-api-access-vjcrk\") pod \"maas-api-revoke-keys-e2e-worker-w4-1fdaa7-m9jds\" (UID: \"547b9a53-5c3d-487b-ad2c-eabcb6c71256\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1fdaa7-m9jds"
Oct 30 21:26:19.536294 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:19.536111    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/547b9a53-5c3d-487b-ad2c-eabcb6c71256-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-1fdaa7-m9jds\" (UID: \"547b9a53-5c3d-487b-ad2c-eabcb6c71256\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1fdaa7-m9jds"
Oct 30 21:26:19.536688 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:19.536652    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/547b9a53-5c3d-487b-ad2c-eabcb6c71256-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-1fdaa7-m9jds\" (UID: \"547b9a53-5c3d-487b-ad2c-eabcb6c71256\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1fdaa7-m9jds"
Oct 30 21:26:19.538586 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:19.538564    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/547b9a53-5c3d-487b-ad2c-eabcb6c71256-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-1fdaa7-m9jds\" (UID: \"547b9a53-5c3d-487b-ad2c-eabcb6c71256\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1fdaa7-m9jds"
Oct 30 21:26:19.592596 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:19.592537    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjcrk\" (UniqueName: \"kubernetes.io/projected/547b9a53-5c3d-487b-ad2c-eabcb6c71256-kube-api-access-vjcrk\") pod \"maas-api-revoke-keys-e2e-worker-w4-1fdaa7-m9jds\" (UID: \"547b9a53-5c3d-487b-ad2c-eabcb6c71256\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1fdaa7-m9jds"
Oct 30 21:26:19.608320 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:19.608293    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1fdaa7-m9jds"
Oct 30 21:26:19.980867 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:19.980733    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1fdaa7-m9jds"]
Oct 30 21:26:19.983353 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:26:19.983329    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod547b9a53_5c3d_487b_ad2c_eabcb6c71256.slice/crio-ba1b81de32419b86b5d3de2a0042f7582cece6ff89bed98ddeb41ca26a186a33 WatchSource:0}: Error finding container ba1b81de32419b86b5d3de2a0042f7582cece6ff89bed98ddeb41ca26a186a33: Status 404 returned error can't find the container with id ba1b81de32419b86b5d3de2a0042f7582cece6ff89bed98ddeb41ca26a186a33
Oct 30 21:26:20.801442 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:20.801403    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1fdaa7-m9jds" event={"ID":"547b9a53-5c3d-487b-ad2c-eabcb6c71256","Type":"ContainerStarted","Data":"4e31423691724b64d6b96d290bf1e36347a15141ae8dbae375ce5da6bb604cea"}
Oct 30 21:26:20.801861 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:20.801488    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1fdaa7-m9jds" event={"ID":"547b9a53-5c3d-487b-ad2c-eabcb6c71256","Type":"ContainerStarted","Data":"ba1b81de32419b86b5d3de2a0042f7582cece6ff89bed98ddeb41ca26a186a33"}
Oct 30 21:26:20.914197 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:20.914148    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1fdaa7-m9jds" podStartSLOduration=1.914130533 podStartE2EDuration="1.914130533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:20.913986266 +0000 UTC m=+2863.978762665" watchObservedRunningTime="2026-10-30 21:26:20.914130533 +0000 UTC m=+2863.978906932"
Oct 30 21:26:21.808935 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:21.808893    5713 generic.go:356] "Generic (PLEG): container finished" podID="547b9a53-5c3d-487b-ad2c-eabcb6c71256" containerID="4e31423691724b64d6b96d290bf1e36347a15141ae8dbae375ce5da6bb604cea" exitCode=0
Oct 30 21:26:21.809366 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:21.808982    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1fdaa7-m9jds" event={"ID":"547b9a53-5c3d-487b-ad2c-eabcb6c71256","Type":"ContainerDied","Data":"4e31423691724b64d6b96d290bf1e36347a15141ae8dbae375ce5da6bb604cea"}
Oct 30 21:26:22.951130 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:22.951103    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1fdaa7-m9jds"
Oct 30 21:26:23.074319 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:23.074234    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/547b9a53-5c3d-487b-ad2c-eabcb6c71256-kube-api-access-vjcrk\" (UniqueName: \"kubernetes.io/projected/547b9a53-5c3d-487b-ad2c-eabcb6c71256-kube-api-access-vjcrk\") pod \"547b9a53-5c3d-487b-ad2c-eabcb6c71256\" (UID: \"547b9a53-5c3d-487b-ad2c-eabcb6c71256\") "
Oct 30 21:26:23.074319 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:23.074279    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/547b9a53-5c3d-487b-ad2c-eabcb6c71256-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/547b9a53-5c3d-487b-ad2c-eabcb6c71256-maas-api-xks-ca\") pod \"547b9a53-5c3d-487b-ad2c-eabcb6c71256\" (UID: \"547b9a53-5c3d-487b-ad2c-eabcb6c71256\") "
Oct 30 21:26:23.074597 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:23.074370    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/547b9a53-5c3d-487b-ad2c-eabcb6c71256-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/547b9a53-5c3d-487b-ad2c-eabcb6c71256-maas-api-openshift-ca\") pod \"547b9a53-5c3d-487b-ad2c-eabcb6c71256\" (UID: \"547b9a53-5c3d-487b-ad2c-eabcb6c71256\") "
Oct 30 21:26:23.075014 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:23.074987    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/547b9a53-5c3d-487b-ad2c-eabcb6c71256-maas-api-openshift-ca" pod "547b9a53-5c3d-487b-ad2c-eabcb6c71256" (UID: "547b9a53-5c3d-487b-ad2c-eabcb6c71256"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:26:23.077506 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:23.077475    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/547b9a53-5c3d-487b-ad2c-eabcb6c71256-kube-api-access-vjcrk" pod "547b9a53-5c3d-487b-ad2c-eabcb6c71256" (UID: "547b9a53-5c3d-487b-ad2c-eabcb6c71256"). InnerVolumeSpecName "kube-api-access-vjcrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:26:23.077621 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:23.077528    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/547b9a53-5c3d-487b-ad2c-eabcb6c71256-maas-api-xks-ca" pod "547b9a53-5c3d-487b-ad2c-eabcb6c71256" (UID: "547b9a53-5c3d-487b-ad2c-eabcb6c71256"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:26:23.175659 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:23.175623    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjcrk\" (UniqueName: \"kubernetes.io/projected/547b9a53-5c3d-487b-ad2c-eabcb6c71256-kube-api-access-vjcrk\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:26:23.175659 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:23.175652    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/547b9a53-5c3d-487b-ad2c-eabcb6c71256-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:26:23.175659 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:23.175662    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/547b9a53-5c3d-487b-ad2c-eabcb6c71256-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:26:23.823513 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:23.823479    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1fdaa7-m9jds"
Oct 30 21:26:23.823700 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:23.823514    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1fdaa7-m9jds" event={"ID":"547b9a53-5c3d-487b-ad2c-eabcb6c71256","Type":"ContainerDied","Data":"ba1b81de32419b86b5d3de2a0042f7582cece6ff89bed98ddeb41ca26a186a33"}
Oct 30 21:26:23.823700 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:23.823562    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba1b81de32419b86b5d3de2a0042f7582cece6ff89bed98ddeb41ca26a186a33"
Oct 30 21:26:24.456873 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:26:24.456839    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:26:30.783199 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:30.783162    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-1fdaa7-6fcb4879fd-bcz9f"]
Oct 30 21:26:30.893085 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:30.893050    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-1fdaa7-bb8847cd7-kq2ml"]
Oct 30 21:26:35.784262 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:35.784198    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-1fdaa7-6fcb4879fd-bcz9f" podUID="5304445a-544a-4cb7-b79a-53502f94fe48" containerName="payload-processing" containerID="cri-o://beca861051be812f67a97f823851e310d809f206f4bf724e896593cd93e3a955" gracePeriod=25
Oct 30 21:26:35.894169 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:35.894130    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1fdaa7-bb8847cd7-kq2ml" podUID="70aef1ff-8862-40b8-b3b7-c8878df1619f" containerName="payload-pre-processing" containerID="cri-o://2f46b08c16b08a4e8cb84d4301eab1832cb7bee4fbf766401131eab9983ac404" gracePeriod=25
Oct 30 21:26:36.354650 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:36.354610    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cd3daaa-s5ctm"]
Oct 30 21:26:36.355103 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:36.355088    5713 cpu_manager.go:414] "containerMap: removing container" podUID="547b9a53-5c3d-487b-ad2c-eabcb6c71256" containerName="revoke-keys"
Oct 30 21:26:36.355164 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:36.355107    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="547b9a53-5c3d-487b-ad2c-eabcb6c71256" containerName="revoke-keys"
Oct 30 21:26:36.355205 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:36.355177    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="547b9a53-5c3d-487b-ad2c-eabcb6c71256" containerName="revoke-keys"
Oct 30 21:26:36.358346 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:36.358331    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd3daaa-s5ctm"
Oct 30 21:26:36.404031 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:36.404008    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78d8a57a-mnsmx"]
Oct 30 21:26:36.405991 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:36.405970    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18d880ed-6e69-43b3-8224-85a2271fe2c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cd3daaa-s5ctm\" (UID: \"18d880ed-6e69-43b3-8224-85a2271fe2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd3daaa-s5ctm"
Oct 30 21:26:36.406060 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:36.406020    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f24b\" (UniqueName: \"kubernetes.io/projected/18d880ed-6e69-43b3-8224-85a2271fe2c8-kube-api-access-5f24b\") pod \"maas-api-revoke-sub-1cd3daaa-s5ctm\" (UID: \"18d880ed-6e69-43b3-8224-85a2271fe2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd3daaa-s5ctm"
Oct 30 21:26:36.406060 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:36.406041    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18d880ed-6e69-43b3-8224-85a2271fe2c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cd3daaa-s5ctm\" (UID: \"18d880ed-6e69-43b3-8224-85a2271fe2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd3daaa-s5ctm"
Oct 30 21:26:36.407538 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:36.407522    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78d8a57a-mnsmx"
Oct 30 21:26:36.451337 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:36.451312    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:26:36.451497 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:36.451337    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:26:36.451497 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:36.451361    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:26:36.451497 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:36.451420    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:26:36.453826 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:36.453802    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cd3daaa-s5ctm"]
Oct 30 21:26:36.465573 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:36.465552    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78d8a57a-mnsmx"]
Oct 30 21:26:36.507538 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:36.507511    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99143411-6bbc-483e-9c3d-002cfce1efdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78d8a57a-mnsmx\" (UID: \"99143411-6bbc-483e-9c3d-002cfce1efdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78d8a57a-mnsmx"
Oct 30 21:26:36.507665 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:36.507550    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f24b\" (UniqueName: \"kubernetes.io/projected/18d880ed-6e69-43b3-8224-85a2271fe2c8-kube-api-access-5f24b\") pod \"maas-api-revoke-sub-1cd3daaa-s5ctm\" (UID: \"18d880ed-6e69-43b3-8224-85a2271fe2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd3daaa-s5ctm"
Oct 30 21:26:36.507665 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:36.507572    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18d880ed-6e69-43b3-8224-85a2271fe2c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cd3daaa-s5ctm\" (UID: \"18d880ed-6e69-43b3-8224-85a2271fe2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd3daaa-s5ctm"
Oct 30 21:26:36.507744 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:36.507725    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18d880ed-6e69-43b3-8224-85a2271fe2c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cd3daaa-s5ctm\" (UID: \"18d880ed-6e69-43b3-8224-85a2271fe2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd3daaa-s5ctm"
Oct 30 21:26:36.507813 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:36.507772    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99143411-6bbc-483e-9c3d-002cfce1efdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78d8a57a-mnsmx\" (UID: \"99143411-6bbc-483e-9c3d-002cfce1efdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78d8a57a-mnsmx"
Oct 30 21:26:36.507864 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:36.507815    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd64z\" (UniqueName: \"kubernetes.io/projected/99143411-6bbc-483e-9c3d-002cfce1efdb-kube-api-access-cd64z\") pod \"maas-api-revoke-sub-78d8a57a-mnsmx\" (UID: \"99143411-6bbc-483e-9c3d-002cfce1efdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78d8a57a-mnsmx"
Oct 30 21:26:36.508384 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:36.508366    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18d880ed-6e69-43b3-8224-85a2271fe2c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cd3daaa-s5ctm\" (UID: \"18d880ed-6e69-43b3-8224-85a2271fe2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd3daaa-s5ctm"
Oct 30 21:26:36.510167 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:36.510150    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18d880ed-6e69-43b3-8224-85a2271fe2c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cd3daaa-s5ctm\" (UID: \"18d880ed-6e69-43b3-8224-85a2271fe2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd3daaa-s5ctm"
Oct 30 21:26:36.566284 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:36.566252    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f24b\" (UniqueName: \"kubernetes.io/projected/18d880ed-6e69-43b3-8224-85a2271fe2c8-kube-api-access-5f24b\") pod \"maas-api-revoke-sub-1cd3daaa-s5ctm\" (UID: \"18d880ed-6e69-43b3-8224-85a2271fe2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd3daaa-s5ctm"
Oct 30 21:26:36.608956 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:36.608893    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99143411-6bbc-483e-9c3d-002cfce1efdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78d8a57a-mnsmx\" (UID: \"99143411-6bbc-483e-9c3d-002cfce1efdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78d8a57a-mnsmx"
Oct 30 21:26:36.608956 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:36.608924    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd64z\" (UniqueName: \"kubernetes.io/projected/99143411-6bbc-483e-9c3d-002cfce1efdb-kube-api-access-cd64z\") pod \"maas-api-revoke-sub-78d8a57a-mnsmx\" (UID: \"99143411-6bbc-483e-9c3d-002cfce1efdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78d8a57a-mnsmx"
Oct 30 21:26:36.608956 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:36.608944    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99143411-6bbc-483e-9c3d-002cfce1efdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78d8a57a-mnsmx\" (UID: \"99143411-6bbc-483e-9c3d-002cfce1efdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78d8a57a-mnsmx"
Oct 30 21:26:36.609564 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:36.609546    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99143411-6bbc-483e-9c3d-002cfce1efdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78d8a57a-mnsmx\" (UID: \"99143411-6bbc-483e-9c3d-002cfce1efdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78d8a57a-mnsmx"
Oct 30 21:26:36.611317 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:36.611296    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99143411-6bbc-483e-9c3d-002cfce1efdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78d8a57a-mnsmx\" (UID: \"99143411-6bbc-483e-9c3d-002cfce1efdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78d8a57a-mnsmx"
Oct 30 21:26:36.666290 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:36.666263    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd64z\" (UniqueName: \"kubernetes.io/projected/99143411-6bbc-483e-9c3d-002cfce1efdb-kube-api-access-cd64z\") pod \"maas-api-revoke-sub-78d8a57a-mnsmx\" (UID: \"99143411-6bbc-483e-9c3d-002cfce1efdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78d8a57a-mnsmx"
Oct 30 21:26:36.671084 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:36.671065    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd3daaa-s5ctm"
Oct 30 21:26:36.721685 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:36.721653    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78d8a57a-mnsmx"
Oct 30 21:26:37.044598 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:37.044567    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cd3daaa-s5ctm"]
Oct 30 21:26:37.046041 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:26:37.046011    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18d880ed_6e69_43b3_8224_85a2271fe2c8.slice/crio-2d59242d83498304ef4f642eb37875ea86b65645ee51daf3302e0b8bdc5fd2ef WatchSource:0}: Error finding container 2d59242d83498304ef4f642eb37875ea86b65645ee51daf3302e0b8bdc5fd2ef: Status 404 returned error can't find the container with id 2d59242d83498304ef4f642eb37875ea86b65645ee51daf3302e0b8bdc5fd2ef
Oct 30 21:26:37.144107 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:37.144077    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78d8a57a-mnsmx"]
Oct 30 21:26:37.145907 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:26:37.145873    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99143411_6bbc_483e_9c3d_002cfce1efdb.slice/crio-4974dbba9738380555a2257f40fac0043d6f6293a8f27a6221ae2027035c6c0a WatchSource:0}: Error finding container 4974dbba9738380555a2257f40fac0043d6f6293a8f27a6221ae2027035c6c0a: Status 404 returned error can't find the container with id 4974dbba9738380555a2257f40fac0043d6f6293a8f27a6221ae2027035c6c0a
Oct 30 21:26:37.909275 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:37.909240    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78d8a57a-mnsmx" event={"ID":"99143411-6bbc-483e-9c3d-002cfce1efdb","Type":"ContainerStarted","Data":"b05c642c9c36ea9ef02a6b3cdf72e43519352e6d619c8195ea8496348a324c25"}
Oct 30 21:26:37.909445 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:37.909344    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78d8a57a-mnsmx" event={"ID":"99143411-6bbc-483e-9c3d-002cfce1efdb","Type":"ContainerStarted","Data":"4974dbba9738380555a2257f40fac0043d6f6293a8f27a6221ae2027035c6c0a"}
Oct 30 21:26:37.911504 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:37.911477    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd3daaa-s5ctm" event={"ID":"18d880ed-6e69-43b3-8224-85a2271fe2c8","Type":"ContainerStarted","Data":"f7fa31b5ecd37ec82365c2d45e7d7754843670d9f674029a53ad46b2a19c45d5"}
Oct 30 21:26:37.911593 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:37.911524    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd3daaa-s5ctm" event={"ID":"18d880ed-6e69-43b3-8224-85a2271fe2c8","Type":"ContainerStarted","Data":"2d59242d83498304ef4f642eb37875ea86b65645ee51daf3302e0b8bdc5fd2ef"}
Oct 30 21:26:38.023603 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:38.023555    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78d8a57a-mnsmx" podStartSLOduration=2.023541492 podStartE2EDuration="2.023541492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:38.021657167 +0000 UTC m=+2881.086433594" watchObservedRunningTime="2026-10-30 21:26:38.023541492 +0000 UTC m=+2881.088317890"
Oct 30 21:26:38.919115 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:38.919078    5713 generic.go:356] "Generic (PLEG): container finished" podID="18d880ed-6e69-43b3-8224-85a2271fe2c8" containerID="f7fa31b5ecd37ec82365c2d45e7d7754843670d9f674029a53ad46b2a19c45d5" exitCode=0
Oct 30 21:26:38.919601 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:38.919172    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd3daaa-s5ctm" event={"ID":"18d880ed-6e69-43b3-8224-85a2271fe2c8","Type":"ContainerDied","Data":"f7fa31b5ecd37ec82365c2d45e7d7754843670d9f674029a53ad46b2a19c45d5"}
Oct 30 21:26:38.921044 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:38.921018    5713 generic.go:356] "Generic (PLEG): container finished" podID="99143411-6bbc-483e-9c3d-002cfce1efdb" containerID="b05c642c9c36ea9ef02a6b3cdf72e43519352e6d619c8195ea8496348a324c25" exitCode=0
Oct 30 21:26:38.921146 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:38.921053    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78d8a57a-mnsmx" event={"ID":"99143411-6bbc-483e-9c3d-002cfce1efdb","Type":"ContainerDied","Data":"b05c642c9c36ea9ef02a6b3cdf72e43519352e6d619c8195ea8496348a324c25"}
Oct 30 21:26:39.051427 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:39.051395    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd3daaa-s5ctm"
Oct 30 21:26:39.133107 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:39.133073    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18d880ed-6e69-43b3-8224-85a2271fe2c8-kube-api-access-5f24b\" (UniqueName: \"kubernetes.io/projected/18d880ed-6e69-43b3-8224-85a2271fe2c8-kube-api-access-5f24b\") pod \"18d880ed-6e69-43b3-8224-85a2271fe2c8\" (UID: \"18d880ed-6e69-43b3-8224-85a2271fe2c8\") "
Oct 30 21:26:39.133107 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:39.133109    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18d880ed-6e69-43b3-8224-85a2271fe2c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18d880ed-6e69-43b3-8224-85a2271fe2c8-maas-api-xks-ca\") pod \"18d880ed-6e69-43b3-8224-85a2271fe2c8\" (UID: \"18d880ed-6e69-43b3-8224-85a2271fe2c8\") "
Oct 30 21:26:39.133345 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:39.133190    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18d880ed-6e69-43b3-8224-85a2271fe2c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18d880ed-6e69-43b3-8224-85a2271fe2c8-maas-api-openshift-ca\") pod \"18d880ed-6e69-43b3-8224-85a2271fe2c8\" (UID: \"18d880ed-6e69-43b3-8224-85a2271fe2c8\") "
Oct 30 21:26:39.133573 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:39.133548    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18d880ed-6e69-43b3-8224-85a2271fe2c8-maas-api-openshift-ca" pod "18d880ed-6e69-43b3-8224-85a2271fe2c8" (UID: "18d880ed-6e69-43b3-8224-85a2271fe2c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:26:39.135200 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:39.135175    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18d880ed-6e69-43b3-8224-85a2271fe2c8-maas-api-xks-ca" pod "18d880ed-6e69-43b3-8224-85a2271fe2c8" (UID: "18d880ed-6e69-43b3-8224-85a2271fe2c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:26:39.135260 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:39.135243    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18d880ed-6e69-43b3-8224-85a2271fe2c8-kube-api-access-5f24b" pod "18d880ed-6e69-43b3-8224-85a2271fe2c8" (UID: "18d880ed-6e69-43b3-8224-85a2271fe2c8"). InnerVolumeSpecName "kube-api-access-5f24b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:26:39.234345 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:39.234309    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5f24b\" (UniqueName: \"kubernetes.io/projected/18d880ed-6e69-43b3-8224-85a2271fe2c8-kube-api-access-5f24b\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:26:39.234345 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:39.234339    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18d880ed-6e69-43b3-8224-85a2271fe2c8-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:26:39.234345 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:39.234350    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18d880ed-6e69-43b3-8224-85a2271fe2c8-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:26:39.853238 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:39.853205    5713 scope.go:122] "RemoveContainer" containerID="3dcd1ac62cd444f42beebbfbb703c50a96bbe8717f8d95b016c5ac63f3843f86"
Oct 30 21:26:39.928565 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:39.928530    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd3daaa-s5ctm"
Oct 30 21:26:39.929028 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:39.928565    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cd3daaa-s5ctm" event={"ID":"18d880ed-6e69-43b3-8224-85a2271fe2c8","Type":"ContainerDied","Data":"2d59242d83498304ef4f642eb37875ea86b65645ee51daf3302e0b8bdc5fd2ef"}
Oct 30 21:26:39.929028 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:39.928612    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d59242d83498304ef4f642eb37875ea86b65645ee51daf3302e0b8bdc5fd2ef"
Oct 30 21:26:40.067590 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:40.067560    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78d8a57a-mnsmx"
Oct 30 21:26:40.142243 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:40.142161    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99143411-6bbc-483e-9c3d-002cfce1efdb-kube-api-access-cd64z\" (UniqueName: \"kubernetes.io/projected/99143411-6bbc-483e-9c3d-002cfce1efdb-kube-api-access-cd64z\") pod \"99143411-6bbc-483e-9c3d-002cfce1efdb\" (UID: \"99143411-6bbc-483e-9c3d-002cfce1efdb\") "
Oct 30 21:26:40.142421 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:40.142254    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99143411-6bbc-483e-9c3d-002cfce1efdb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99143411-6bbc-483e-9c3d-002cfce1efdb-maas-api-xks-ca\") pod \"99143411-6bbc-483e-9c3d-002cfce1efdb\" (UID: \"99143411-6bbc-483e-9c3d-002cfce1efdb\") "
Oct 30 21:26:40.142421 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:40.142327    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99143411-6bbc-483e-9c3d-002cfce1efdb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99143411-6bbc-483e-9c3d-002cfce1efdb-maas-api-openshift-ca\") pod \"99143411-6bbc-483e-9c3d-002cfce1efdb\" (UID: \"99143411-6bbc-483e-9c3d-002cfce1efdb\") "
Oct 30 21:26:40.142666 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:40.142632    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99143411-6bbc-483e-9c3d-002cfce1efdb-maas-api-openshift-ca" pod "99143411-6bbc-483e-9c3d-002cfce1efdb" (UID: "99143411-6bbc-483e-9c3d-002cfce1efdb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:26:40.142846 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:40.142759    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99143411-6bbc-483e-9c3d-002cfce1efdb-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:26:40.144614 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:40.144590    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99143411-6bbc-483e-9c3d-002cfce1efdb-maas-api-xks-ca" pod "99143411-6bbc-483e-9c3d-002cfce1efdb" (UID: "99143411-6bbc-483e-9c3d-002cfce1efdb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:26:40.144709 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:40.144662    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99143411-6bbc-483e-9c3d-002cfce1efdb-kube-api-access-cd64z" pod "99143411-6bbc-483e-9c3d-002cfce1efdb" (UID: "99143411-6bbc-483e-9c3d-002cfce1efdb"). InnerVolumeSpecName "kube-api-access-cd64z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:26:40.244261 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:40.244226    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99143411-6bbc-483e-9c3d-002cfce1efdb-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:26:40.244261 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:40.244255    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cd64z\" (UniqueName: \"kubernetes.io/projected/99143411-6bbc-483e-9c3d-002cfce1efdb-kube-api-access-cd64z\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:26:40.642084 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:40.642041    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-1fdaa7-6fcb4879fd-bcz9f" podUID="5304445a-544a-4cb7-b79a-53502f94fe48" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:26:40.642084 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:40.642060    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1fdaa7-bb8847cd7-kq2ml" podUID="70aef1ff-8862-40b8-b3b7-c8878df1619f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:26:40.935321 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:40.935292    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78d8a57a-mnsmx"
Oct 30 21:26:40.935735 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:40.935333    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78d8a57a-mnsmx" event={"ID":"99143411-6bbc-483e-9c3d-002cfce1efdb","Type":"ContainerDied","Data":"4974dbba9738380555a2257f40fac0043d6f6293a8f27a6221ae2027035c6c0a"}
Oct 30 21:26:40.935735 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:40.935381    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4974dbba9738380555a2257f40fac0043d6f6293a8f27a6221ae2027035c6c0a"
Oct 30 21:26:50.641289 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:50.641194    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1fdaa7-bb8847cd7-kq2ml" podUID="70aef1ff-8862-40b8-b3b7-c8878df1619f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:26:50.641289 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:50.641252    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-1fdaa7-6fcb4879fd-bcz9f" podUID="5304445a-544a-4cb7-b79a-53502f94fe48" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:26:53.868759 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:53.868717    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1fdaa7-m9jds"]
Oct 30 21:26:53.917908 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:53.917878    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1fdaa7-m9jds"]
Oct 30 21:26:54.458398 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:26:54.458359    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:26:55.061906 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:55.061872    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="547b9a53-5c3d-487b-ad2c-eabcb6c71256" path="/var/lib/kubelet/pods/547b9a53-5c3d-487b-ad2c-eabcb6c71256/volumes"
Oct 30 21:26:56.032672 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:56.032636    5713 generic.go:356] "Generic (PLEG): container finished" podID="5304445a-544a-4cb7-b79a-53502f94fe48" containerID="beca861051be812f67a97f823851e310d809f206f4bf724e896593cd93e3a955" exitCode=0
Oct 30 21:26:56.032856 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:56.032720    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-1fdaa7-6fcb4879fd-bcz9f" event={"ID":"5304445a-544a-4cb7-b79a-53502f94fe48","Type":"ContainerDied","Data":"beca861051be812f67a97f823851e310d809f206f4bf724e896593cd93e3a955"}
Oct 30 21:26:56.032856 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:56.032825    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-1fdaa7-6fcb4879fd-bcz9f" event={"ID":"5304445a-544a-4cb7-b79a-53502f94fe48","Type":"ContainerDied","Data":"0f7df3d9f7bc13f1ef94b83bc06a0a9dc901a25c95326f4838cc1e8ccfdb985f"}
Oct 30 21:26:56.032856 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:56.032851    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f7df3d9f7bc13f1ef94b83bc06a0a9dc901a25c95326f4838cc1e8ccfdb985f"
Oct 30 21:26:56.034688 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:56.034666    5713 generic.go:356] "Generic (PLEG): container finished" podID="70aef1ff-8862-40b8-b3b7-c8878df1619f" containerID="2f46b08c16b08a4e8cb84d4301eab1832cb7bee4fbf766401131eab9983ac404" exitCode=0
Oct 30 21:26:56.034816 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:56.034735    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1fdaa7-bb8847cd7-kq2ml" event={"ID":"70aef1ff-8862-40b8-b3b7-c8878df1619f","Type":"ContainerDied","Data":"2f46b08c16b08a4e8cb84d4301eab1832cb7bee4fbf766401131eab9983ac404"}
Oct 30 21:26:56.044631 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:56.044508    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-1fdaa7-6fcb4879fd-bcz9f"
Oct 30 21:26:56.156636 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:56.156608    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1fdaa7-bb8847cd7-kq2ml"
Oct 30 21:26:56.197930 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:56.197891    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5304445a-544a-4cb7-b79a-53502f94fe48-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5304445a-544a-4cb7-b79a-53502f94fe48-plugins-config-volume\") pod \"5304445a-544a-4cb7-b79a-53502f94fe48\" (UID: \"5304445a-544a-4cb7-b79a-53502f94fe48\") "
Oct 30 21:26:56.198135 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:56.197998    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5304445a-544a-4cb7-b79a-53502f94fe48-kube-api-access-msdps\" (UniqueName: \"kubernetes.io/projected/5304445a-544a-4cb7-b79a-53502f94fe48-kube-api-access-msdps\") pod \"5304445a-544a-4cb7-b79a-53502f94fe48\" (UID: \"5304445a-544a-4cb7-b79a-53502f94fe48\") "
Oct 30 21:26:56.198875 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:56.198341    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5304445a-544a-4cb7-b79a-53502f94fe48-plugins-config-volume" pod "5304445a-544a-4cb7-b79a-53502f94fe48" (UID: "5304445a-544a-4cb7-b79a-53502f94fe48"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:26:56.198875 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:56.198752    5713 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5304445a-544a-4cb7-b79a-53502f94fe48-plugins-config-volume\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:26:56.200406 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:56.200384    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5304445a-544a-4cb7-b79a-53502f94fe48-kube-api-access-msdps" pod "5304445a-544a-4cb7-b79a-53502f94fe48" (UID: "5304445a-544a-4cb7-b79a-53502f94fe48"). InnerVolumeSpecName "kube-api-access-msdps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:26:56.300173 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:56.300077    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70aef1ff-8862-40b8-b3b7-c8878df1619f-kube-api-access-ph4xw\" (UniqueName: \"kubernetes.io/projected/70aef1ff-8862-40b8-b3b7-c8878df1619f-kube-api-access-ph4xw\") pod \"70aef1ff-8862-40b8-b3b7-c8878df1619f\" (UID: \"70aef1ff-8862-40b8-b3b7-c8878df1619f\") "
Oct 30 21:26:56.300173 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:56.300137    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70aef1ff-8862-40b8-b3b7-c8878df1619f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70aef1ff-8862-40b8-b3b7-c8878df1619f-plugins-config-volume\") pod \"70aef1ff-8862-40b8-b3b7-c8878df1619f\" (UID: \"70aef1ff-8862-40b8-b3b7-c8878df1619f\") "
Oct 30 21:26:56.300499 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:56.300480    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msdps\" (UniqueName: \"kubernetes.io/projected/5304445a-544a-4cb7-b79a-53502f94fe48-kube-api-access-msdps\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:26:56.300619 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:56.300589    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70aef1ff-8862-40b8-b3b7-c8878df1619f-plugins-config-volume" pod "70aef1ff-8862-40b8-b3b7-c8878df1619f" (UID: "70aef1ff-8862-40b8-b3b7-c8878df1619f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:26:56.302344 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:56.302322    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70aef1ff-8862-40b8-b3b7-c8878df1619f-kube-api-access-ph4xw" pod "70aef1ff-8862-40b8-b3b7-c8878df1619f" (UID: "70aef1ff-8862-40b8-b3b7-c8878df1619f"). InnerVolumeSpecName "kube-api-access-ph4xw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:26:56.401675 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:56.401633    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph4xw\" (UniqueName: \"kubernetes.io/projected/70aef1ff-8862-40b8-b3b7-c8878df1619f-kube-api-access-ph4xw\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:26:56.401675 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:56.401669    5713 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70aef1ff-8862-40b8-b3b7-c8878df1619f-plugins-config-volume\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:26:57.042499 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:57.042474    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1fdaa7-bb8847cd7-kq2ml"
Oct 30 21:26:57.042499 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:57.042492    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-1fdaa7-bb8847cd7-kq2ml" event={"ID":"70aef1ff-8862-40b8-b3b7-c8878df1619f","Type":"ContainerDied","Data":"459a351f71410b9142c4404d4a69248a2c2b831178a53897d0183bb6bc7f6353"}
Oct 30 21:26:57.042768 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:57.042538    5713 scope.go:122] "RemoveContainer" containerID="2f46b08c16b08a4e8cb84d4301eab1832cb7bee4fbf766401131eab9983ac404"
Oct 30 21:26:57.042768 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:57.042477    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-1fdaa7-6fcb4879fd-bcz9f"
Oct 30 21:26:57.262902 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:57.262869    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-1fdaa7-6fcb4879fd-bcz9f"]
Oct 30 21:26:57.312203 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:57.312123    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-1fdaa7-6fcb4879fd-bcz9f"]
Oct 30 21:26:57.372473 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:57.372441    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-1fdaa7-bb8847cd7-kq2ml"]
Oct 30 21:26:57.376277 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:57.376243    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-1fdaa7-bb8847cd7-kq2ml"]
Oct 30 21:26:59.061163 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:59.061132    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5304445a-544a-4cb7-b79a-53502f94fe48" path="/var/lib/kubelet/pods/5304445a-544a-4cb7-b79a-53502f94fe48/volumes"
Oct 30 21:26:59.061551 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:26:59.061504    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70aef1ff-8862-40b8-b3b7-c8878df1619f" path="/var/lib/kubelet/pods/70aef1ff-8862-40b8-b3b7-c8878df1619f/volumes"
Oct 30 21:27:24.457820 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:27:24.457755    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:27:25.013632 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:25.013600    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8nzgh"]
Oct 30 21:27:25.014057 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:25.014041    5713 cpu_manager.go:414] "containerMap: removing container" podUID="70aef1ff-8862-40b8-b3b7-c8878df1619f" containerName="payload-pre-processing"
Oct 30 21:27:25.014112 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:25.014060    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70aef1ff-8862-40b8-b3b7-c8878df1619f" containerName="payload-pre-processing"
Oct 30 21:27:25.014112 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:25.014086    5713 cpu_manager.go:414] "containerMap: removing container" podUID="5304445a-544a-4cb7-b79a-53502f94fe48" containerName="payload-processing"
Oct 30 21:27:25.014112 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:25.014091    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5304445a-544a-4cb7-b79a-53502f94fe48" containerName="payload-processing"
Oct 30 21:27:25.014112 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:25.014105    5713 cpu_manager.go:414] "containerMap: removing container" podUID="99143411-6bbc-483e-9c3d-002cfce1efdb" containerName="revoke-keys"
Oct 30 21:27:25.014112 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:25.014110    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99143411-6bbc-483e-9c3d-002cfce1efdb" containerName="revoke-keys"
Oct 30 21:27:25.014280 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:25.014126    5713 cpu_manager.go:414] "containerMap: removing container" podUID="18d880ed-6e69-43b3-8224-85a2271fe2c8" containerName="revoke-keys"
Oct 30 21:27:25.014280 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:25.014134    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18d880ed-6e69-43b3-8224-85a2271fe2c8" containerName="revoke-keys"
Oct 30 21:27:25.014280 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:25.014195    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="18d880ed-6e69-43b3-8224-85a2271fe2c8" containerName="revoke-keys"
Oct 30 21:27:25.014280 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:25.014207    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="70aef1ff-8862-40b8-b3b7-c8878df1619f" containerName="payload-pre-processing"
Oct 30 21:27:25.014280 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:25.014222    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="5304445a-544a-4cb7-b79a-53502f94fe48" containerName="payload-processing"
Oct 30 21:27:25.014280 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:25.014231    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="99143411-6bbc-483e-9c3d-002cfce1efdb" containerName="revoke-keys"
Oct 30 21:27:25.017776 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:25.017750    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8nzgh"
Oct 30 21:27:25.073682 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:25.073646    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d91e8f40-c2d6-455c-8d73-9f83b615f3dd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-8nzgh\" (UID: \"d91e8f40-c2d6-455c-8d73-9f83b615f3dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8nzgh"
Oct 30 21:27:25.073871 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:25.073690    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91e8f40-c2d6-455c-8d73-9f83b615f3dd-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-8nzgh\" (UID: \"d91e8f40-c2d6-455c-8d73-9f83b615f3dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8nzgh"
Oct 30 21:27:25.073917 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:25.073878    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d91e8f40-c2d6-455c-8d73-9f83b615f3dd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-8nzgh\" (UID: \"d91e8f40-c2d6-455c-8d73-9f83b615f3dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8nzgh"
Oct 30 21:27:25.073955 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:25.073921    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d91e8f40-c2d6-455c-8d73-9f83b615f3dd-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-8nzgh\" (UID: \"d91e8f40-c2d6-455c-8d73-9f83b615f3dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8nzgh"
Oct 30 21:27:25.073992 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:25.073966    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d91e8f40-c2d6-455c-8d73-9f83b615f3dd-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-8nzgh\" (UID: \"d91e8f40-c2d6-455c-8d73-9f83b615f3dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8nzgh"
Oct 30 21:27:25.074053 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:25.074028    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91e8f40-c2d6-455c-8d73-9f83b615f3dd-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-8nzgh\" (UID: \"d91e8f40-c2d6-455c-8d73-9f83b615f3dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8nzgh"
Oct 30 21:27:25.074095 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:25.074073    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d91e8f40-c2d6-455c-8d73-9f83b615f3dd-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-8nzgh\" (UID: \"d91e8f40-c2d6-455c-8d73-9f83b615f3dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8nzgh"
Oct 30 21:27:25.074095 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:25.074089    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwmw5\" (UniqueName: \"kubernetes.io/projected/d91e8f40-c2d6-455c-8d73-9f83b615f3dd-kube-api-access-bwmw5\") pod \"maas-default-gateway-openshift-default-769df7846c-8nzgh\" (UID: \"d91e8f40-c2d6-455c-8d73-9f83b615f3dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8nzgh"
Oct 30 21:27:25.074177 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:25.074165    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d91e8f40-c2d6-455c-8d73-9f83b615f3dd-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-8nzgh\" (UID: \"d91e8f40-c2d6-455c-8d73-9f83b615f3dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8nzgh"
Oct 30 21:27:25.175616 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:25.175580    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91e8f40-c2d6-455c-8d73-9f83b615f3dd-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-8nzgh\" (UID: \"d91e8f40-c2d6-455c-8d73-9f83b615f3dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8nzgh"
Oct 30 21:27:25.175880 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:25.175853    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d91e8f40-c2d6-455c-8d73-9f83b615f3dd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-8nzgh\" (UID: \"d91e8f40-c2d6-455c-8d73-9f83b615f3dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8nzgh"
Oct 30 21:27:25.175987 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:25.175906    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d91e8f40-c2d6-455c-8d73-9f83b615f3dd-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-8nzgh\" (UID: \"d91e8f40-c2d6-455c-8d73-9f83b615f3dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8nzgh"
Oct 30 21:27:25.175987 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:25.175950    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d91e8f40-c2d6-455c-8d73-9f83b615f3dd-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-8nzgh\" (UID: \"d91e8f40-c2d6-455c-8d73-9f83b615f3dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8nzgh"
Oct 30 21:27:25.176115 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:25.176018    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91e8f40-c2d6-455c-8d73-9f83b615f3dd-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-8nzgh\" (UID: \"d91e8f40-c2d6-455c-8d73-9f83b615f3dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8nzgh"
Oct 30 21:27:25.176115 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:25.176015    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91e8f40-c2d6-455c-8d73-9f83b615f3dd-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-8nzgh\" (UID: \"d91e8f40-c2d6-455c-8d73-9f83b615f3dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8nzgh"
Oct 30 21:27:25.176115 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:25.176104    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d91e8f40-c2d6-455c-8d73-9f83b615f3dd-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-8nzgh\" (UID: \"d91e8f40-c2d6-455c-8d73-9f83b615f3dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8nzgh"
Oct 30 21:27:25.176303 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:25.176133    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwmw5\" (UniqueName: \"kubernetes.io/projected/d91e8f40-c2d6-455c-8d73-9f83b615f3dd-kube-api-access-bwmw5\") pod \"maas-default-gateway-openshift-default-769df7846c-8nzgh\" (UID: \"d91e8f40-c2d6-455c-8d73-9f83b615f3dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8nzgh"
Oct 30 21:27:25.176303 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:25.176188    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d91e8f40-c2d6-455c-8d73-9f83b615f3dd-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-8nzgh\" (UID: \"d91e8f40-c2d6-455c-8d73-9f83b615f3dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8nzgh"
Oct 30 21:27:25.176303 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:25.176230    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d91e8f40-c2d6-455c-8d73-9f83b615f3dd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-8nzgh\" (UID: \"d91e8f40-c2d6-455c-8d73-9f83b615f3dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8nzgh"
Oct 30 21:27:25.176444 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:25.176317    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d91e8f40-c2d6-455c-8d73-9f83b615f3dd-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-8nzgh\" (UID: \"d91e8f40-c2d6-455c-8d73-9f83b615f3dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8nzgh"
Oct 30 21:27:25.176444 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:25.176354    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91e8f40-c2d6-455c-8d73-9f83b615f3dd-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-8nzgh\" (UID: \"d91e8f40-c2d6-455c-8d73-9f83b615f3dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8nzgh"
Oct 30 21:27:25.176531 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:25.176513    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d91e8f40-c2d6-455c-8d73-9f83b615f3dd-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-8nzgh\" (UID: \"d91e8f40-c2d6-455c-8d73-9f83b615f3dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8nzgh"
Oct 30 21:27:25.177192 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:25.177164    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d91e8f40-c2d6-455c-8d73-9f83b615f3dd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-8nzgh\" (UID: \"d91e8f40-c2d6-455c-8d73-9f83b615f3dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8nzgh"
Oct 30 21:27:25.178556 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:25.178534    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d91e8f40-c2d6-455c-8d73-9f83b615f3dd-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-8nzgh\" (UID: \"d91e8f40-c2d6-455c-8d73-9f83b615f3dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8nzgh"
Oct 30 21:27:25.178672 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:25.178654    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d91e8f40-c2d6-455c-8d73-9f83b615f3dd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-8nzgh\" (UID: \"d91e8f40-c2d6-455c-8d73-9f83b615f3dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8nzgh"
Oct 30 21:27:25.221556 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:25.221519    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8nzgh"]
Oct 30 21:27:25.233225 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:25.233193    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d91e8f40-c2d6-455c-8d73-9f83b615f3dd-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-8nzgh\" (UID: \"d91e8f40-c2d6-455c-8d73-9f83b615f3dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8nzgh"
Oct 30 21:27:25.233373 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:25.233241    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwmw5\" (UniqueName: \"kubernetes.io/projected/d91e8f40-c2d6-455c-8d73-9f83b615f3dd-kube-api-access-bwmw5\") pod \"maas-default-gateway-openshift-default-769df7846c-8nzgh\" (UID: \"d91e8f40-c2d6-455c-8d73-9f83b615f3dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8nzgh"
Oct 30 21:27:25.331104 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:25.330991    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8nzgh"
Oct 30 21:27:25.760895 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:25.760846    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8nzgh"]
Oct 30 21:27:25.763426 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:27:25.763400    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd91e8f40_c2d6_455c_8d73_9f83b615f3dd.slice/crio-9f7fa730853cb757c1e4d2c0b0d4985a97c7512746f5529f320eeed2bc4d0cba WatchSource:0}: Error finding container 9f7fa730853cb757c1e4d2c0b0d4985a97c7512746f5529f320eeed2bc4d0cba: Status 404 returned error can't find the container with id 9f7fa730853cb757c1e4d2c0b0d4985a97c7512746f5529f320eeed2bc4d0cba
Oct 30 21:27:25.765637 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:25.765605    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:27:25.765752 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:25.765687    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:27:25.765752 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:25.765728    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:27:26.225852 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:26.225774    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8nzgh" event={"ID":"d91e8f40-c2d6-455c-8d73-9f83b615f3dd","Type":"ContainerStarted","Data":"84529d9247eec13f8f11a736b6bb7728e222f2789a74c269e7420cfedb18932c"}
Oct 30 21:27:26.226047 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:26.225881    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8nzgh" event={"ID":"d91e8f40-c2d6-455c-8d73-9f83b615f3dd","Type":"ContainerStarted","Data":"9f7fa730853cb757c1e4d2c0b0d4985a97c7512746f5529f320eeed2bc4d0cba"}
Oct 30 21:27:26.331501 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:26.331471    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8nzgh"
Oct 30 21:27:26.333122 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:26.333097    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-8nzgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body=
Oct 30 21:27:26.333269 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:26.333143    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8nzgh" podUID="d91e8f40-c2d6-455c-8d73-9f83b615f3dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused"
Oct 30 21:27:26.356813 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:26.355973    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8nzgh" podStartSLOduration=2.355937967 podStartE2EDuration="2.355937967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:26.354904274 +0000 UTC m=+2929.419680673" watchObservedRunningTime="2026-10-30 21:27:26.355937967 +0000 UTC m=+2929.420714367"
Oct 30 21:27:27.331860 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:27.331778    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-8nzgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body=
Oct 30 21:27:27.332289 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:27.331876    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8nzgh" podUID="d91e8f40-c2d6-455c-8d73-9f83b615f3dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused"
Oct 30 21:27:28.053208 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:28.053169    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27240488-frzll"]
Oct 30 21:27:28.060841 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:28.060809    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27240488-frzll"]
Oct 30 21:27:28.332335 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:28.332261    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-8nzgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body=
Oct 30 21:27:28.332335 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:28.332314    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8nzgh" podUID="d91e8f40-c2d6-455c-8d73-9f83b615f3dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused"
Oct 30 21:27:29.064317 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:29.064287    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea3af8c8-8a0e-4e42-8d3e-40d59316fb49" path="/var/lib/kubelet/pods/ea3af8c8-8a0e-4e42-8d3e-40d59316fb49/volumes"
Oct 30 21:27:29.331863 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:29.331755    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-8nzgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body=
Oct 30 21:27:29.332035 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:29.331852    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8nzgh" podUID="d91e8f40-c2d6-455c-8d73-9f83b615f3dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused"
Oct 30 21:27:30.332081 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:30.332048    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-8nzgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body=
Oct 30 21:27:30.332545 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:30.332108    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8nzgh" podUID="d91e8f40-c2d6-455c-8d73-9f83b615f3dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused"
Oct 30 21:27:31.332101 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:31.332059    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-8nzgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body=
Oct 30 21:27:31.332539 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:31.332123    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8nzgh" podUID="d91e8f40-c2d6-455c-8d73-9f83b615f3dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused"
Oct 30 21:27:32.331806 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:32.331745    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-8nzgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body=
Oct 30 21:27:32.332007 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:32.331853    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8nzgh" podUID="d91e8f40-c2d6-455c-8d73-9f83b615f3dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused"
Oct 30 21:27:34.094322 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:34.094286    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8nzgh"
Oct 30 21:27:34.095067 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:34.094751    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8nzgh"
Oct 30 21:27:34.095697 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:34.095674    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8nzgh"
Oct 30 21:27:39.893644 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:39.893613    5713 scope.go:122] "RemoveContainer" containerID="5f3cde77f50f67fc03f49933eb6a2e86ec9be29add90c98e9dda321aaf6951eb"
Oct 30 21:27:39.903131 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:39.903109    5713 scope.go:122] "RemoveContainer" containerID="beca861051be812f67a97f823851e310d809f206f4bf724e896593cd93e3a955"
Oct 30 21:27:40.132822 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:40.132774    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-fdp2g"]
Oct 30 21:27:40.137854 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:40.137829    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-fdp2g"
Oct 30 21:27:40.242135 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:40.242106    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-fdp2g"]
Oct 30 21:27:40.253331 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:40.253293    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b905ceb0-d3ed-426a-90fe-3717158cee72-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-fdp2g\" (UID: \"b905ceb0-d3ed-426a-90fe-3717158cee72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-fdp2g"
Oct 30 21:27:40.253473 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:40.253440    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5vfp\" (UniqueName: \"kubernetes.io/projected/b905ceb0-d3ed-426a-90fe-3717158cee72-kube-api-access-k5vfp\") pod \"maas-default-gateway-openshift-default-769df7846c-fdp2g\" (UID: \"b905ceb0-d3ed-426a-90fe-3717158cee72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-fdp2g"
Oct 30 21:27:40.253520 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:40.253482    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b905ceb0-d3ed-426a-90fe-3717158cee72-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-fdp2g\" (UID: \"b905ceb0-d3ed-426a-90fe-3717158cee72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-fdp2g"
Oct 30 21:27:40.253644 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:40.253622    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b905ceb0-d3ed-426a-90fe-3717158cee72-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-fdp2g\" (UID: \"b905ceb0-d3ed-426a-90fe-3717158cee72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-fdp2g"
Oct 30 21:27:40.253685 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:40.253654    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b905ceb0-d3ed-426a-90fe-3717158cee72-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-fdp2g\" (UID: \"b905ceb0-d3ed-426a-90fe-3717158cee72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-fdp2g"
Oct 30 21:27:40.253685 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:40.253672    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b905ceb0-d3ed-426a-90fe-3717158cee72-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-fdp2g\" (UID: \"b905ceb0-d3ed-426a-90fe-3717158cee72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-fdp2g"
Oct 30 21:27:40.253753 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:40.253691    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b905ceb0-d3ed-426a-90fe-3717158cee72-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-fdp2g\" (UID: \"b905ceb0-d3ed-426a-90fe-3717158cee72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-fdp2g"
Oct 30 21:27:40.253839 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:40.253817    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b905ceb0-d3ed-426a-90fe-3717158cee72-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-fdp2g\" (UID: \"b905ceb0-d3ed-426a-90fe-3717158cee72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-fdp2g"
Oct 30 21:27:40.253884 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:40.253864    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b905ceb0-d3ed-426a-90fe-3717158cee72-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-fdp2g\" (UID: \"b905ceb0-d3ed-426a-90fe-3717158cee72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-fdp2g"
Oct 30 21:27:40.354734 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:40.354686    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5vfp\" (UniqueName: \"kubernetes.io/projected/b905ceb0-d3ed-426a-90fe-3717158cee72-kube-api-access-k5vfp\") pod \"maas-default-gateway-openshift-default-769df7846c-fdp2g\" (UID: \"b905ceb0-d3ed-426a-90fe-3717158cee72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-fdp2g"
Oct 30 21:27:40.354734 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:40.354736    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b905ceb0-d3ed-426a-90fe-3717158cee72-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-fdp2g\" (UID: \"b905ceb0-d3ed-426a-90fe-3717158cee72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-fdp2g"
Oct 30 21:27:40.355018 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:40.354810    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b905ceb0-d3ed-426a-90fe-3717158cee72-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-fdp2g\" (UID: \"b905ceb0-d3ed-426a-90fe-3717158cee72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-fdp2g"
Oct 30 21:27:40.355018 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:40.354837    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b905ceb0-d3ed-426a-90fe-3717158cee72-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-fdp2g\" (UID: \"b905ceb0-d3ed-426a-90fe-3717158cee72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-fdp2g"
Oct 30 21:27:40.355018 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:40.354872    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b905ceb0-d3ed-426a-90fe-3717158cee72-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-fdp2g\" (UID: \"b905ceb0-d3ed-426a-90fe-3717158cee72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-fdp2g"
Oct 30 21:27:40.355018 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:40.354902    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b905ceb0-d3ed-426a-90fe-3717158cee72-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-fdp2g\" (UID: \"b905ceb0-d3ed-426a-90fe-3717158cee72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-fdp2g"
Oct 30 21:27:40.355018 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:40.354953    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b905ceb0-d3ed-426a-90fe-3717158cee72-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-fdp2g\" (UID: \"b905ceb0-d3ed-426a-90fe-3717158cee72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-fdp2g"
Oct 30 21:27:40.355018 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:40.354980    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b905ceb0-d3ed-426a-90fe-3717158cee72-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-fdp2g\" (UID: \"b905ceb0-d3ed-426a-90fe-3717158cee72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-fdp2g"
Oct 30 21:27:40.355425 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:40.355027    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b905ceb0-d3ed-426a-90fe-3717158cee72-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-fdp2g\" (UID: \"b905ceb0-d3ed-426a-90fe-3717158cee72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-fdp2g"
Oct 30 21:27:40.355425 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:40.355335    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b905ceb0-d3ed-426a-90fe-3717158cee72-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-fdp2g\" (UID: \"b905ceb0-d3ed-426a-90fe-3717158cee72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-fdp2g"
Oct 30 21:27:40.355425 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:40.355393    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b905ceb0-d3ed-426a-90fe-3717158cee72-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-fdp2g\" (UID: \"b905ceb0-d3ed-426a-90fe-3717158cee72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-fdp2g"
Oct 30 21:27:40.355567 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:40.355440    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b905ceb0-d3ed-426a-90fe-3717158cee72-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-fdp2g\" (UID: \"b905ceb0-d3ed-426a-90fe-3717158cee72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-fdp2g"
Oct 30 21:27:40.355567 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:40.355476    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b905ceb0-d3ed-426a-90fe-3717158cee72-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-fdp2g\" (UID: \"b905ceb0-d3ed-426a-90fe-3717158cee72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-fdp2g"
Oct 30 21:27:40.355706 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:40.355684    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b905ceb0-d3ed-426a-90fe-3717158cee72-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-fdp2g\" (UID: \"b905ceb0-d3ed-426a-90fe-3717158cee72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-fdp2g"
Oct 30 21:27:40.357450 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:40.357427    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b905ceb0-d3ed-426a-90fe-3717158cee72-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-fdp2g\" (UID: \"b905ceb0-d3ed-426a-90fe-3717158cee72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-fdp2g"
Oct 30 21:27:40.357622 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:40.357598    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b905ceb0-d3ed-426a-90fe-3717158cee72-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-fdp2g\" (UID: \"b905ceb0-d3ed-426a-90fe-3717158cee72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-fdp2g"
Oct 30 21:27:40.411684 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:40.411613    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b905ceb0-d3ed-426a-90fe-3717158cee72-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-fdp2g\" (UID: \"b905ceb0-d3ed-426a-90fe-3717158cee72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-fdp2g"
Oct 30 21:27:40.411684 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:40.411653    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5vfp\" (UniqueName: \"kubernetes.io/projected/b905ceb0-d3ed-426a-90fe-3717158cee72-kube-api-access-k5vfp\") pod \"maas-default-gateway-openshift-default-769df7846c-fdp2g\" (UID: \"b905ceb0-d3ed-426a-90fe-3717158cee72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-fdp2g"
Oct 30 21:27:40.450553 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:40.450517    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-fdp2g"
Oct 30 21:27:40.882593 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:40.882566    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-fdp2g"]
Oct 30 21:27:40.884436 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:27:40.884396    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb905ceb0_d3ed_426a_90fe_3717158cee72.slice/crio-b1804adfd3d1282b3a7f5122b1dcf45ae46db3b0ce67bf0023727121e501c5c5 WatchSource:0}: Error finding container b1804adfd3d1282b3a7f5122b1dcf45ae46db3b0ce67bf0023727121e501c5c5: Status 404 returned error can't find the container with id b1804adfd3d1282b3a7f5122b1dcf45ae46db3b0ce67bf0023727121e501c5c5
Oct 30 21:27:40.886578 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:40.886542    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:27:40.886690 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:40.886621    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:27:40.886690 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:40.886664    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:27:41.046962 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:41.046928    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-114defe4-vktnw"]
Oct 30 21:27:41.050094 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:41.050063    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-114defe4-vktnw"]
Oct 30 21:27:41.063140 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:41.063108    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84380376-2a47-47d8-a71e-a7fa6355f0c3" path="/var/lib/kubelet/pods/84380376-2a47-47d8-a71e-a7fa6355f0c3/volumes"
Oct 30 21:27:41.365240 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:41.365196    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-fdp2g" event={"ID":"b905ceb0-d3ed-426a-90fe-3717158cee72","Type":"ContainerStarted","Data":"485e7e194e23b7f42dc6c29eb2e8556c3202ae4af2264aaa15e6f9359b07f034"}
Oct 30 21:27:41.365458 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:41.365256    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-fdp2g" event={"ID":"b905ceb0-d3ed-426a-90fe-3717158cee72","Type":"ContainerStarted","Data":"b1804adfd3d1282b3a7f5122b1dcf45ae46db3b0ce67bf0023727121e501c5c5"}
Oct 30 21:27:41.458421 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:41.456342    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-fdp2g"
Oct 30 21:27:41.458762 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:41.458733    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-fdp2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body=
Oct 30 21:27:41.459039 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:41.458979    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-fdp2g" podUID="b905ceb0-d3ed-426a-90fe-3717158cee72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused"
Oct 30 21:27:41.483298 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:41.483239    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-fdp2g" podStartSLOduration=1.48322459 podStartE2EDuration="1.48322459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:41.481290349 +0000 UTC m=+2944.546066760" watchObservedRunningTime="2026-10-30 21:27:41.48322459 +0000 UTC m=+2944.548000988"
Oct 30 21:27:42.452410 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:42.452370    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-fdp2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body=
Oct 30 21:27:42.453051 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:42.453015    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-fdp2g" podUID="b905ceb0-d3ed-426a-90fe-3717158cee72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused"
Oct 30 21:27:43.454815 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:43.453950    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-fdp2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body=
Oct 30 21:27:43.454815 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:43.454009    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-fdp2g" podUID="b905ceb0-d3ed-426a-90fe-3717158cee72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused"
Oct 30 21:27:44.455677 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:44.455213    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-fdp2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body=
Oct 30 21:27:44.455677 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:44.455281    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-fdp2g" podUID="b905ceb0-d3ed-426a-90fe-3717158cee72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused"
Oct 30 21:27:45.459293 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:45.459258    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-fdp2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body=
Oct 30 21:27:45.459824 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:45.459730    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-fdp2g" podUID="b905ceb0-d3ed-426a-90fe-3717158cee72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused"
Oct 30 21:27:46.452275 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:46.452234    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-fdp2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body=
Oct 30 21:27:46.452594 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:46.452565    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-fdp2g" podUID="b905ceb0-d3ed-426a-90fe-3717158cee72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused"
Oct 30 21:27:47.459764 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:47.459720    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-fdp2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body=
Oct 30 21:27:47.460452 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:47.460414    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-fdp2g" podUID="b905ceb0-d3ed-426a-90fe-3717158cee72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused"
Oct 30 21:27:49.201408 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:49.201366    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-fdp2g"
Oct 30 21:27:49.202103 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:49.202060    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-fdp2g"
Oct 30 21:27:49.203481 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:27:49.203459    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-fdp2g"
Oct 30 21:27:54.458885 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:27:54.458846    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:28:20.046231 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:28:20.046198    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c69b6088-mtrx4"]
Oct 30 21:28:20.095781 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:28:20.095754    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c69b6088-mtrx4"]
Oct 30 21:28:21.061783 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:28:21.061750    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4ecc41c-33fd-4558-bf79-e4ef840c0aa1" path="/var/lib/kubelet/pods/b4ecc41c-33fd-4558-bf79-e4ef840c0aa1/volumes"
Oct 30 21:28:24.457710 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:28:24.457672    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:28:37.758748 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:28:37.758715    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d4597073-8ff0-4a4e-933c-c64e934fe977/prometheus/0.log"
Oct 30 21:28:37.763490 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:28:37.763468    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d4597073-8ff0-4a4e-933c-c64e934fe977/prometheus/0.log"
Oct 30 21:28:37.773371 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:28:37.773339    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r7lpf_764461d5-e0b2-4856-a97e-0c5d3d34a497/ovn-acl-logging/0.log"
Oct 30 21:28:37.778588 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:28:37.778570    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r7lpf_764461d5-e0b2-4856-a97e-0c5d3d34a497/ovn-acl-logging/0.log"
Oct 30 21:28:39.934779 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:28:39.934740    5713 scope.go:122] "RemoveContainer" containerID="af280fe0f0ee44f1469d87fc0f1c8b2dd9a4550c37fb390f91c65a16a926207a"
Oct 30 21:28:39.944686 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:28:39.944668    5713 scope.go:122] "RemoveContainer" containerID="5f554b6a489a5bb6a34123459bbc03e4e83cc5b3e916373e5badcdf8af87fe8e"
Oct 30 21:28:52.045370 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:28:52.045324    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e547c79b-m84wh"]
Oct 30 21:28:52.095063 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:28:52.095029    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e547c79b-m84wh"]
Oct 30 21:28:53.064993 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:28:53.064953    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afaab19a-aaae-4a24-9e43-0a7e4594641d" path="/var/lib/kubelet/pods/afaab19a-aaae-4a24-9e43-0a7e4594641d/volumes"
Oct 30 21:28:54.457127 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:28:54.457089    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:29:09.733665 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:09.733626    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5xsnk"]
Oct 30 21:29:09.737381 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:09.737361    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5xsnk"
Oct 30 21:29:09.816874 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:09.816838    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74888553-9594-4086-9148-8d788092c3fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-5xsnk\" (UID: \"74888553-9594-4086-9148-8d788092c3fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5xsnk"
Oct 30 21:29:09.817044 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:09.816970    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtzxj\" (UniqueName: \"kubernetes.io/projected/74888553-9594-4086-9148-8d788092c3fa-kube-api-access-gtzxj\") pod \"maas-api-revoke-sub-2e1f5b5f-5xsnk\" (UID: \"74888553-9594-4086-9148-8d788092c3fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5xsnk"
Oct 30 21:29:09.817044 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:09.817021    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74888553-9594-4086-9148-8d788092c3fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-5xsnk\" (UID: \"74888553-9594-4086-9148-8d788092c3fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5xsnk"
Oct 30 21:29:09.840831 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:09.840779    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:29:09.840831 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:09.840819    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:29:09.841045 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:09.840835    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:29:09.841045 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:09.840908    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:29:09.842979 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:09.842955    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5xsnk"]
Oct 30 21:29:09.918423 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:09.918388    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74888553-9594-4086-9148-8d788092c3fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-5xsnk\" (UID: \"74888553-9594-4086-9148-8d788092c3fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5xsnk"
Oct 30 21:29:09.918586 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:09.918456    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtzxj\" (UniqueName: \"kubernetes.io/projected/74888553-9594-4086-9148-8d788092c3fa-kube-api-access-gtzxj\") pod \"maas-api-revoke-sub-2e1f5b5f-5xsnk\" (UID: \"74888553-9594-4086-9148-8d788092c3fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5xsnk"
Oct 30 21:29:09.918586 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:09.918482    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74888553-9594-4086-9148-8d788092c3fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-5xsnk\" (UID: \"74888553-9594-4086-9148-8d788092c3fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5xsnk"
Oct 30 21:29:09.919098 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:09.919079    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74888553-9594-4086-9148-8d788092c3fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-5xsnk\" (UID: \"74888553-9594-4086-9148-8d788092c3fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5xsnk"
Oct 30 21:29:09.920918 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:09.920902    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74888553-9594-4086-9148-8d788092c3fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-5xsnk\" (UID: \"74888553-9594-4086-9148-8d788092c3fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5xsnk"
Oct 30 21:29:09.975259 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:09.975237    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtzxj\" (UniqueName: \"kubernetes.io/projected/74888553-9594-4086-9148-8d788092c3fa-kube-api-access-gtzxj\") pod \"maas-api-revoke-sub-2e1f5b5f-5xsnk\" (UID: \"74888553-9594-4086-9148-8d788092c3fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5xsnk"
Oct 30 21:29:10.050241 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:10.050166    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5xsnk"
Oct 30 21:29:10.426956 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:10.426932    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5xsnk"]
Oct 30 21:29:10.428960 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:29:10.428933    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74888553_9594_4086_9148_8d788092c3fa.slice/crio-259bf9cd3b2d50a7be59971aa4f53364a38e59a7801b177dce0632cd8e92a0a2 WatchSource:0}: Error finding container 259bf9cd3b2d50a7be59971aa4f53364a38e59a7801b177dce0632cd8e92a0a2: Status 404 returned error can't find the container with id 259bf9cd3b2d50a7be59971aa4f53364a38e59a7801b177dce0632cd8e92a0a2
Oct 30 21:29:10.954863 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:10.954824    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5xsnk" event={"ID":"74888553-9594-4086-9148-8d788092c3fa","Type":"ContainerStarted","Data":"8b12f5269b8ee2a9d14eae71cdc8bdf77f492054181adb2da9d551929bd7df74"}
Oct 30 21:29:10.955298 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:10.954884    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5xsnk" event={"ID":"74888553-9594-4086-9148-8d788092c3fa","Type":"ContainerStarted","Data":"259bf9cd3b2d50a7be59971aa4f53364a38e59a7801b177dce0632cd8e92a0a2"}
Oct 30 21:29:11.118966 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:11.118861    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5xsnk" podStartSLOduration=2.118847027 podStartE2EDuration="2.118847027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:29:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:29:11.068236631 +0000 UTC m=+3034.133013028" watchObservedRunningTime="2026-10-30 21:29:11.118847027 +0000 UTC m=+3034.183623424"
Oct 30 21:29:11.963552 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:11.963513    5713 generic.go:356] "Generic (PLEG): container finished" podID="74888553-9594-4086-9148-8d788092c3fa" containerID="8b12f5269b8ee2a9d14eae71cdc8bdf77f492054181adb2da9d551929bd7df74" exitCode=0
Oct 30 21:29:11.963996 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:11.963590    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5xsnk" event={"ID":"74888553-9594-4086-9148-8d788092c3fa","Type":"ContainerDied","Data":"8b12f5269b8ee2a9d14eae71cdc8bdf77f492054181adb2da9d551929bd7df74"}
Oct 30 21:29:13.111522 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:13.111495    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5xsnk"
Oct 30 21:29:13.253142 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:13.253048    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74888553-9594-4086-9148-8d788092c3fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74888553-9594-4086-9148-8d788092c3fa-maas-api-xks-ca\") pod \"74888553-9594-4086-9148-8d788092c3fa\" (UID: \"74888553-9594-4086-9148-8d788092c3fa\") "
Oct 30 21:29:13.253396 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:13.253176    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74888553-9594-4086-9148-8d788092c3fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74888553-9594-4086-9148-8d788092c3fa-maas-api-openshift-ca\") pod \"74888553-9594-4086-9148-8d788092c3fa\" (UID: \"74888553-9594-4086-9148-8d788092c3fa\") "
Oct 30 21:29:13.253396 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:13.253205    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74888553-9594-4086-9148-8d788092c3fa-kube-api-access-gtzxj\" (UniqueName: \"kubernetes.io/projected/74888553-9594-4086-9148-8d788092c3fa-kube-api-access-gtzxj\") pod \"74888553-9594-4086-9148-8d788092c3fa\" (UID: \"74888553-9594-4086-9148-8d788092c3fa\") "
Oct 30 21:29:13.253590 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:13.253564    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74888553-9594-4086-9148-8d788092c3fa-maas-api-openshift-ca" pod "74888553-9594-4086-9148-8d788092c3fa" (UID: "74888553-9594-4086-9148-8d788092c3fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:29:13.253696 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:13.253677    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74888553-9594-4086-9148-8d788092c3fa-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:29:13.255559 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:13.255530    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74888553-9594-4086-9148-8d788092c3fa-kube-api-access-gtzxj" pod "74888553-9594-4086-9148-8d788092c3fa" (UID: "74888553-9594-4086-9148-8d788092c3fa"). InnerVolumeSpecName "kube-api-access-gtzxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:29:13.255690 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:13.255529    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74888553-9594-4086-9148-8d788092c3fa-maas-api-xks-ca" pod "74888553-9594-4086-9148-8d788092c3fa" (UID: "74888553-9594-4086-9148-8d788092c3fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:29:13.356243 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:13.356200    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74888553-9594-4086-9148-8d788092c3fa-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:29:13.356516 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:13.356496    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtzxj\" (UniqueName: \"kubernetes.io/projected/74888553-9594-4086-9148-8d788092c3fa-kube-api-access-gtzxj\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:29:13.978270 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:13.978236    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5xsnk"
Oct 30 21:29:13.978450 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:13.978265    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5xsnk" event={"ID":"74888553-9594-4086-9148-8d788092c3fa","Type":"ContainerDied","Data":"259bf9cd3b2d50a7be59971aa4f53364a38e59a7801b177dce0632cd8e92a0a2"}
Oct 30 21:29:13.978450 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:13.978347    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="259bf9cd3b2d50a7be59971aa4f53364a38e59a7801b177dce0632cd8e92a0a2"
Oct 30 21:29:22.220969 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:22.220932    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2zjt5"]
Oct 30 21:29:22.221502 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:22.221484    5713 cpu_manager.go:414] "containerMap: removing container" podUID="74888553-9594-4086-9148-8d788092c3fa" containerName="revoke-keys"
Oct 30 21:29:22.221555 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:22.221504    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74888553-9594-4086-9148-8d788092c3fa" containerName="revoke-keys"
Oct 30 21:29:22.221600 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:22.221588    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="74888553-9594-4086-9148-8d788092c3fa" containerName="revoke-keys"
Oct 30 21:29:22.224839 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:22.224823    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2zjt5"
Oct 30 21:29:22.250400 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:22.250366    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7067c7a6-7e16-4dea-9379-227c0ec27956-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-2zjt5\" (UID: \"7067c7a6-7e16-4dea-9379-227c0ec27956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2zjt5"
Oct 30 21:29:22.250547 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:22.250495    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7067c7a6-7e16-4dea-9379-227c0ec27956-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-2zjt5\" (UID: \"7067c7a6-7e16-4dea-9379-227c0ec27956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2zjt5"
Oct 30 21:29:22.250610 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:22.250594    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4zjn\" (UniqueName: \"kubernetes.io/projected/7067c7a6-7e16-4dea-9379-227c0ec27956-kube-api-access-m4zjn\") pod \"maas-api-revoke-sub-ec843018-2zjt5\" (UID: \"7067c7a6-7e16-4dea-9379-227c0ec27956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2zjt5"
Oct 30 21:29:22.278602 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:22.278568    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:29:22.328428 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:22.328389    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:29:22.328593 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:22.328426    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:29:22.328593 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:22.328560    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:29:22.330745 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:22.330722    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2zjt5"]
Oct 30 21:29:22.351554 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:22.351529    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7067c7a6-7e16-4dea-9379-227c0ec27956-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-2zjt5\" (UID: \"7067c7a6-7e16-4dea-9379-227c0ec27956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2zjt5"
Oct 30 21:29:22.351693 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:22.351584    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7067c7a6-7e16-4dea-9379-227c0ec27956-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-2zjt5\" (UID: \"7067c7a6-7e16-4dea-9379-227c0ec27956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2zjt5"
Oct 30 21:29:22.351693 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:22.351620    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4zjn\" (UniqueName: \"kubernetes.io/projected/7067c7a6-7e16-4dea-9379-227c0ec27956-kube-api-access-m4zjn\") pod \"maas-api-revoke-sub-ec843018-2zjt5\" (UID: \"7067c7a6-7e16-4dea-9379-227c0ec27956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2zjt5"
Oct 30 21:29:22.352188 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:22.352159    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7067c7a6-7e16-4dea-9379-227c0ec27956-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-2zjt5\" (UID: \"7067c7a6-7e16-4dea-9379-227c0ec27956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2zjt5"
Oct 30 21:29:22.354090 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:22.354070    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7067c7a6-7e16-4dea-9379-227c0ec27956-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-2zjt5\" (UID: \"7067c7a6-7e16-4dea-9379-227c0ec27956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2zjt5"
Oct 30 21:29:22.408551 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:22.408518    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4zjn\" (UniqueName: \"kubernetes.io/projected/7067c7a6-7e16-4dea-9379-227c0ec27956-kube-api-access-m4zjn\") pod \"maas-api-revoke-sub-ec843018-2zjt5\" (UID: \"7067c7a6-7e16-4dea-9379-227c0ec27956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2zjt5"
Oct 30 21:29:22.537929 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:22.537854    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2zjt5"
Oct 30 21:29:22.911655 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:22.911626    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2zjt5"]
Oct 30 21:29:22.913524 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:29:22.913494    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7067c7a6_7e16_4dea_9379_227c0ec27956.slice/crio-ae137268d0f9c4bab1d80a3262d690600794f3085dbd58c10567897bc378b87b WatchSource:0}: Error finding container ae137268d0f9c4bab1d80a3262d690600794f3085dbd58c10567897bc378b87b: Status 404 returned error can't find the container with id ae137268d0f9c4bab1d80a3262d690600794f3085dbd58c10567897bc378b87b
Oct 30 21:29:23.034050 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:23.034018    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2zjt5" event={"ID":"7067c7a6-7e16-4dea-9379-227c0ec27956","Type":"ContainerStarted","Data":"ae137268d0f9c4bab1d80a3262d690600794f3085dbd58c10567897bc378b87b"}
Oct 30 21:29:24.049160 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:24.049060    5713 generic.go:356] "Generic (PLEG): container finished" podID="7067c7a6-7e16-4dea-9379-227c0ec27956" containerID="44ef0706b52c385bd6a3376aa98fd12ea67c08f5190cefbf7187552f29f4f9a4" exitCode=0
Oct 30 21:29:24.049565 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:24.049145    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2zjt5" event={"ID":"7067c7a6-7e16-4dea-9379-227c0ec27956","Type":"ContainerDied","Data":"44ef0706b52c385bd6a3376aa98fd12ea67c08f5190cefbf7187552f29f4f9a4"}
Oct 30 21:29:24.458382 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:29:24.458342    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:29:25.200503 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:25.200477    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2zjt5"
Oct 30 21:29:25.277920 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:25.277881    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7067c7a6-7e16-4dea-9379-227c0ec27956-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7067c7a6-7e16-4dea-9379-227c0ec27956-maas-api-openshift-ca\") pod \"7067c7a6-7e16-4dea-9379-227c0ec27956\" (UID: \"7067c7a6-7e16-4dea-9379-227c0ec27956\") "
Oct 30 21:29:25.278118 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:25.277942    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7067c7a6-7e16-4dea-9379-227c0ec27956-kube-api-access-m4zjn\" (UniqueName: \"kubernetes.io/projected/7067c7a6-7e16-4dea-9379-227c0ec27956-kube-api-access-m4zjn\") pod \"7067c7a6-7e16-4dea-9379-227c0ec27956\" (UID: \"7067c7a6-7e16-4dea-9379-227c0ec27956\") "
Oct 30 21:29:25.278118 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:25.278028    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7067c7a6-7e16-4dea-9379-227c0ec27956-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7067c7a6-7e16-4dea-9379-227c0ec27956-maas-api-xks-ca\") pod \"7067c7a6-7e16-4dea-9379-227c0ec27956\" (UID: \"7067c7a6-7e16-4dea-9379-227c0ec27956\") "
Oct 30 21:29:25.278306 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:25.278282    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7067c7a6-7e16-4dea-9379-227c0ec27956-maas-api-openshift-ca" pod "7067c7a6-7e16-4dea-9379-227c0ec27956" (UID: "7067c7a6-7e16-4dea-9379-227c0ec27956"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:29:25.278393 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:25.278379    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7067c7a6-7e16-4dea-9379-227c0ec27956-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:29:25.280370 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:25.280342    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7067c7a6-7e16-4dea-9379-227c0ec27956-maas-api-xks-ca" pod "7067c7a6-7e16-4dea-9379-227c0ec27956" (UID: "7067c7a6-7e16-4dea-9379-227c0ec27956"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:29:25.280370 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:25.280346    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7067c7a6-7e16-4dea-9379-227c0ec27956-kube-api-access-m4zjn" pod "7067c7a6-7e16-4dea-9379-227c0ec27956" (UID: "7067c7a6-7e16-4dea-9379-227c0ec27956"). InnerVolumeSpecName "kube-api-access-m4zjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:29:25.379777 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:25.379689    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7067c7a6-7e16-4dea-9379-227c0ec27956-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:29:25.379777 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:25.379722    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4zjn\" (UniqueName: \"kubernetes.io/projected/7067c7a6-7e16-4dea-9379-227c0ec27956-kube-api-access-m4zjn\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:29:26.063244 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:26.063212    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2zjt5"
Oct 30 21:29:26.063244 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:26.063237    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2zjt5" event={"ID":"7067c7a6-7e16-4dea-9379-227c0ec27956","Type":"ContainerDied","Data":"ae137268d0f9c4bab1d80a3262d690600794f3085dbd58c10567897bc378b87b"}
Oct 30 21:29:26.063468 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:26.063284    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae137268d0f9c4bab1d80a3262d690600794f3085dbd58c10567897bc378b87b"
Oct 30 21:29:39.976681 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:39.976639    5713 scope.go:122] "RemoveContainer" containerID="b2d822dd8135f76a5d3a34e1e43818b80a0a83e4c3fe5afb78dc9af238c877ed"
Oct 30 21:29:39.988721 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:39.988697    5713 scope.go:122] "RemoveContainer" containerID="acdbaf0edc8e390fdccf693dc68f058d775574a71919631bee029e0bcd341603"
Oct 30 21:29:40.000229 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:40.000206    5713 scope.go:122] "RemoveContainer" containerID="681b52304e085676252963dd556f4a0bc5918e190d80add968e58d0c87236de9"
Oct 30 21:29:40.009181 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:40.009158    5713 scope.go:122] "RemoveContainer" containerID="b7e8085ea675ba9ae1c1c8a2853a092e79dbc7c1fb7161a5e387e12d092eacfe"
Oct 30 21:29:54.458116 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:29:54.458018    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:29:57.187397 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:57.187356    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7vjdw"]
Oct 30 21:29:57.187916 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:57.187898    5713 cpu_manager.go:414] "containerMap: removing container" podUID="7067c7a6-7e16-4dea-9379-227c0ec27956" containerName="revoke-keys"
Oct 30 21:29:57.187982 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:57.187917    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7067c7a6-7e16-4dea-9379-227c0ec27956" containerName="revoke-keys"
Oct 30 21:29:57.188030 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:57.188024    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="7067c7a6-7e16-4dea-9379-227c0ec27956" containerName="revoke-keys"
Oct 30 21:29:57.192778 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:57.192759    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7vjdw"
Oct 30 21:29:57.248629 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:57.248591    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7vjdw"]
Oct 30 21:29:57.252151 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:57.252125    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98ee900c-6775-47d5-8b32-cd71c283e14b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-7vjdw\" (UID: \"98ee900c-6775-47d5-8b32-cd71c283e14b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7vjdw"
Oct 30 21:29:57.252345 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:57.252211    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98ee900c-6775-47d5-8b32-cd71c283e14b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-7vjdw\" (UID: \"98ee900c-6775-47d5-8b32-cd71c283e14b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7vjdw"
Oct 30 21:29:57.252345 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:57.252316    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7wht\" (UniqueName: \"kubernetes.io/projected/98ee900c-6775-47d5-8b32-cd71c283e14b-kube-api-access-x7wht\") pod \"maas-api-revoke-sub-046e1343-7vjdw\" (UID: \"98ee900c-6775-47d5-8b32-cd71c283e14b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7vjdw"
Oct 30 21:29:57.295713 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:57.295668    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:29:57.295713 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:57.295686    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:29:57.295952 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:57.295755    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:29:57.295952 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:57.295845    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:29:57.353176 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:57.353132    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98ee900c-6775-47d5-8b32-cd71c283e14b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-7vjdw\" (UID: \"98ee900c-6775-47d5-8b32-cd71c283e14b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7vjdw"
Oct 30 21:29:57.353395 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:57.353210    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7wht\" (UniqueName: \"kubernetes.io/projected/98ee900c-6775-47d5-8b32-cd71c283e14b-kube-api-access-x7wht\") pod \"maas-api-revoke-sub-046e1343-7vjdw\" (UID: \"98ee900c-6775-47d5-8b32-cd71c283e14b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7vjdw"
Oct 30 21:29:57.353395 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:57.353284    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98ee900c-6775-47d5-8b32-cd71c283e14b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-7vjdw\" (UID: \"98ee900c-6775-47d5-8b32-cd71c283e14b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7vjdw"
Oct 30 21:29:57.353918 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:57.353897    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98ee900c-6775-47d5-8b32-cd71c283e14b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-7vjdw\" (UID: \"98ee900c-6775-47d5-8b32-cd71c283e14b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7vjdw"
Oct 30 21:29:57.355887 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:57.355869    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98ee900c-6775-47d5-8b32-cd71c283e14b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-7vjdw\" (UID: \"98ee900c-6775-47d5-8b32-cd71c283e14b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7vjdw"
Oct 30 21:29:57.410901 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:57.410866    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7wht\" (UniqueName: \"kubernetes.io/projected/98ee900c-6775-47d5-8b32-cd71c283e14b-kube-api-access-x7wht\") pod \"maas-api-revoke-sub-046e1343-7vjdw\" (UID: \"98ee900c-6775-47d5-8b32-cd71c283e14b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7vjdw"
Oct 30 21:29:57.507332 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:57.507242    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7vjdw"
Oct 30 21:29:57.884577 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:57.884542    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7vjdw"]
Oct 30 21:29:57.886703 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:29:57.886672    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98ee900c_6775_47d5_8b32_cd71c283e14b.slice/crio-e719084e762547c21b6f2acda2789291bfedd67e826beca8e563a3609824b47d WatchSource:0}: Error finding container e719084e762547c21b6f2acda2789291bfedd67e826beca8e563a3609824b47d: Status 404 returned error can't find the container with id e719084e762547c21b6f2acda2789291bfedd67e826beca8e563a3609824b47d
Oct 30 21:29:58.291116 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:58.291075    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7vjdw" event={"ID":"98ee900c-6775-47d5-8b32-cd71c283e14b","Type":"ContainerStarted","Data":"c569612ff49af3e90d9fecb6c92a1a072a575683dd91aca88afc63a287497606"}
Oct 30 21:29:58.291532 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:58.291141    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7vjdw" event={"ID":"98ee900c-6775-47d5-8b32-cd71c283e14b","Type":"ContainerStarted","Data":"e719084e762547c21b6f2acda2789291bfedd67e826beca8e563a3609824b47d"}
Oct 30 21:29:58.405608 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:58.405501    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7vjdw" podStartSLOduration=1.405486161 podStartE2EDuration="1.405486161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:29:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:29:58.40364267 +0000 UTC m=+3081.468419070" watchObservedRunningTime="2026-10-30 21:29:58.405486161 +0000 UTC m=+3081.470262559"
Oct 30 21:29:59.299226 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:59.299194    5713 generic.go:356] "Generic (PLEG): container finished" podID="98ee900c-6775-47d5-8b32-cd71c283e14b" containerID="c569612ff49af3e90d9fecb6c92a1a072a575683dd91aca88afc63a287497606" exitCode=0
Oct 30 21:29:59.299664 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:29:59.299287    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7vjdw" event={"ID":"98ee900c-6775-47d5-8b32-cd71c283e14b","Type":"ContainerDied","Data":"c569612ff49af3e90d9fecb6c92a1a072a575683dd91aca88afc63a287497606"}
Oct 30 21:30:00.439611 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:00.439587    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7vjdw"
Oct 30 21:30:00.485015 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:00.484988    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98ee900c-6775-47d5-8b32-cd71c283e14b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98ee900c-6775-47d5-8b32-cd71c283e14b-maas-api-openshift-ca\") pod \"98ee900c-6775-47d5-8b32-cd71c283e14b\" (UID: \"98ee900c-6775-47d5-8b32-cd71c283e14b\") "
Oct 30 21:30:00.485155 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:00.485031    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98ee900c-6775-47d5-8b32-cd71c283e14b-kube-api-access-x7wht\" (UniqueName: \"kubernetes.io/projected/98ee900c-6775-47d5-8b32-cd71c283e14b-kube-api-access-x7wht\") pod \"98ee900c-6775-47d5-8b32-cd71c283e14b\" (UID: \"98ee900c-6775-47d5-8b32-cd71c283e14b\") "
Oct 30 21:30:00.485155 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:00.485116    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98ee900c-6775-47d5-8b32-cd71c283e14b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98ee900c-6775-47d5-8b32-cd71c283e14b-maas-api-xks-ca\") pod \"98ee900c-6775-47d5-8b32-cd71c283e14b\" (UID: \"98ee900c-6775-47d5-8b32-cd71c283e14b\") "
Oct 30 21:30:00.485392 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:00.485367    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98ee900c-6775-47d5-8b32-cd71c283e14b-maas-api-openshift-ca" pod "98ee900c-6775-47d5-8b32-cd71c283e14b" (UID: "98ee900c-6775-47d5-8b32-cd71c283e14b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:00.485484 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:00.485470    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98ee900c-6775-47d5-8b32-cd71c283e14b-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:30:00.487271 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:00.487245    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98ee900c-6775-47d5-8b32-cd71c283e14b-maas-api-xks-ca" pod "98ee900c-6775-47d5-8b32-cd71c283e14b" (UID: "98ee900c-6775-47d5-8b32-cd71c283e14b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:30:00.487271 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:00.487256    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98ee900c-6775-47d5-8b32-cd71c283e14b-kube-api-access-x7wht" pod "98ee900c-6775-47d5-8b32-cd71c283e14b" (UID: "98ee900c-6775-47d5-8b32-cd71c283e14b"). InnerVolumeSpecName "kube-api-access-x7wht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:00.586527 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:00.586452    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7wht\" (UniqueName: \"kubernetes.io/projected/98ee900c-6775-47d5-8b32-cd71c283e14b-kube-api-access-x7wht\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:30:00.586527 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:00.586480    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98ee900c-6775-47d5-8b32-cd71c283e14b-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:30:01.313188 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:01.313151    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7vjdw" event={"ID":"98ee900c-6775-47d5-8b32-cd71c283e14b","Type":"ContainerDied","Data":"e719084e762547c21b6f2acda2789291bfedd67e826beca8e563a3609824b47d"}
Oct 30 21:30:01.313188 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:01.313174    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7vjdw"
Oct 30 21:30:01.313402 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:01.313201    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e719084e762547c21b6f2acda2789291bfedd67e826beca8e563a3609824b47d"
Oct 30 21:30:07.964568 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:07.964529    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mgz9h"]
Oct 30 21:30:07.965032 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:07.965012    5713 cpu_manager.go:414] "containerMap: removing container" podUID="98ee900c-6775-47d5-8b32-cd71c283e14b" containerName="revoke-keys"
Oct 30 21:30:07.965032 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:07.965026    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98ee900c-6775-47d5-8b32-cd71c283e14b" containerName="revoke-keys"
Oct 30 21:30:07.965111 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:07.965093    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="98ee900c-6775-47d5-8b32-cd71c283e14b" containerName="revoke-keys"
Oct 30 21:30:07.968373 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:07.968351    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mgz9h"
Oct 30 21:30:08.024776 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:08.024739    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mgz9h"]
Oct 30 21:30:08.069888 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:08.069845    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q56zv\" (UniqueName: \"kubernetes.io/projected/1f3ed3e1-ad8e-4605-9007-85d4016c19cd-kube-api-access-q56zv\") pod \"maas-api-revoke-sub-d3fba056-mgz9h\" (UID: \"1f3ed3e1-ad8e-4605-9007-85d4016c19cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mgz9h"
Oct 30 21:30:08.070111 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:08.070009    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f3ed3e1-ad8e-4605-9007-85d4016c19cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-mgz9h\" (UID: \"1f3ed3e1-ad8e-4605-9007-85d4016c19cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mgz9h"
Oct 30 21:30:08.070111 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:08.070061    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f3ed3e1-ad8e-4605-9007-85d4016c19cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-mgz9h\" (UID: \"1f3ed3e1-ad8e-4605-9007-85d4016c19cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mgz9h"
Oct 30 21:30:08.070781 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:08.070764    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:30:08.070781 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:08.070773    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:30:08.070959 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:08.070828    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:30:08.070959 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:08.070918    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:30:08.170735 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:08.170693    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f3ed3e1-ad8e-4605-9007-85d4016c19cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-mgz9h\" (UID: \"1f3ed3e1-ad8e-4605-9007-85d4016c19cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mgz9h"
Oct 30 21:30:08.170991 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:08.170746    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f3ed3e1-ad8e-4605-9007-85d4016c19cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-mgz9h\" (UID: \"1f3ed3e1-ad8e-4605-9007-85d4016c19cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mgz9h"
Oct 30 21:30:08.170991 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:08.170928    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q56zv\" (UniqueName: \"kubernetes.io/projected/1f3ed3e1-ad8e-4605-9007-85d4016c19cd-kube-api-access-q56zv\") pod \"maas-api-revoke-sub-d3fba056-mgz9h\" (UID: \"1f3ed3e1-ad8e-4605-9007-85d4016c19cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mgz9h"
Oct 30 21:30:08.171576 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:08.171545    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f3ed3e1-ad8e-4605-9007-85d4016c19cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-mgz9h\" (UID: \"1f3ed3e1-ad8e-4605-9007-85d4016c19cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mgz9h"
Oct 30 21:30:08.173432 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:08.173411    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f3ed3e1-ad8e-4605-9007-85d4016c19cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-mgz9h\" (UID: \"1f3ed3e1-ad8e-4605-9007-85d4016c19cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mgz9h"
Oct 30 21:30:08.228661 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:08.228588    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q56zv\" (UniqueName: \"kubernetes.io/projected/1f3ed3e1-ad8e-4605-9007-85d4016c19cd-kube-api-access-q56zv\") pod \"maas-api-revoke-sub-d3fba056-mgz9h\" (UID: \"1f3ed3e1-ad8e-4605-9007-85d4016c19cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mgz9h"
Oct 30 21:30:08.281552 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:08.281508    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mgz9h"
Oct 30 21:30:08.656873 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:08.656836    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mgz9h"]
Oct 30 21:30:08.659165 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:30:08.659141    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f3ed3e1_ad8e_4605_9007_85d4016c19cd.slice/crio-1986376689cf0de36ee78436b916db79bfe9610af064d9dd2447cd04a13ec106 WatchSource:0}: Error finding container 1986376689cf0de36ee78436b916db79bfe9610af064d9dd2447cd04a13ec106: Status 404 returned error can't find the container with id 1986376689cf0de36ee78436b916db79bfe9610af064d9dd2447cd04a13ec106
Oct 30 21:30:09.362778 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:09.362686    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mgz9h" event={"ID":"1f3ed3e1-ad8e-4605-9007-85d4016c19cd","Type":"ContainerStarted","Data":"1d73186fbbd7c1c39025681de1b15c3609dabb952c53433dae21c8561a116a2e"}
Oct 30 21:30:09.362778 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:09.362745    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mgz9h" event={"ID":"1f3ed3e1-ad8e-4605-9007-85d4016c19cd","Type":"ContainerStarted","Data":"1986376689cf0de36ee78436b916db79bfe9610af064d9dd2447cd04a13ec106"}
Oct 30 21:30:09.479357 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:09.479307    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mgz9h" podStartSLOduration=2.4792910089999998 podStartE2EDuration="2.479291009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:09.478245883 +0000 UTC m=+3092.543022282" watchObservedRunningTime="2026-10-30 21:30:09.479291009 +0000 UTC m=+3092.544067403"
Oct 30 21:30:10.371436 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:10.371391    5713 generic.go:356] "Generic (PLEG): container finished" podID="1f3ed3e1-ad8e-4605-9007-85d4016c19cd" containerID="1d73186fbbd7c1c39025681de1b15c3609dabb952c53433dae21c8561a116a2e" exitCode=0
Oct 30 21:30:10.371914 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:10.371503    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mgz9h" event={"ID":"1f3ed3e1-ad8e-4605-9007-85d4016c19cd","Type":"ContainerDied","Data":"1d73186fbbd7c1c39025681de1b15c3609dabb952c53433dae21c8561a116a2e"}
Oct 30 21:30:11.809006 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:11.808968    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mgz9h"
Oct 30 21:30:11.919941 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:11.919901    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f3ed3e1-ad8e-4605-9007-85d4016c19cd-kube-api-access-q56zv\" (UniqueName: \"kubernetes.io/projected/1f3ed3e1-ad8e-4605-9007-85d4016c19cd-kube-api-access-q56zv\") pod \"1f3ed3e1-ad8e-4605-9007-85d4016c19cd\" (UID: \"1f3ed3e1-ad8e-4605-9007-85d4016c19cd\") "
Oct 30 21:30:11.920137 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:11.919951    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f3ed3e1-ad8e-4605-9007-85d4016c19cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f3ed3e1-ad8e-4605-9007-85d4016c19cd-maas-api-xks-ca\") pod \"1f3ed3e1-ad8e-4605-9007-85d4016c19cd\" (UID: \"1f3ed3e1-ad8e-4605-9007-85d4016c19cd\") "
Oct 30 21:30:11.920137 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:11.919979    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f3ed3e1-ad8e-4605-9007-85d4016c19cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f3ed3e1-ad8e-4605-9007-85d4016c19cd-maas-api-openshift-ca\") pod \"1f3ed3e1-ad8e-4605-9007-85d4016c19cd\" (UID: \"1f3ed3e1-ad8e-4605-9007-85d4016c19cd\") "
Oct 30 21:30:11.920387 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:11.920363    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f3ed3e1-ad8e-4605-9007-85d4016c19cd-maas-api-openshift-ca" pod "1f3ed3e1-ad8e-4605-9007-85d4016c19cd" (UID: "1f3ed3e1-ad8e-4605-9007-85d4016c19cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:11.922277 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:11.922251    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f3ed3e1-ad8e-4605-9007-85d4016c19cd-maas-api-xks-ca" pod "1f3ed3e1-ad8e-4605-9007-85d4016c19cd" (UID: "1f3ed3e1-ad8e-4605-9007-85d4016c19cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:30:11.922389 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:11.922318    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f3ed3e1-ad8e-4605-9007-85d4016c19cd-kube-api-access-q56zv" pod "1f3ed3e1-ad8e-4605-9007-85d4016c19cd" (UID: "1f3ed3e1-ad8e-4605-9007-85d4016c19cd"). InnerVolumeSpecName "kube-api-access-q56zv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:12.021893 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:12.021760    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q56zv\" (UniqueName: \"kubernetes.io/projected/1f3ed3e1-ad8e-4605-9007-85d4016c19cd-kube-api-access-q56zv\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:30:12.021893 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:12.021827    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f3ed3e1-ad8e-4605-9007-85d4016c19cd-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:30:12.021893 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:12.021844    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f3ed3e1-ad8e-4605-9007-85d4016c19cd-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:30:12.411431 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:12.411333    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mgz9h" event={"ID":"1f3ed3e1-ad8e-4605-9007-85d4016c19cd","Type":"ContainerDied","Data":"1986376689cf0de36ee78436b916db79bfe9610af064d9dd2447cd04a13ec106"}
Oct 30 21:30:12.411669 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:12.411650    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1986376689cf0de36ee78436b916db79bfe9610af064d9dd2447cd04a13ec106"
Oct 30 21:30:12.411879 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:12.411862    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mgz9h"
Oct 30 21:30:17.047971 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:17.047937    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1be5211-l9qt4"]
Oct 30 21:30:17.097515 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:17.097482    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1be5211-l9qt4"]
Oct 30 21:30:17.110845 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:30:17.110779    5713 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-sub-d1be5211-l9qt4\" is forbidden: User \"system:node:ip-10-0-128-165.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-128-165.us-west-1.compute.internal' and this object" podUID="4e5ad572-be75-4519-a23b-32004eeab2b7" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1be5211-l9qt4"
Oct 30 21:30:18.046060 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:18.046021    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-664cc788-rdbnn"]
Oct 30 21:30:18.095842 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:18.095809    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-664cc788-rdbnn"]
Oct 30 21:30:19.062083 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:19.062050    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="403b83a9-0bc8-4dc5-8053-139663aea455" path="/var/lib/kubelet/pods/403b83a9-0bc8-4dc5-8053-139663aea455/volumes"
Oct 30 21:30:19.062472 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:19.062458    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e5ad572-be75-4519-a23b-32004eeab2b7" path="/var/lib/kubelet/pods/4e5ad572-be75-4519-a23b-32004eeab2b7/volumes"
Oct 30 21:30:24.458367 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:30:24.458332    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:30:39.046871 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:39.046836    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47b7c7b6-7fmvd"]
Oct 30 21:30:39.095808 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:39.095758    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-430f8459-rc6bx"]
Oct 30 21:30:39.097678 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:39.097655    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47b7c7b6-7fmvd"]
Oct 30 21:30:39.102414 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:39.102391    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-430f8459-rc6bx"]
Oct 30 21:30:40.052878 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:40.052844    5713 scope.go:122] "RemoveContainer" containerID="60a574a7ffcf27b3657bb97d5bc31db90f72e7a4eeb445944e703423d6f600b2"
Oct 30 21:30:40.062424 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:40.062394    5713 scope.go:122] "RemoveContainer" containerID="567a08a3a0e5764238904d74e0842d7b2ee4dedea92b43847711e0675bc39e31"
Oct 30 21:30:40.071855 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:40.071832    5713 scope.go:122] "RemoveContainer" containerID="294f2f5bfe666597c321a5672f94a0f813f1bc4c485ac620b525267466f9302c"
Oct 30 21:30:41.061962 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:41.061923    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37663328-2eda-4b83-bc43-c60883c8694c" path="/var/lib/kubelet/pods/37663328-2eda-4b83-bc43-c60883c8694c/volumes"
Oct 30 21:30:41.062397 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:41.062381    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37c106d7-0120-4409-acc2-ef0db4a8b003" path="/var/lib/kubelet/pods/37c106d7-0120-4409-acc2-ef0db4a8b003/volumes"
Oct 30 21:30:42.878100 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:42.878067    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5625c"]
Oct 30 21:30:42.878590 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:42.878485    5713 cpu_manager.go:414] "containerMap: removing container" podUID="1f3ed3e1-ad8e-4605-9007-85d4016c19cd" containerName="revoke-keys"
Oct 30 21:30:42.878590 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:42.878498    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f3ed3e1-ad8e-4605-9007-85d4016c19cd" containerName="revoke-keys"
Oct 30 21:30:42.878590 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:42.878571    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f3ed3e1-ad8e-4605-9007-85d4016c19cd" containerName="revoke-keys"
Oct 30 21:30:42.883359 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:42.883339    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5625c"
Oct 30 21:30:42.939084 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:42.939051    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:30:42.954555 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:42.954522    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63ca2987-423e-43ed-90a4-f1cdb5b058c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-5625c\" (UID: \"63ca2987-423e-43ed-90a4-f1cdb5b058c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5625c"
Oct 30 21:30:42.954725 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:42.954658    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z64w6\" (UniqueName: \"kubernetes.io/projected/63ca2987-423e-43ed-90a4-f1cdb5b058c1-kube-api-access-z64w6\") pod \"maas-api-revoke-sub-ae594979-5625c\" (UID: \"63ca2987-423e-43ed-90a4-f1cdb5b058c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5625c"
Oct 30 21:30:42.954725 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:42.954714    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63ca2987-423e-43ed-90a4-f1cdb5b058c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-5625c\" (UID: \"63ca2987-423e-43ed-90a4-f1cdb5b058c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5625c"
Oct 30 21:30:42.987970 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:42.987927    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:30:42.988168 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:42.988138    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:30:42.988266 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:42.988168    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:30:42.990518 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:42.990491    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5625c"]
Oct 30 21:30:43.056342 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:43.056312    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z64w6\" (UniqueName: \"kubernetes.io/projected/63ca2987-423e-43ed-90a4-f1cdb5b058c1-kube-api-access-z64w6\") pod \"maas-api-revoke-sub-ae594979-5625c\" (UID: \"63ca2987-423e-43ed-90a4-f1cdb5b058c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5625c"
Oct 30 21:30:43.056508 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:43.056365    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63ca2987-423e-43ed-90a4-f1cdb5b058c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-5625c\" (UID: \"63ca2987-423e-43ed-90a4-f1cdb5b058c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5625c"
Oct 30 21:30:43.056569 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:43.056543    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63ca2987-423e-43ed-90a4-f1cdb5b058c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-5625c\" (UID: \"63ca2987-423e-43ed-90a4-f1cdb5b058c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5625c"
Oct 30 21:30:43.057143 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:43.057117    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63ca2987-423e-43ed-90a4-f1cdb5b058c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-5625c\" (UID: \"63ca2987-423e-43ed-90a4-f1cdb5b058c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5625c"
Oct 30 21:30:43.058862 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:43.058831    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63ca2987-423e-43ed-90a4-f1cdb5b058c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-5625c\" (UID: \"63ca2987-423e-43ed-90a4-f1cdb5b058c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5625c"
Oct 30 21:30:43.113727 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:43.113692    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z64w6\" (UniqueName: \"kubernetes.io/projected/63ca2987-423e-43ed-90a4-f1cdb5b058c1-kube-api-access-z64w6\") pod \"maas-api-revoke-sub-ae594979-5625c\" (UID: \"63ca2987-423e-43ed-90a4-f1cdb5b058c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5625c"
Oct 30 21:30:43.196027 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:43.196002    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5625c"
Oct 30 21:30:43.571330 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:43.571277    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5625c"]
Oct 30 21:30:43.574162 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:30:43.574130    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63ca2987_423e_43ed_90a4_f1cdb5b058c1.slice/crio-ceb898dfea8fb8ec9a0058e874bed236adf408fd3a443585a304d822d4d97b57 WatchSource:0}: Error finding container ceb898dfea8fb8ec9a0058e874bed236adf408fd3a443585a304d822d4d97b57: Status 404 returned error can't find the container with id ceb898dfea8fb8ec9a0058e874bed236adf408fd3a443585a304d822d4d97b57
Oct 30 21:30:43.612098 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:43.612059    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5625c" event={"ID":"63ca2987-423e-43ed-90a4-f1cdb5b058c1","Type":"ContainerStarted","Data":"ceb898dfea8fb8ec9a0058e874bed236adf408fd3a443585a304d822d4d97b57"}
Oct 30 21:30:44.619601 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:44.619567    5713 generic.go:356] "Generic (PLEG): container finished" podID="63ca2987-423e-43ed-90a4-f1cdb5b058c1" containerID="d99a0dd3e22b8bdde846557083fa3a43a5ea4cff68ed0a13b55038dbc386ebca" exitCode=0
Oct 30 21:30:44.620036 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:44.619656    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5625c" event={"ID":"63ca2987-423e-43ed-90a4-f1cdb5b058c1","Type":"ContainerDied","Data":"d99a0dd3e22b8bdde846557083fa3a43a5ea4cff68ed0a13b55038dbc386ebca"}
Oct 30 21:30:45.779126 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:45.779096    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5625c"
Oct 30 21:30:45.881933 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:45.881843    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63ca2987-423e-43ed-90a4-f1cdb5b058c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63ca2987-423e-43ed-90a4-f1cdb5b058c1-maas-api-openshift-ca\") pod \"63ca2987-423e-43ed-90a4-f1cdb5b058c1\" (UID: \"63ca2987-423e-43ed-90a4-f1cdb5b058c1\") "
Oct 30 21:30:45.882109 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:45.881953    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63ca2987-423e-43ed-90a4-f1cdb5b058c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63ca2987-423e-43ed-90a4-f1cdb5b058c1-maas-api-xks-ca\") pod \"63ca2987-423e-43ed-90a4-f1cdb5b058c1\" (UID: \"63ca2987-423e-43ed-90a4-f1cdb5b058c1\") "
Oct 30 21:30:45.882109 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:45.881981    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63ca2987-423e-43ed-90a4-f1cdb5b058c1-kube-api-access-z64w6\" (UniqueName: \"kubernetes.io/projected/63ca2987-423e-43ed-90a4-f1cdb5b058c1-kube-api-access-z64w6\") pod \"63ca2987-423e-43ed-90a4-f1cdb5b058c1\" (UID: \"63ca2987-423e-43ed-90a4-f1cdb5b058c1\") "
Oct 30 21:30:45.882277 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:45.882250    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63ca2987-423e-43ed-90a4-f1cdb5b058c1-maas-api-openshift-ca" pod "63ca2987-423e-43ed-90a4-f1cdb5b058c1" (UID: "63ca2987-423e-43ed-90a4-f1cdb5b058c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:45.884325 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:45.884298    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63ca2987-423e-43ed-90a4-f1cdb5b058c1-maas-api-xks-ca" pod "63ca2987-423e-43ed-90a4-f1cdb5b058c1" (UID: "63ca2987-423e-43ed-90a4-f1cdb5b058c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:30:45.884439 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:45.884395    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63ca2987-423e-43ed-90a4-f1cdb5b058c1-kube-api-access-z64w6" pod "63ca2987-423e-43ed-90a4-f1cdb5b058c1" (UID: "63ca2987-423e-43ed-90a4-f1cdb5b058c1"). InnerVolumeSpecName "kube-api-access-z64w6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:45.983197 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:45.983153    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63ca2987-423e-43ed-90a4-f1cdb5b058c1-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:30:45.983197 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:45.983188    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z64w6\" (UniqueName: \"kubernetes.io/projected/63ca2987-423e-43ed-90a4-f1cdb5b058c1-kube-api-access-z64w6\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:30:45.983197 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:45.983202    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63ca2987-423e-43ed-90a4-f1cdb5b058c1-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:30:46.633653 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:46.633618    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5625c"
Oct 30 21:30:46.633653 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:46.633643    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5625c" event={"ID":"63ca2987-423e-43ed-90a4-f1cdb5b058c1","Type":"ContainerDied","Data":"ceb898dfea8fb8ec9a0058e874bed236adf408fd3a443585a304d822d4d97b57"}
Oct 30 21:30:46.633930 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:30:46.633693    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ceb898dfea8fb8ec9a0058e874bed236adf408fd3a443585a304d822d4d97b57"
Oct 30 21:30:54.458299 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:30:54.458255    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:31:24.459632 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:31:24.459583    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:31:40.046045 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:31:40.046010    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cd3daaa-s5ctm"]
Oct 30 21:31:40.095866 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:31:40.095829    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cd3daaa-s5ctm"]
Oct 30 21:31:40.104133 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:31:40.104102    5713 scope.go:122] "RemoveContainer" containerID="4f8466573277a3a7fdb4fd53190d81fceda4627cfbe907c798bbae1b10e8f37d"
Oct 30 21:31:40.114249 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:31:40.114219    5713 scope.go:122] "RemoveContainer" containerID="6bb0dabed4de25bc5ca696d1fb589828ca4f6bf9a2f8558ff04c0247a34d94da"
Oct 30 21:31:40.123764 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:31:40.123743    5713 scope.go:122] "RemoveContainer" containerID="183ad8e0a51a68b2f3c329e7d6c44d8914dd91a92fa5cf4a4fb612ccaf0fdfcd"
Oct 30 21:31:40.134764 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:31:40.134738    5713 scope.go:122] "RemoveContainer" containerID="1c9006288c5cf8c256fbfb4d85deae72113b9d1eff97dd9f9b1981f8931bc763"
Oct 30 21:31:40.145084 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:31:40.145058    5713 scope.go:122] "RemoveContainer" containerID="2a4295411daee5452740a8de8bdf52430577ba59f56c54be015cac966f71c334"
Oct 30 21:31:40.160368 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:31:40.160328    5713 scope.go:122] "RemoveContainer" containerID="264ea12b3678a7471e360a66aa272a2307e5545291b99e20dd12eef5cb93dffd"
Oct 30 21:31:40.170375 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:31:40.170348    5713 scope.go:122] "RemoveContainer" containerID="75fded7c3ce819ecfabf758745be1881ce8d661cf8e8d5f94cfbaad34a0ba08f"
Oct 30 21:31:41.044974 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:31:41.044931    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78d8a57a-mnsmx"]
Oct 30 21:31:41.061864 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:31:41.061830    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18d880ed-6e69-43b3-8224-85a2271fe2c8" path="/var/lib/kubelet/pods/18d880ed-6e69-43b3-8224-85a2271fe2c8/volumes"
Oct 30 21:31:41.095266 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:31:41.095233    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78d8a57a-mnsmx"]
Oct 30 21:31:41.707019 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:31:41.706984    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vjc66"]
Oct 30 21:31:41.707399 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:31:41.707387    5713 cpu_manager.go:414] "containerMap: removing container" podUID="63ca2987-423e-43ed-90a4-f1cdb5b058c1" containerName="revoke-keys"
Oct 30 21:31:41.707450 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:31:41.707400    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63ca2987-423e-43ed-90a4-f1cdb5b058c1" containerName="revoke-keys"
Oct 30 21:31:41.707497 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:31:41.707488    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="63ca2987-423e-43ed-90a4-f1cdb5b058c1" containerName="revoke-keys"
Oct 30 21:31:41.712462 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:31:41.712445    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vjc66"
Oct 30 21:31:41.767949 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:31:41.767915    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vjc66"]
Oct 30 21:31:41.809222 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:31:41.809183    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p559h\" (UniqueName: \"kubernetes.io/projected/ad47d82f-6b21-4a15-9f2a-3ebb3d4134dc-kube-api-access-p559h\") pod \"maas-api-revoke-sub-79805ca9-vjc66\" (UID: \"ad47d82f-6b21-4a15-9f2a-3ebb3d4134dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vjc66"
Oct 30 21:31:41.809424 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:31:41.809236    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad47d82f-6b21-4a15-9f2a-3ebb3d4134dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-vjc66\" (UID: \"ad47d82f-6b21-4a15-9f2a-3ebb3d4134dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vjc66"
Oct 30 21:31:41.809424 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:31:41.809337    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad47d82f-6b21-4a15-9f2a-3ebb3d4134dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-vjc66\" (UID: \"ad47d82f-6b21-4a15-9f2a-3ebb3d4134dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vjc66"
Oct 30 21:31:41.814905 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:31:41.814872    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:31:41.814905 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:31:41.814878    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:31:41.815078 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:31:41.814908    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:31:41.815078 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:31:41.814883    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:31:41.910446 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:31:41.910414    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p559h\" (UniqueName: \"kubernetes.io/projected/ad47d82f-6b21-4a15-9f2a-3ebb3d4134dc-kube-api-access-p559h\") pod \"maas-api-revoke-sub-79805ca9-vjc66\" (UID: \"ad47d82f-6b21-4a15-9f2a-3ebb3d4134dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vjc66"
Oct 30 21:31:41.910636 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:31:41.910459    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad47d82f-6b21-4a15-9f2a-3ebb3d4134dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-vjc66\" (UID: \"ad47d82f-6b21-4a15-9f2a-3ebb3d4134dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vjc66"
Oct 30 21:31:41.910636 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:31:41.910510    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad47d82f-6b21-4a15-9f2a-3ebb3d4134dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-vjc66\" (UID: \"ad47d82f-6b21-4a15-9f2a-3ebb3d4134dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vjc66"
Oct 30 21:31:41.911307 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:31:41.911281    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad47d82f-6b21-4a15-9f2a-3ebb3d4134dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-vjc66\" (UID: \"ad47d82f-6b21-4a15-9f2a-3ebb3d4134dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vjc66"
Oct 30 21:31:41.913216 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:31:41.913190    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad47d82f-6b21-4a15-9f2a-3ebb3d4134dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-vjc66\" (UID: \"ad47d82f-6b21-4a15-9f2a-3ebb3d4134dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vjc66"
Oct 30 21:31:41.967578 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:31:41.967510    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p559h\" (UniqueName: \"kubernetes.io/projected/ad47d82f-6b21-4a15-9f2a-3ebb3d4134dc-kube-api-access-p559h\") pod \"maas-api-revoke-sub-79805ca9-vjc66\" (UID: \"ad47d82f-6b21-4a15-9f2a-3ebb3d4134dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vjc66"
Oct 30 21:31:42.025178 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:31:42.025144    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vjc66"
Oct 30 21:31:42.397933 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:31:42.397908    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vjc66"]
Oct 30 21:31:42.400235 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:31:42.400210    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad47d82f_6b21_4a15_9f2a_3ebb3d4134dc.slice/crio-c8da0207fa85e126fd17f8e7b786892f9a1668e4a590ad8d4040494b8a88d9a6 WatchSource:0}: Error finding container c8da0207fa85e126fd17f8e7b786892f9a1668e4a590ad8d4040494b8a88d9a6: Status 404 returned error can't find the container with id c8da0207fa85e126fd17f8e7b786892f9a1668e4a590ad8d4040494b8a88d9a6
Oct 30 21:31:42.973869 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:31:42.973838    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vjc66" event={"ID":"ad47d82f-6b21-4a15-9f2a-3ebb3d4134dc","Type":"ContainerStarted","Data":"bab809122973d6ad4da20dd816e16c5bc4e03f9d67549c825a91905e83ef1256"}
Oct 30 21:31:42.974072 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:31:42.973894    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vjc66" event={"ID":"ad47d82f-6b21-4a15-9f2a-3ebb3d4134dc","Type":"ContainerStarted","Data":"c8da0207fa85e126fd17f8e7b786892f9a1668e4a590ad8d4040494b8a88d9a6"}
Oct 30 21:31:43.062120 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:31:43.062091    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99143411-6bbc-483e-9c3d-002cfce1efdb" path="/var/lib/kubelet/pods/99143411-6bbc-483e-9c3d-002cfce1efdb/volumes"
Oct 30 21:31:43.089039 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:31:43.088983    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vjc66" podStartSLOduration=2.088969774 podStartE2EDuration="2.088969774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:31:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:31:43.087208111 +0000 UTC m=+3186.151984510" watchObservedRunningTime="2026-10-30 21:31:43.088969774 +0000 UTC m=+3186.153746173"
Oct 30 21:31:43.981112 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:31:43.981074    5713 generic.go:356] "Generic (PLEG): container finished" podID="ad47d82f-6b21-4a15-9f2a-3ebb3d4134dc" containerID="bab809122973d6ad4da20dd816e16c5bc4e03f9d67549c825a91905e83ef1256" exitCode=0
Oct 30 21:31:43.981517 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:31:43.981155    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vjc66" event={"ID":"ad47d82f-6b21-4a15-9f2a-3ebb3d4134dc","Type":"ContainerDied","Data":"bab809122973d6ad4da20dd816e16c5bc4e03f9d67549c825a91905e83ef1256"}
Oct 30 21:31:45.126520 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:31:45.126476    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vjc66"
Oct 30 21:31:45.245915 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:31:45.245845    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad47d82f-6b21-4a15-9f2a-3ebb3d4134dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad47d82f-6b21-4a15-9f2a-3ebb3d4134dc-maas-api-openshift-ca\") pod \"ad47d82f-6b21-4a15-9f2a-3ebb3d4134dc\" (UID: \"ad47d82f-6b21-4a15-9f2a-3ebb3d4134dc\") "
Oct 30 21:31:45.246055 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:31:45.245920    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad47d82f-6b21-4a15-9f2a-3ebb3d4134dc-kube-api-access-p559h\" (UniqueName: \"kubernetes.io/projected/ad47d82f-6b21-4a15-9f2a-3ebb3d4134dc-kube-api-access-p559h\") pod \"ad47d82f-6b21-4a15-9f2a-3ebb3d4134dc\" (UID: \"ad47d82f-6b21-4a15-9f2a-3ebb3d4134dc\") "
Oct 30 21:31:45.246133 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:31:45.246081    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad47d82f-6b21-4a15-9f2a-3ebb3d4134dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad47d82f-6b21-4a15-9f2a-3ebb3d4134dc-maas-api-xks-ca\") pod \"ad47d82f-6b21-4a15-9f2a-3ebb3d4134dc\" (UID: \"ad47d82f-6b21-4a15-9f2a-3ebb3d4134dc\") "
Oct 30 21:31:45.246235 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:31:45.246211    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad47d82f-6b21-4a15-9f2a-3ebb3d4134dc-maas-api-openshift-ca" pod "ad47d82f-6b21-4a15-9f2a-3ebb3d4134dc" (UID: "ad47d82f-6b21-4a15-9f2a-3ebb3d4134dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:31:45.246442 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:31:45.246425    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad47d82f-6b21-4a15-9f2a-3ebb3d4134dc-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:31:45.248192 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:31:45.248167    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad47d82f-6b21-4a15-9f2a-3ebb3d4134dc-kube-api-access-p559h" pod "ad47d82f-6b21-4a15-9f2a-3ebb3d4134dc" (UID: "ad47d82f-6b21-4a15-9f2a-3ebb3d4134dc"). InnerVolumeSpecName "kube-api-access-p559h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:31:45.248578 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:31:45.248551    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad47d82f-6b21-4a15-9f2a-3ebb3d4134dc-maas-api-xks-ca" pod "ad47d82f-6b21-4a15-9f2a-3ebb3d4134dc" (UID: "ad47d82f-6b21-4a15-9f2a-3ebb3d4134dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:31:45.347867 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:31:45.347830    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p559h\" (UniqueName: \"kubernetes.io/projected/ad47d82f-6b21-4a15-9f2a-3ebb3d4134dc-kube-api-access-p559h\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:31:45.347867 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:31:45.347863    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad47d82f-6b21-4a15-9f2a-3ebb3d4134dc-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:31:45.993204 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:31:45.993172    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vjc66"
Oct 30 21:31:45.993204 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:31:45.993190    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vjc66" event={"ID":"ad47d82f-6b21-4a15-9f2a-3ebb3d4134dc","Type":"ContainerDied","Data":"c8da0207fa85e126fd17f8e7b786892f9a1668e4a590ad8d4040494b8a88d9a6"}
Oct 30 21:31:45.993429 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:31:45.993260    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8da0207fa85e126fd17f8e7b786892f9a1668e4a590ad8d4040494b8a88d9a6"
Oct 30 21:31:54.456981 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:31:54.456939    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:32:02.477028 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:02.476988    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-hzt4v"]
Oct 30 21:32:02.477484 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:02.477466    5713 cpu_manager.go:414] "containerMap: removing container" podUID="ad47d82f-6b21-4a15-9f2a-3ebb3d4134dc" containerName="revoke-keys"
Oct 30 21:32:02.477534 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:02.477487    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad47d82f-6b21-4a15-9f2a-3ebb3d4134dc" containerName="revoke-keys"
Oct 30 21:32:02.477633 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:02.477621    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad47d82f-6b21-4a15-9f2a-3ebb3d4134dc" containerName="revoke-keys"
Oct 30 21:32:02.481568 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:02.481541    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-hzt4v"
Oct 30 21:32:02.538941 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:02.538906    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-hzt4v"]
Oct 30 21:32:02.556469 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:02.556427    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/204b868f-aba0-4de3-86ed-7a16f4382dfc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-hzt4v\" (UID: \"204b868f-aba0-4de3-86ed-7a16f4382dfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-hzt4v"
Oct 30 21:32:02.556645 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:02.556546    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj54p\" (UniqueName: \"kubernetes.io/projected/204b868f-aba0-4de3-86ed-7a16f4382dfc-kube-api-access-xj54p\") pod \"maas-api-revoke-sub-0b7e8edb-hzt4v\" (UID: \"204b868f-aba0-4de3-86ed-7a16f4382dfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-hzt4v"
Oct 30 21:32:02.556645 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:02.556606    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/204b868f-aba0-4de3-86ed-7a16f4382dfc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-hzt4v\" (UID: \"204b868f-aba0-4de3-86ed-7a16f4382dfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-hzt4v"
Oct 30 21:32:02.584915 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:02.584879    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:32:02.585121 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:02.584940    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:32:02.585121 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:02.584952    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:32:02.585121 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:02.584984    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:32:02.658231 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:02.658189    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/204b868f-aba0-4de3-86ed-7a16f4382dfc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-hzt4v\" (UID: \"204b868f-aba0-4de3-86ed-7a16f4382dfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-hzt4v"
Oct 30 21:32:02.658447 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:02.658240    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj54p\" (UniqueName: \"kubernetes.io/projected/204b868f-aba0-4de3-86ed-7a16f4382dfc-kube-api-access-xj54p\") pod \"maas-api-revoke-sub-0b7e8edb-hzt4v\" (UID: \"204b868f-aba0-4de3-86ed-7a16f4382dfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-hzt4v"
Oct 30 21:32:02.658447 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:02.658278    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/204b868f-aba0-4de3-86ed-7a16f4382dfc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-hzt4v\" (UID: \"204b868f-aba0-4de3-86ed-7a16f4382dfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-hzt4v"
Oct 30 21:32:02.658983 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:02.658963    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/204b868f-aba0-4de3-86ed-7a16f4382dfc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-hzt4v\" (UID: \"204b868f-aba0-4de3-86ed-7a16f4382dfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-hzt4v"
Oct 30 21:32:02.661030 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:02.661007    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/204b868f-aba0-4de3-86ed-7a16f4382dfc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-hzt4v\" (UID: \"204b868f-aba0-4de3-86ed-7a16f4382dfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-hzt4v"
Oct 30 21:32:02.716307 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:02.716275    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj54p\" (UniqueName: \"kubernetes.io/projected/204b868f-aba0-4de3-86ed-7a16f4382dfc-kube-api-access-xj54p\") pod \"maas-api-revoke-sub-0b7e8edb-hzt4v\" (UID: \"204b868f-aba0-4de3-86ed-7a16f4382dfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-hzt4v"
Oct 30 21:32:02.796081 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:02.795993    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-hzt4v"
Oct 30 21:32:03.168140 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:03.168110    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-hzt4v"]
Oct 30 21:32:03.170036 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:32:03.170005    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod204b868f_aba0_4de3_86ed_7a16f4382dfc.slice/crio-1364f91f703181f12d9d5771f6f512f5789d9e3e382dabc1d55ec6fefdf5ba1e WatchSource:0}: Error finding container 1364f91f703181f12d9d5771f6f512f5789d9e3e382dabc1d55ec6fefdf5ba1e: Status 404 returned error can't find the container with id 1364f91f703181f12d9d5771f6f512f5789d9e3e382dabc1d55ec6fefdf5ba1e
Oct 30 21:32:04.137175 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:04.137135    5713 generic.go:356] "Generic (PLEG): container finished" podID="204b868f-aba0-4de3-86ed-7a16f4382dfc" containerID="27302790f9275276e40c5347387ce22cb77124cc266ba129a1273e8b50c5095e" exitCode=0
Oct 30 21:32:04.137581 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:04.137230    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-hzt4v" event={"ID":"204b868f-aba0-4de3-86ed-7a16f4382dfc","Type":"ContainerDied","Data":"27302790f9275276e40c5347387ce22cb77124cc266ba129a1273e8b50c5095e"}
Oct 30 21:32:04.137581 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:04.137285    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-hzt4v" event={"ID":"204b868f-aba0-4de3-86ed-7a16f4382dfc","Type":"ContainerStarted","Data":"1364f91f703181f12d9d5771f6f512f5789d9e3e382dabc1d55ec6fefdf5ba1e"}
Oct 30 21:32:05.279560 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:05.279536    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-hzt4v"
Oct 30 21:32:05.385749 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:05.385710    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/204b868f-aba0-4de3-86ed-7a16f4382dfc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/204b868f-aba0-4de3-86ed-7a16f4382dfc-maas-api-openshift-ca\") pod \"204b868f-aba0-4de3-86ed-7a16f4382dfc\" (UID: \"204b868f-aba0-4de3-86ed-7a16f4382dfc\") "
Oct 30 21:32:05.385973 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:05.385839    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/204b868f-aba0-4de3-86ed-7a16f4382dfc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/204b868f-aba0-4de3-86ed-7a16f4382dfc-maas-api-xks-ca\") pod \"204b868f-aba0-4de3-86ed-7a16f4382dfc\" (UID: \"204b868f-aba0-4de3-86ed-7a16f4382dfc\") "
Oct 30 21:32:05.385973 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:05.385866    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/204b868f-aba0-4de3-86ed-7a16f4382dfc-kube-api-access-xj54p\" (UniqueName: \"kubernetes.io/projected/204b868f-aba0-4de3-86ed-7a16f4382dfc-kube-api-access-xj54p\") pod \"204b868f-aba0-4de3-86ed-7a16f4382dfc\" (UID: \"204b868f-aba0-4de3-86ed-7a16f4382dfc\") "
Oct 30 21:32:05.386148 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:05.386119    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/204b868f-aba0-4de3-86ed-7a16f4382dfc-maas-api-openshift-ca" pod "204b868f-aba0-4de3-86ed-7a16f4382dfc" (UID: "204b868f-aba0-4de3-86ed-7a16f4382dfc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:32:05.388152 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:05.388127    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/204b868f-aba0-4de3-86ed-7a16f4382dfc-maas-api-xks-ca" pod "204b868f-aba0-4de3-86ed-7a16f4382dfc" (UID: "204b868f-aba0-4de3-86ed-7a16f4382dfc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:32:05.388261 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:05.388223    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/204b868f-aba0-4de3-86ed-7a16f4382dfc-kube-api-access-xj54p" pod "204b868f-aba0-4de3-86ed-7a16f4382dfc" (UID: "204b868f-aba0-4de3-86ed-7a16f4382dfc"). InnerVolumeSpecName "kube-api-access-xj54p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:05.486560 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:05.486521    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/204b868f-aba0-4de3-86ed-7a16f4382dfc-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:32:05.486560 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:05.486556    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xj54p\" (UniqueName: \"kubernetes.io/projected/204b868f-aba0-4de3-86ed-7a16f4382dfc-kube-api-access-xj54p\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:32:05.486560 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:05.486567    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/204b868f-aba0-4de3-86ed-7a16f4382dfc-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:32:06.150336 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:06.150301    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-hzt4v"
Oct 30 21:32:06.150336 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:06.150321    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-hzt4v" event={"ID":"204b868f-aba0-4de3-86ed-7a16f4382dfc","Type":"ContainerDied","Data":"1364f91f703181f12d9d5771f6f512f5789d9e3e382dabc1d55ec6fefdf5ba1e"}
Oct 30 21:32:06.150564 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:06.150368    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1364f91f703181f12d9d5771f6f512f5789d9e3e382dabc1d55ec6fefdf5ba1e"
Oct 30 21:32:13.807376 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:13.807343    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wlddm"]
Oct 30 21:32:13.807819 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:13.807805    5713 cpu_manager.go:414] "containerMap: removing container" podUID="204b868f-aba0-4de3-86ed-7a16f4382dfc" containerName="revoke-keys"
Oct 30 21:32:13.807873 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:13.807821    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="204b868f-aba0-4de3-86ed-7a16f4382dfc" containerName="revoke-keys"
Oct 30 21:32:13.807916 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:13.807911    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="204b868f-aba0-4de3-86ed-7a16f4382dfc" containerName="revoke-keys"
Oct 30 21:32:13.811102 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:13.811080    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wlddm"
Oct 30 21:32:13.870256 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:13.870211    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wlddm"]
Oct 30 21:32:13.870444 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:13.870342    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc5gp\" (UniqueName: \"kubernetes.io/projected/9f6b14b9-fcf0-4eab-9431-0964f8977310-kube-api-access-zc5gp\") pod \"maas-api-revoke-sub-3d4156f1-wlddm\" (UID: \"9f6b14b9-fcf0-4eab-9431-0964f8977310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wlddm"
Oct 30 21:32:13.870444 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:13.870399    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f6b14b9-fcf0-4eab-9431-0964f8977310-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wlddm\" (UID: \"9f6b14b9-fcf0-4eab-9431-0964f8977310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wlddm"
Oct 30 21:32:13.870579 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:13.870462    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f6b14b9-fcf0-4eab-9431-0964f8977310-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wlddm\" (UID: \"9f6b14b9-fcf0-4eab-9431-0964f8977310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wlddm"
Oct 30 21:32:13.915523 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:13.915492    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:32:13.915523 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:13.915492    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:32:13.915771 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:13.915528    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:32:13.915771 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:13.915586    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:32:13.971081 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:13.971046    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc5gp\" (UniqueName: \"kubernetes.io/projected/9f6b14b9-fcf0-4eab-9431-0964f8977310-kube-api-access-zc5gp\") pod \"maas-api-revoke-sub-3d4156f1-wlddm\" (UID: \"9f6b14b9-fcf0-4eab-9431-0964f8977310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wlddm"
Oct 30 21:32:13.971252 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:13.971100    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f6b14b9-fcf0-4eab-9431-0964f8977310-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wlddm\" (UID: \"9f6b14b9-fcf0-4eab-9431-0964f8977310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wlddm"
Oct 30 21:32:13.971252 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:13.971121    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f6b14b9-fcf0-4eab-9431-0964f8977310-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wlddm\" (UID: \"9f6b14b9-fcf0-4eab-9431-0964f8977310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wlddm"
Oct 30 21:32:13.971747 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:13.971730    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f6b14b9-fcf0-4eab-9431-0964f8977310-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wlddm\" (UID: \"9f6b14b9-fcf0-4eab-9431-0964f8977310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wlddm"
Oct 30 21:32:13.973765 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:13.973746    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f6b14b9-fcf0-4eab-9431-0964f8977310-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wlddm\" (UID: \"9f6b14b9-fcf0-4eab-9431-0964f8977310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wlddm"
Oct 30 21:32:14.030820 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:14.030755    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc5gp\" (UniqueName: \"kubernetes.io/projected/9f6b14b9-fcf0-4eab-9431-0964f8977310-kube-api-access-zc5gp\") pod \"maas-api-revoke-sub-3d4156f1-wlddm\" (UID: \"9f6b14b9-fcf0-4eab-9431-0964f8977310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wlddm"
Oct 30 21:32:14.124368 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:14.124292    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wlddm"
Oct 30 21:32:14.496973 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:14.496946    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wlddm"]
Oct 30 21:32:14.498983 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:32:14.498949    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f6b14b9_fcf0_4eab_9431_0964f8977310.slice/crio-3a00c64fa3788d8823478104c3c9947a014db230f4e138a627ed84ad4a4e0528 WatchSource:0}: Error finding container 3a00c64fa3788d8823478104c3c9947a014db230f4e138a627ed84ad4a4e0528: Status 404 returned error can't find the container with id 3a00c64fa3788d8823478104c3c9947a014db230f4e138a627ed84ad4a4e0528
Oct 30 21:32:15.202137 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:15.202100    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wlddm" event={"ID":"9f6b14b9-fcf0-4eab-9431-0964f8977310","Type":"ContainerStarted","Data":"63e01e8f6e7f3e32afe9d0b677e235f0ee37da75597593fca4e030be2b52713f"}
Oct 30 21:32:15.202568 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:15.202155    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wlddm" event={"ID":"9f6b14b9-fcf0-4eab-9431-0964f8977310","Type":"ContainerStarted","Data":"3a00c64fa3788d8823478104c3c9947a014db230f4e138a627ed84ad4a4e0528"}
Oct 30 21:32:15.315347 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:15.315285    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wlddm" podStartSLOduration=2.315266292 podStartE2EDuration="2.315266292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:15.314980236 +0000 UTC m=+3218.379756632" watchObservedRunningTime="2026-10-30 21:32:15.315266292 +0000 UTC m=+3218.380042693"
Oct 30 21:32:16.216073 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:16.216034    5713 generic.go:356] "Generic (PLEG): container finished" podID="9f6b14b9-fcf0-4eab-9431-0964f8977310" containerID="63e01e8f6e7f3e32afe9d0b677e235f0ee37da75597593fca4e030be2b52713f" exitCode=0
Oct 30 21:32:16.216617 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:16.216185    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wlddm" event={"ID":"9f6b14b9-fcf0-4eab-9431-0964f8977310","Type":"ContainerDied","Data":"63e01e8f6e7f3e32afe9d0b677e235f0ee37da75597593fca4e030be2b52713f"}
Oct 30 21:32:17.586172 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:17.585888    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wlddm"
Oct 30 21:32:17.627287 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:17.627244    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f6b14b9-fcf0-4eab-9431-0964f8977310-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f6b14b9-fcf0-4eab-9431-0964f8977310-maas-api-xks-ca\") pod \"9f6b14b9-fcf0-4eab-9431-0964f8977310\" (UID: \"9f6b14b9-fcf0-4eab-9431-0964f8977310\") "
Oct 30 21:32:17.629718 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:17.629688    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f6b14b9-fcf0-4eab-9431-0964f8977310-maas-api-xks-ca" pod "9f6b14b9-fcf0-4eab-9431-0964f8977310" (UID: "9f6b14b9-fcf0-4eab-9431-0964f8977310"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:32:17.728140 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:17.728100    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f6b14b9-fcf0-4eab-9431-0964f8977310-kube-api-access-zc5gp\" (UniqueName: \"kubernetes.io/projected/9f6b14b9-fcf0-4eab-9431-0964f8977310-kube-api-access-zc5gp\") pod \"9f6b14b9-fcf0-4eab-9431-0964f8977310\" (UID: \"9f6b14b9-fcf0-4eab-9431-0964f8977310\") "
Oct 30 21:32:17.728336 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:17.728156    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f6b14b9-fcf0-4eab-9431-0964f8977310-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f6b14b9-fcf0-4eab-9431-0964f8977310-maas-api-openshift-ca\") pod \"9f6b14b9-fcf0-4eab-9431-0964f8977310\" (UID: \"9f6b14b9-fcf0-4eab-9431-0964f8977310\") "
Oct 30 21:32:17.728547 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:17.728519    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f6b14b9-fcf0-4eab-9431-0964f8977310-maas-api-openshift-ca" pod "9f6b14b9-fcf0-4eab-9431-0964f8977310" (UID: "9f6b14b9-fcf0-4eab-9431-0964f8977310"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:32:17.728618 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:17.728530    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f6b14b9-fcf0-4eab-9431-0964f8977310-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:32:17.730650 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:17.730619    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f6b14b9-fcf0-4eab-9431-0964f8977310-kube-api-access-zc5gp" pod "9f6b14b9-fcf0-4eab-9431-0964f8977310" (UID: "9f6b14b9-fcf0-4eab-9431-0964f8977310"). InnerVolumeSpecName "kube-api-access-zc5gp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:17.829400 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:17.829303    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc5gp\" (UniqueName: \"kubernetes.io/projected/9f6b14b9-fcf0-4eab-9431-0964f8977310-kube-api-access-zc5gp\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:32:17.829400 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:17.829344    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f6b14b9-fcf0-4eab-9431-0964f8977310-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:32:18.229262 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:18.229233    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wlddm"
Oct 30 21:32:18.229443 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:18.229275    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wlddm" event={"ID":"9f6b14b9-fcf0-4eab-9431-0964f8977310","Type":"ContainerDied","Data":"3a00c64fa3788d8823478104c3c9947a014db230f4e138a627ed84ad4a4e0528"}
Oct 30 21:32:18.229443 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:18.229324    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a00c64fa3788d8823478104c3c9947a014db230f4e138a627ed84ad4a4e0528"
Oct 30 21:32:24.457661 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:32:24.457614    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:32:35.048780 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:35.048736    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wlddm"]
Oct 30 21:32:35.098252 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:35.098219    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wlddm"]
Oct 30 21:32:35.165818 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:35.165771    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ks6x5"]
Oct 30 21:32:35.166208 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:35.166196    5713 cpu_manager.go:414] "containerMap: removing container" podUID="9f6b14b9-fcf0-4eab-9431-0964f8977310" containerName="revoke-keys"
Oct 30 21:32:35.166256 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:35.166210    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f6b14b9-fcf0-4eab-9431-0964f8977310" containerName="revoke-keys"
Oct 30 21:32:35.166293 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:35.166286    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f6b14b9-fcf0-4eab-9431-0964f8977310" containerName="revoke-keys"
Oct 30 21:32:35.169475 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:35.169457    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ks6x5"
Oct 30 21:32:35.223727 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:35.223691    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:32:35.224017 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:35.223733    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:32:35.224175 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:35.223746    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:32:35.224307 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:35.223898    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:32:35.226845 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:35.226817    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ks6x5"]
Oct 30 21:32:35.257219 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:35.257178    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g98q\" (UniqueName: \"kubernetes.io/projected/8885719a-f22d-48c1-a816-478d0802cc92-kube-api-access-6g98q\") pod \"maas-api-revoke-sub-3d4156f1-ks6x5\" (UID: \"8885719a-f22d-48c1-a816-478d0802cc92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ks6x5"
Oct 30 21:32:35.257404 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:35.257238    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8885719a-f22d-48c1-a816-478d0802cc92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ks6x5\" (UID: \"8885719a-f22d-48c1-a816-478d0802cc92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ks6x5"
Oct 30 21:32:35.257404 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:35.257378    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8885719a-f22d-48c1-a816-478d0802cc92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ks6x5\" (UID: \"8885719a-f22d-48c1-a816-478d0802cc92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ks6x5"
Oct 30 21:32:35.358687 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:35.358600    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8885719a-f22d-48c1-a816-478d0802cc92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ks6x5\" (UID: \"8885719a-f22d-48c1-a816-478d0802cc92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ks6x5"
Oct 30 21:32:35.358896 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:35.358700    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g98q\" (UniqueName: \"kubernetes.io/projected/8885719a-f22d-48c1-a816-478d0802cc92-kube-api-access-6g98q\") pod \"maas-api-revoke-sub-3d4156f1-ks6x5\" (UID: \"8885719a-f22d-48c1-a816-478d0802cc92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ks6x5"
Oct 30 21:32:35.358896 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:35.358743    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8885719a-f22d-48c1-a816-478d0802cc92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ks6x5\" (UID: \"8885719a-f22d-48c1-a816-478d0802cc92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ks6x5"
Oct 30 21:32:35.359518 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:35.359452    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8885719a-f22d-48c1-a816-478d0802cc92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ks6x5\" (UID: \"8885719a-f22d-48c1-a816-478d0802cc92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ks6x5"
Oct 30 21:32:35.361368 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:35.361341    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8885719a-f22d-48c1-a816-478d0802cc92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ks6x5\" (UID: \"8885719a-f22d-48c1-a816-478d0802cc92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ks6x5"
Oct 30 21:32:35.416281 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:35.416241    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g98q\" (UniqueName: \"kubernetes.io/projected/8885719a-f22d-48c1-a816-478d0802cc92-kube-api-access-6g98q\") pod \"maas-api-revoke-sub-3d4156f1-ks6x5\" (UID: \"8885719a-f22d-48c1-a816-478d0802cc92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ks6x5"
Oct 30 21:32:35.482299 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:35.482260    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ks6x5"
Oct 30 21:32:35.864909 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:35.864878    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ks6x5"]
Oct 30 21:32:35.867818 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:32:35.867773    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8885719a_f22d_48c1_a816_478d0802cc92.slice/crio-1e4d3751e9a0d36778efa5788105f74ca499925254c1df9b14fd0db35e9e6534 WatchSource:0}: Error finding container 1e4d3751e9a0d36778efa5788105f74ca499925254c1df9b14fd0db35e9e6534: Status 404 returned error can't find the container with id 1e4d3751e9a0d36778efa5788105f74ca499925254c1df9b14fd0db35e9e6534
Oct 30 21:32:36.349523 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:36.349486    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ks6x5" event={"ID":"8885719a-f22d-48c1-a816-478d0802cc92","Type":"ContainerStarted","Data":"aa42a1bc8842228412c6428d702baabaf40824e42168b8605c01f84cd7ef9f73"}
Oct 30 21:32:36.349951 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:36.349551    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ks6x5" event={"ID":"8885719a-f22d-48c1-a816-478d0802cc92","Type":"ContainerStarted","Data":"1e4d3751e9a0d36778efa5788105f74ca499925254c1df9b14fd0db35e9e6534"}
Oct 30 21:32:37.062447 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:37.062360    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f6b14b9-fcf0-4eab-9431-0964f8977310" path="/var/lib/kubelet/pods/9f6b14b9-fcf0-4eab-9431-0964f8977310/volumes"
Oct 30 21:32:37.356441 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:37.356345    5713 generic.go:356] "Generic (PLEG): container finished" podID="8885719a-f22d-48c1-a816-478d0802cc92" containerID="aa42a1bc8842228412c6428d702baabaf40824e42168b8605c01f84cd7ef9f73" exitCode=0
Oct 30 21:32:37.356441 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:37.356426    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ks6x5" event={"ID":"8885719a-f22d-48c1-a816-478d0802cc92","Type":"ContainerDied","Data":"aa42a1bc8842228412c6428d702baabaf40824e42168b8605c01f84cd7ef9f73"}
Oct 30 21:32:38.513169 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:38.513139    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ks6x5"
Oct 30 21:32:38.594003 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:38.593967    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8885719a-f22d-48c1-a816-478d0802cc92-kube-api-access-6g98q\" (UniqueName: \"kubernetes.io/projected/8885719a-f22d-48c1-a816-478d0802cc92-kube-api-access-6g98q\") pod \"8885719a-f22d-48c1-a816-478d0802cc92\" (UID: \"8885719a-f22d-48c1-a816-478d0802cc92\") "
Oct 30 21:32:38.594204 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:38.594065    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8885719a-f22d-48c1-a816-478d0802cc92-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8885719a-f22d-48c1-a816-478d0802cc92-maas-api-openshift-ca\") pod \"8885719a-f22d-48c1-a816-478d0802cc92\" (UID: \"8885719a-f22d-48c1-a816-478d0802cc92\") "
Oct 30 21:32:38.594204 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:38.594134    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8885719a-f22d-48c1-a816-478d0802cc92-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8885719a-f22d-48c1-a816-478d0802cc92-maas-api-xks-ca\") pod \"8885719a-f22d-48c1-a816-478d0802cc92\" (UID: \"8885719a-f22d-48c1-a816-478d0802cc92\") "
Oct 30 21:32:38.594409 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:38.594384    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8885719a-f22d-48c1-a816-478d0802cc92-maas-api-openshift-ca" pod "8885719a-f22d-48c1-a816-478d0802cc92" (UID: "8885719a-f22d-48c1-a816-478d0802cc92"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:32:38.596245 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:38.596226    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8885719a-f22d-48c1-a816-478d0802cc92-kube-api-access-6g98q" pod "8885719a-f22d-48c1-a816-478d0802cc92" (UID: "8885719a-f22d-48c1-a816-478d0802cc92"). InnerVolumeSpecName "kube-api-access-6g98q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:38.596413 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:38.596391    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8885719a-f22d-48c1-a816-478d0802cc92-maas-api-xks-ca" pod "8885719a-f22d-48c1-a816-478d0802cc92" (UID: "8885719a-f22d-48c1-a816-478d0802cc92"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:32:38.695210 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:38.695177    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8885719a-f22d-48c1-a816-478d0802cc92-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:32:38.695210 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:38.695206    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6g98q\" (UniqueName: \"kubernetes.io/projected/8885719a-f22d-48c1-a816-478d0802cc92-kube-api-access-6g98q\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:32:38.695210 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:38.695216    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8885719a-f22d-48c1-a816-478d0802cc92-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:32:39.370014 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:39.369927    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ks6x5" event={"ID":"8885719a-f22d-48c1-a816-478d0802cc92","Type":"ContainerDied","Data":"1e4d3751e9a0d36778efa5788105f74ca499925254c1df9b14fd0db35e9e6534"}
Oct 30 21:32:39.370014 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:39.369976    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e4d3751e9a0d36778efa5788105f74ca499925254c1df9b14fd0db35e9e6534"
Oct 30 21:32:39.370014 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:39.369940    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ks6x5"
Oct 30 21:32:40.220950 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:40.220919    5713 scope.go:122] "RemoveContainer" containerID="be9e42f3f1c03db862a405b5f2d0907e947d6003a191517fe270c87cfc496fa2"
Oct 30 21:32:40.232710 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:40.232688    5713 scope.go:122] "RemoveContainer" containerID="b05c642c9c36ea9ef02a6b3cdf72e43519352e6d619c8195ea8496348a324c25"
Oct 30 21:32:40.241504 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:40.241484    5713 scope.go:122] "RemoveContainer" containerID="f7fa31b5ecd37ec82365c2d45e7d7754843670d9f674029a53ad46b2a19c45d5"
Oct 30 21:32:40.249628 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:32:40.249605    5713 scope.go:122] "RemoveContainer" containerID="4e31423691724b64d6b96d290bf1e36347a15141ae8dbae375ce5da6bb604cea"
Oct 30 21:32:54.458727 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:32:54.458646    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:33:10.088471 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:10.088437    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rvj4r"]
Oct 30 21:33:10.088977 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:10.088854    5713 cpu_manager.go:414] "containerMap: removing container" podUID="8885719a-f22d-48c1-a816-478d0802cc92" containerName="revoke-keys"
Oct 30 21:33:10.088977 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:10.088866    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8885719a-f22d-48c1-a816-478d0802cc92" containerName="revoke-keys"
Oct 30 21:33:10.088977 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:10.088948    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="8885719a-f22d-48c1-a816-478d0802cc92" containerName="revoke-keys"
Oct 30 21:33:10.092083 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:10.092066    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rvj4r"
Oct 30 21:33:10.147984 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:10.147933    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:33:10.153909 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:10.153822    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw8mn\" (UniqueName: \"kubernetes.io/projected/eb0e8b23-8074-49f5-b460-dcf5f750cb63-kube-api-access-hw8mn\") pod \"maas-api-revoke-sub-e2fc9c5b-rvj4r\" (UID: \"eb0e8b23-8074-49f5-b460-dcf5f750cb63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rvj4r"
Oct 30 21:33:10.154253 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:10.154224    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb0e8b23-8074-49f5-b460-dcf5f750cb63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-rvj4r\" (UID: \"eb0e8b23-8074-49f5-b460-dcf5f750cb63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rvj4r"
Oct 30 21:33:10.154394 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:10.154375    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb0e8b23-8074-49f5-b460-dcf5f750cb63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-rvj4r\" (UID: \"eb0e8b23-8074-49f5-b460-dcf5f750cb63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rvj4r"
Oct 30 21:33:10.197297 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:10.197258    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:33:10.197476 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:10.197303    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:33:10.197476 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:10.197384    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:33:10.199661 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:10.199641    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rvj4r"]
Oct 30 21:33:10.255663 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:10.255627    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw8mn\" (UniqueName: \"kubernetes.io/projected/eb0e8b23-8074-49f5-b460-dcf5f750cb63-kube-api-access-hw8mn\") pod \"maas-api-revoke-sub-e2fc9c5b-rvj4r\" (UID: \"eb0e8b23-8074-49f5-b460-dcf5f750cb63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rvj4r"
Oct 30 21:33:10.255663 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:10.255669    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb0e8b23-8074-49f5-b460-dcf5f750cb63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-rvj4r\" (UID: \"eb0e8b23-8074-49f5-b460-dcf5f750cb63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rvj4r"
Oct 30 21:33:10.255997 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:10.255723    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb0e8b23-8074-49f5-b460-dcf5f750cb63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-rvj4r\" (UID: \"eb0e8b23-8074-49f5-b460-dcf5f750cb63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rvj4r"
Oct 30 21:33:10.256365 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:10.256345    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb0e8b23-8074-49f5-b460-dcf5f750cb63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-rvj4r\" (UID: \"eb0e8b23-8074-49f5-b460-dcf5f750cb63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rvj4r"
Oct 30 21:33:10.258246 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:10.258224    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb0e8b23-8074-49f5-b460-dcf5f750cb63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-rvj4r\" (UID: \"eb0e8b23-8074-49f5-b460-dcf5f750cb63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rvj4r"
Oct 30 21:33:10.313640 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:10.313602    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw8mn\" (UniqueName: \"kubernetes.io/projected/eb0e8b23-8074-49f5-b460-dcf5f750cb63-kube-api-access-hw8mn\") pod \"maas-api-revoke-sub-e2fc9c5b-rvj4r\" (UID: \"eb0e8b23-8074-49f5-b460-dcf5f750cb63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rvj4r"
Oct 30 21:33:10.405200 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:10.405108    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rvj4r"
Oct 30 21:33:10.783831 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:10.783805    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rvj4r"]
Oct 30 21:33:10.786256 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:33:10.786232    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb0e8b23_8074_49f5_b460_dcf5f750cb63.slice/crio-40ae09b651abe9f83eee5c56775810f677e23ad774211e9842b5507b4883e752 WatchSource:0}: Error finding container 40ae09b651abe9f83eee5c56775810f677e23ad774211e9842b5507b4883e752: Status 404 returned error can't find the container with id 40ae09b651abe9f83eee5c56775810f677e23ad774211e9842b5507b4883e752
Oct 30 21:33:11.556720 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:11.556683    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rvj4r" event={"ID":"eb0e8b23-8074-49f5-b460-dcf5f750cb63","Type":"ContainerStarted","Data":"7881b4912c5143d50ebcb0e26a0e14ea6ee10c4e9cb64dd783ddbc6624f566b5"}
Oct 30 21:33:11.557153 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:11.556757    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rvj4r" event={"ID":"eb0e8b23-8074-49f5-b460-dcf5f750cb63","Type":"ContainerStarted","Data":"40ae09b651abe9f83eee5c56775810f677e23ad774211e9842b5507b4883e752"}
Oct 30 21:33:11.670023 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:11.669973    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rvj4r" podStartSLOduration=1.669957685 podStartE2EDuration="1.669957685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:33:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:33:11.669063712 +0000 UTC m=+3274.733840120" watchObservedRunningTime="2026-10-30 21:33:11.669957685 +0000 UTC m=+3274.734734083"
Oct 30 21:33:12.565082 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:12.565043    5713 generic.go:356] "Generic (PLEG): container finished" podID="eb0e8b23-8074-49f5-b460-dcf5f750cb63" containerID="7881b4912c5143d50ebcb0e26a0e14ea6ee10c4e9cb64dd783ddbc6624f566b5" exitCode=0
Oct 30 21:33:12.565554 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:12.565135    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rvj4r" event={"ID":"eb0e8b23-8074-49f5-b460-dcf5f750cb63","Type":"ContainerDied","Data":"7881b4912c5143d50ebcb0e26a0e14ea6ee10c4e9cb64dd783ddbc6624f566b5"}
Oct 30 21:33:13.709574 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:13.709547    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rvj4r"
Oct 30 21:33:13.795691 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:13.795655    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb0e8b23-8074-49f5-b460-dcf5f750cb63-kube-api-access-hw8mn\" (UniqueName: \"kubernetes.io/projected/eb0e8b23-8074-49f5-b460-dcf5f750cb63-kube-api-access-hw8mn\") pod \"eb0e8b23-8074-49f5-b460-dcf5f750cb63\" (UID: \"eb0e8b23-8074-49f5-b460-dcf5f750cb63\") "
Oct 30 21:33:13.795691 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:13.795698    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb0e8b23-8074-49f5-b460-dcf5f750cb63-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb0e8b23-8074-49f5-b460-dcf5f750cb63-maas-api-xks-ca\") pod \"eb0e8b23-8074-49f5-b460-dcf5f750cb63\" (UID: \"eb0e8b23-8074-49f5-b460-dcf5f750cb63\") "
Oct 30 21:33:13.796020 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:13.795842    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb0e8b23-8074-49f5-b460-dcf5f750cb63-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb0e8b23-8074-49f5-b460-dcf5f750cb63-maas-api-openshift-ca\") pod \"eb0e8b23-8074-49f5-b460-dcf5f750cb63\" (UID: \"eb0e8b23-8074-49f5-b460-dcf5f750cb63\") "
Oct 30 21:33:13.796240 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:13.796213    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb0e8b23-8074-49f5-b460-dcf5f750cb63-maas-api-openshift-ca" pod "eb0e8b23-8074-49f5-b460-dcf5f750cb63" (UID: "eb0e8b23-8074-49f5-b460-dcf5f750cb63"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:33:13.798183 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:13.798152    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb0e8b23-8074-49f5-b460-dcf5f750cb63-maas-api-xks-ca" pod "eb0e8b23-8074-49f5-b460-dcf5f750cb63" (UID: "eb0e8b23-8074-49f5-b460-dcf5f750cb63"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:33:13.798288 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:13.798158    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb0e8b23-8074-49f5-b460-dcf5f750cb63-kube-api-access-hw8mn" pod "eb0e8b23-8074-49f5-b460-dcf5f750cb63" (UID: "eb0e8b23-8074-49f5-b460-dcf5f750cb63"). InnerVolumeSpecName "kube-api-access-hw8mn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:33:13.897176 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:13.897086    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hw8mn\" (UniqueName: \"kubernetes.io/projected/eb0e8b23-8074-49f5-b460-dcf5f750cb63-kube-api-access-hw8mn\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:33:13.897176 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:13.897121    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb0e8b23-8074-49f5-b460-dcf5f750cb63-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:33:13.897176 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:13.897130    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb0e8b23-8074-49f5-b460-dcf5f750cb63-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:33:14.577917 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:14.577885    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rvj4r"
Oct 30 21:33:14.577917 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:14.577909    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rvj4r" event={"ID":"eb0e8b23-8074-49f5-b460-dcf5f750cb63","Type":"ContainerDied","Data":"40ae09b651abe9f83eee5c56775810f677e23ad774211e9842b5507b4883e752"}
Oct 30 21:33:14.578162 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:14.577956    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40ae09b651abe9f83eee5c56775810f677e23ad774211e9842b5507b4883e752"
Oct 30 21:33:24.521022 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:33:24.520980    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:33:37.799810 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:37.799759    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d4597073-8ff0-4a4e-933c-c64e934fe977/prometheus/0.log"
Oct 30 21:33:37.805633 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:37.805611    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d4597073-8ff0-4a4e-933c-c64e934fe977/prometheus/0.log"
Oct 30 21:33:37.813776 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:37.813754    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r7lpf_764461d5-e0b2-4856-a97e-0c5d3d34a497/ovn-acl-logging/0.log"
Oct 30 21:33:37.819311 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:37.819293    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r7lpf_764461d5-e0b2-4856-a97e-0c5d3d34a497/ovn-acl-logging/0.log"
Oct 30 21:33:38.049775 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:38.049745    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ks6x5"]
Oct 30 21:33:38.099207 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:38.099139    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ks6x5"]
Oct 30 21:33:38.171412 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:38.171385    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-slbn6"]
Oct 30 21:33:38.171851 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:38.171837    5713 cpu_manager.go:414] "containerMap: removing container" podUID="eb0e8b23-8074-49f5-b460-dcf5f750cb63" containerName="revoke-keys"
Oct 30 21:33:38.171908 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:38.171852    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb0e8b23-8074-49f5-b460-dcf5f750cb63" containerName="revoke-keys"
Oct 30 21:33:38.171952 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:38.171942    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb0e8b23-8074-49f5-b460-dcf5f750cb63" containerName="revoke-keys"
Oct 30 21:33:38.175045 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:38.175028    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-slbn6"
Oct 30 21:33:38.231314 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:38.231224    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:33:38.231314 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:38.231226    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:33:38.231314 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:38.231285    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:33:38.231314 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:38.231290    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:33:38.232845 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:38.232821    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-slbn6"]
Oct 30 21:33:38.250948 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:38.250926    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dae11545-f309-491f-a41f-5154396960b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-slbn6\" (UID: \"dae11545-f309-491f-a41f-5154396960b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-slbn6"
Oct 30 21:33:38.251062 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:38.250974    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb96w\" (UniqueName: \"kubernetes.io/projected/dae11545-f309-491f-a41f-5154396960b1-kube-api-access-bb96w\") pod \"maas-api-revoke-sub-3d4156f1-slbn6\" (UID: \"dae11545-f309-491f-a41f-5154396960b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-slbn6"
Oct 30 21:33:38.251062 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:38.251046    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dae11545-f309-491f-a41f-5154396960b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-slbn6\" (UID: \"dae11545-f309-491f-a41f-5154396960b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-slbn6"
Oct 30 21:33:38.351831 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:38.351739    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dae11545-f309-491f-a41f-5154396960b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-slbn6\" (UID: \"dae11545-f309-491f-a41f-5154396960b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-slbn6"
Oct 30 21:33:38.351831 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:38.351814    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb96w\" (UniqueName: \"kubernetes.io/projected/dae11545-f309-491f-a41f-5154396960b1-kube-api-access-bb96w\") pod \"maas-api-revoke-sub-3d4156f1-slbn6\" (UID: \"dae11545-f309-491f-a41f-5154396960b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-slbn6"
Oct 30 21:33:38.352015 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:38.351850    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dae11545-f309-491f-a41f-5154396960b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-slbn6\" (UID: \"dae11545-f309-491f-a41f-5154396960b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-slbn6"
Oct 30 21:33:38.352518 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:38.352453    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dae11545-f309-491f-a41f-5154396960b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-slbn6\" (UID: \"dae11545-f309-491f-a41f-5154396960b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-slbn6"
Oct 30 21:33:38.354193 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:38.354174    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dae11545-f309-491f-a41f-5154396960b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-slbn6\" (UID: \"dae11545-f309-491f-a41f-5154396960b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-slbn6"
Oct 30 21:33:38.408251 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:38.408223    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb96w\" (UniqueName: \"kubernetes.io/projected/dae11545-f309-491f-a41f-5154396960b1-kube-api-access-bb96w\") pod \"maas-api-revoke-sub-3d4156f1-slbn6\" (UID: \"dae11545-f309-491f-a41f-5154396960b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-slbn6"
Oct 30 21:33:38.488235 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:38.488207    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-slbn6"
Oct 30 21:33:38.858840 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:38.858817    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-slbn6"]
Oct 30 21:33:38.860576 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:33:38.860547    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddae11545_f309_491f_a41f_5154396960b1.slice/crio-91172ed20b75a6ce3bfff94207eee7311660c119c7b5c06b4b498810784f285f WatchSource:0}: Error finding container 91172ed20b75a6ce3bfff94207eee7311660c119c7b5c06b4b498810784f285f: Status 404 returned error can't find the container with id 91172ed20b75a6ce3bfff94207eee7311660c119c7b5c06b4b498810784f285f
Oct 30 21:33:39.061754 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:39.061722    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8885719a-f22d-48c1-a816-478d0802cc92" path="/var/lib/kubelet/pods/8885719a-f22d-48c1-a816-478d0802cc92/volumes"
Oct 30 21:33:39.732014 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:39.731979    5713 generic.go:356] "Generic (PLEG): container finished" podID="dae11545-f309-491f-a41f-5154396960b1" containerID="d66068f65643fdf4c2431984d9bf02947093e2d5914850f5333712a979262d11" exitCode=0
Oct 30 21:33:39.732172 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:39.732019    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-slbn6" event={"ID":"dae11545-f309-491f-a41f-5154396960b1","Type":"ContainerDied","Data":"d66068f65643fdf4c2431984d9bf02947093e2d5914850f5333712a979262d11"}
Oct 30 21:33:39.732172 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:39.732071    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-slbn6" event={"ID":"dae11545-f309-491f-a41f-5154396960b1","Type":"ContainerStarted","Data":"91172ed20b75a6ce3bfff94207eee7311660c119c7b5c06b4b498810784f285f"}
Oct 30 21:33:40.872846 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:40.872818    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-slbn6"
Oct 30 21:33:40.977342 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:40.977312    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dae11545-f309-491f-a41f-5154396960b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dae11545-f309-491f-a41f-5154396960b1-maas-api-openshift-ca\") pod \"dae11545-f309-491f-a41f-5154396960b1\" (UID: \"dae11545-f309-491f-a41f-5154396960b1\") "
Oct 30 21:33:40.977504 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:40.977365    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dae11545-f309-491f-a41f-5154396960b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dae11545-f309-491f-a41f-5154396960b1-maas-api-xks-ca\") pod \"dae11545-f309-491f-a41f-5154396960b1\" (UID: \"dae11545-f309-491f-a41f-5154396960b1\") "
Oct 30 21:33:40.977581 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:40.977498    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dae11545-f309-491f-a41f-5154396960b1-kube-api-access-bb96w\" (UniqueName: \"kubernetes.io/projected/dae11545-f309-491f-a41f-5154396960b1-kube-api-access-bb96w\") pod \"dae11545-f309-491f-a41f-5154396960b1\" (UID: \"dae11545-f309-491f-a41f-5154396960b1\") "
Oct 30 21:33:40.977760 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:40.977737    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dae11545-f309-491f-a41f-5154396960b1-maas-api-openshift-ca" pod "dae11545-f309-491f-a41f-5154396960b1" (UID: "dae11545-f309-491f-a41f-5154396960b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:33:40.978386 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:40.978356    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dae11545-f309-491f-a41f-5154396960b1-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:33:40.989642 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:40.980911    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dae11545-f309-491f-a41f-5154396960b1-kube-api-access-bb96w" pod "dae11545-f309-491f-a41f-5154396960b1" (UID: "dae11545-f309-491f-a41f-5154396960b1"). InnerVolumeSpecName "kube-api-access-bb96w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:33:40.989642 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:40.989288    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dae11545-f309-491f-a41f-5154396960b1-maas-api-xks-ca" pod "dae11545-f309-491f-a41f-5154396960b1" (UID: "dae11545-f309-491f-a41f-5154396960b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:33:41.079579 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:41.079552    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dae11545-f309-491f-a41f-5154396960b1-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:33:41.079741 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:41.079580    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bb96w\" (UniqueName: \"kubernetes.io/projected/dae11545-f309-491f-a41f-5154396960b1-kube-api-access-bb96w\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:33:41.745922 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:41.745882    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-slbn6" event={"ID":"dae11545-f309-491f-a41f-5154396960b1","Type":"ContainerDied","Data":"91172ed20b75a6ce3bfff94207eee7311660c119c7b5c06b4b498810784f285f"}
Oct 30 21:33:41.745922 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:41.745908    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-slbn6"
Oct 30 21:33:41.746196 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:33:41.745937    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91172ed20b75a6ce3bfff94207eee7311660c119c7b5c06b4b498810784f285f"
Oct 30 21:33:54.459195 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:33:54.459149    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:34:02.702439 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:02.702401    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-89af8e-54695b5qtr"]
Oct 30 21:34:02.703119 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:02.703030    5713 cpu_manager.go:414] "containerMap: removing container" podUID="dae11545-f309-491f-a41f-5154396960b1" containerName="revoke-keys"
Oct 30 21:34:02.703119 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:02.703052    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dae11545-f309-491f-a41f-5154396960b1" containerName="revoke-keys"
Oct 30 21:34:02.703253 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:02.703179    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="dae11545-f309-491f-a41f-5154396960b1" containerName="revoke-keys"
Oct 30 21:34:02.706692 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:02.706665    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-89af8e-54695b5qtr"
Oct 30 21:34:02.767938 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:02.767893    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-89af8e\""
Oct 30 21:34:02.768140 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:02.768058    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-19285ec6-dockercfg-64v7h\""
Oct 30 21:34:02.797511 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:02.797475    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch46l\" (UniqueName: \"kubernetes.io/projected/1ea5a200-89c8-4cd6-b319-a438e0c9e553-kube-api-access-ch46l\") pod \"payload-pre-processing-e2e-worker-single-main-89af8e-54695b5qtr\" (UID: \"1ea5a200-89c8-4cd6-b319-a438e0c9e553\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-89af8e-54695b5qtr"
Oct 30 21:34:02.797691 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:02.797534    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ea5a200-89c8-4cd6-b319-a438e0c9e553-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-89af8e-54695b5qtr\" (UID: \"1ea5a200-89c8-4cd6-b319-a438e0c9e553\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-89af8e-54695b5qtr"
Oct 30 21:34:02.811181 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:02.811148    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-89af8e-54695b5qtr"]
Oct 30 21:34:02.844310 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:02.844248    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-89af8e-7bd9b978bfbfhx"]
Oct 30 21:34:02.848381 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:02.848359    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-89af8e-7bd9b978bfbfhx"
Oct 30 21:34:02.898891 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:02.898847    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrsfd\" (UniqueName: \"kubernetes.io/projected/cf9fde33-48d1-462c-be89-ef10584fd6a5-kube-api-access-vrsfd\") pod \"payload-processing-e2e-worker-single-main-89af8e-7bd9b978bfbfhx\" (UID: \"cf9fde33-48d1-462c-be89-ef10584fd6a5\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-89af8e-7bd9b978bfbfhx"
Oct 30 21:34:02.899086 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:02.898972    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf9fde33-48d1-462c-be89-ef10584fd6a5-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-89af8e-7bd9b978bfbfhx\" (UID: \"cf9fde33-48d1-462c-be89-ef10584fd6a5\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-89af8e-7bd9b978bfbfhx"
Oct 30 21:34:02.899086 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:02.899030    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch46l\" (UniqueName: \"kubernetes.io/projected/1ea5a200-89c8-4cd6-b319-a438e0c9e553-kube-api-access-ch46l\") pod \"payload-pre-processing-e2e-worker-single-main-89af8e-54695b5qtr\" (UID: \"1ea5a200-89c8-4cd6-b319-a438e0c9e553\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-89af8e-54695b5qtr"
Oct 30 21:34:02.899086 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:02.899079    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ea5a200-89c8-4cd6-b319-a438e0c9e553-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-89af8e-54695b5qtr\" (UID: \"1ea5a200-89c8-4cd6-b319-a438e0c9e553\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-89af8e-54695b5qtr"
Oct 30 21:34:02.899694 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:02.899676    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ea5a200-89c8-4cd6-b319-a438e0c9e553-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-89af8e-54695b5qtr\" (UID: \"1ea5a200-89c8-4cd6-b319-a438e0c9e553\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-89af8e-54695b5qtr"
Oct 30 21:34:02.904940 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:02.904912    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-89af8e-7bd9b978bfbfhx"]
Oct 30 21:34:02.955768 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:02.955699    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch46l\" (UniqueName: \"kubernetes.io/projected/1ea5a200-89c8-4cd6-b319-a438e0c9e553-kube-api-access-ch46l\") pod \"payload-pre-processing-e2e-worker-single-main-89af8e-54695b5qtr\" (UID: \"1ea5a200-89c8-4cd6-b319-a438e0c9e553\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-89af8e-54695b5qtr"
Oct 30 21:34:03.000259 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:03.000223    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf9fde33-48d1-462c-be89-ef10584fd6a5-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-89af8e-7bd9b978bfbfhx\" (UID: \"cf9fde33-48d1-462c-be89-ef10584fd6a5\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-89af8e-7bd9b978bfbfhx"
Oct 30 21:34:03.000440 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:03.000335    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrsfd\" (UniqueName: \"kubernetes.io/projected/cf9fde33-48d1-462c-be89-ef10584fd6a5-kube-api-access-vrsfd\") pod \"payload-processing-e2e-worker-single-main-89af8e-7bd9b978bfbfhx\" (UID: \"cf9fde33-48d1-462c-be89-ef10584fd6a5\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-89af8e-7bd9b978bfbfhx"
Oct 30 21:34:03.000984 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:03.000962    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf9fde33-48d1-462c-be89-ef10584fd6a5-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-89af8e-7bd9b978bfbfhx\" (UID: \"cf9fde33-48d1-462c-be89-ef10584fd6a5\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-89af8e-7bd9b978bfbfhx"
Oct 30 21:34:03.019650 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:03.019621    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-89af8e-54695b5qtr"
Oct 30 21:34:03.056954 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:03.056925    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrsfd\" (UniqueName: \"kubernetes.io/projected/cf9fde33-48d1-462c-be89-ef10584fd6a5-kube-api-access-vrsfd\") pod \"payload-processing-e2e-worker-single-main-89af8e-7bd9b978bfbfhx\" (UID: \"cf9fde33-48d1-462c-be89-ef10584fd6a5\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-89af8e-7bd9b978bfbfhx"
Oct 30 21:34:03.161840 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:03.161776    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-89af8e-7bd9b978bfbfhx"
Oct 30 21:34:03.399935 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:03.399904    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-89af8e-54695b5qtr"]
Oct 30 21:34:03.401520 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:34:03.401492    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ea5a200_89c8_4cd6_b319_a438e0c9e553.slice/crio-952f7c4666c63747e14aa2f3d1c80cadab90d748e2321ed87da0dbdb8389421a WatchSource:0}: Error finding container 952f7c4666c63747e14aa2f3d1c80cadab90d748e2321ed87da0dbdb8389421a: Status 404 returned error can't find the container with id 952f7c4666c63747e14aa2f3d1c80cadab90d748e2321ed87da0dbdb8389421a
Oct 30 21:34:03.540258 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:03.540232    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-89af8e-7bd9b978bfbfhx"]
Oct 30 21:34:03.541749 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:34:03.541722    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf9fde33_48d1_462c_be89_ef10584fd6a5.slice/crio-f2539bf5219bd81072ccd175c3ef7f26ab0171a1660bb553f87a40f678448775 WatchSource:0}: Error finding container f2539bf5219bd81072ccd175c3ef7f26ab0171a1660bb553f87a40f678448775: Status 404 returned error can't find the container with id f2539bf5219bd81072ccd175c3ef7f26ab0171a1660bb553f87a40f678448775
Oct 30 21:34:03.875490 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:03.875399    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-89af8e-7bd9b978bfbfhx" event={"ID":"cf9fde33-48d1-462c-be89-ef10584fd6a5","Type":"ContainerStarted","Data":"8d20f8671667fe024a685a16f20cff23712bdfa781c9b843aef67638c22c211c"}
Oct 30 21:34:03.875490 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:03.875453    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-89af8e-7bd9b978bfbfhx" event={"ID":"cf9fde33-48d1-462c-be89-ef10584fd6a5","Type":"ContainerStarted","Data":"f2539bf5219bd81072ccd175c3ef7f26ab0171a1660bb553f87a40f678448775"}
Oct 30 21:34:03.876046 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:03.875519    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-89af8e-7bd9b978bfbfhx"
Oct 30 21:34:03.877230 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:03.877197    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-89af8e-54695b5qtr" event={"ID":"1ea5a200-89c8-4cd6-b319-a438e0c9e553","Type":"ContainerStarted","Data":"8a15ae4d8e29507d27b86543a7c3ad1ec05d19c96e1fea4d040eb46aa404e24a"}
Oct 30 21:34:03.877380 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:03.877248    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-89af8e-54695b5qtr" event={"ID":"1ea5a200-89c8-4cd6-b319-a438e0c9e553","Type":"ContainerStarted","Data":"952f7c4666c63747e14aa2f3d1c80cadab90d748e2321ed87da0dbdb8389421a"}
Oct 30 21:34:03.877380 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:03.877276    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-89af8e-54695b5qtr"
Oct 30 21:34:03.991370 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:03.991320    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-89af8e-7bd9b978bfbfhx" podStartSLOduration=1.99130423 podStartE2EDuration="1.99130423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:34:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:34:03.988329508 +0000 UTC m=+3327.053105907" watchObservedRunningTime="2026-10-30 21:34:03.99130423 +0000 UTC m=+3327.056080627"
Oct 30 21:34:04.104818 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:04.104653    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-89af8e-54695b5qtr" podStartSLOduration=2.104639153 podStartE2EDuration="2.104639153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:34:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:34:04.101037145 +0000 UTC m=+3327.165813547" watchObservedRunningTime="2026-10-30 21:34:04.104639153 +0000 UTC m=+3327.169415551"
Oct 30 21:34:14.047684 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:14.047649    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5xsnk"]
Oct 30 21:34:14.097327 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:14.097299    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5xsnk"]
Oct 30 21:34:14.885128 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:14.885102    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-89af8e-7bd9b978bfbfhx"
Oct 30 21:34:14.885309 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:14.885169    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-89af8e-54695b5qtr"
Oct 30 21:34:15.062417 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:15.062385    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74888553-9594-4086-9148-8d788092c3fa" path="/var/lib/kubelet/pods/74888553-9594-4086-9148-8d788092c3fa/volumes"
Oct 30 21:34:24.459830 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:34:24.459772    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:34:26.050125 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:26.050085    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2zjt5"]
Oct 30 21:34:26.099607 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:26.099575    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-2zjt5"]
Oct 30 21:34:27.009184 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:27.009154    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-89af8e/simulator-single-main-89af8e-kserve-756c5cc577-qph4k"]
Oct 30 21:34:27.013234 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:27.013210    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-89af8e/simulator-single-main-89af8e-kserve-756c5cc577-qph4k"
Oct 30 21:34:27.062225 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:27.062190    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7067c7a6-7e16-4dea-9379-227c0ec27956" path="/var/lib/kubelet/pods/7067c7a6-7e16-4dea-9379-227c0ec27956/volumes"
Oct 30 21:34:27.116371 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:27.116331    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-89af8e/simulator-single-main-89af8e-kserve-756c5cc577-qph4k"]
Oct 30 21:34:27.119074 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:27.119052    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-89af8e\"/\"openshift-service-ca.crt\""
Oct 30 21:34:27.119455 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:27.119435    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-89af8e\"/\"default-dockercfg-25wgw\""
Oct 30 21:34:27.119551 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:27.119507    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-89af8e\"/\"kube-root-ca.crt\""
Oct 30 21:34:27.119551 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:27.119527    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-89af8e\"/\"simulator-single-main-89af8e-kserve-self-signed-certs\""
Oct 30 21:34:27.145254 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:27.145214    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84mph\" (UniqueName: \"kubernetes.io/projected/22330f0d-0188-4060-a692-86686cfd517a-kube-api-access-84mph\") pod \"simulator-single-main-89af8e-kserve-756c5cc577-qph4k\" (UID: \"22330f0d-0188-4060-a692-86686cfd517a\") " pod="e2e-models-e2e-worker-single-main-89af8e/simulator-single-main-89af8e-kserve-756c5cc577-qph4k"
Oct 30 21:34:27.145472 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:27.145442    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22330f0d-0188-4060-a692-86686cfd517a-home\") pod \"simulator-single-main-89af8e-kserve-756c5cc577-qph4k\" (UID: \"22330f0d-0188-4060-a692-86686cfd517a\") " pod="e2e-models-e2e-worker-single-main-89af8e/simulator-single-main-89af8e-kserve-756c5cc577-qph4k"
Oct 30 21:34:27.145592 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:27.145566    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22330f0d-0188-4060-a692-86686cfd517a-tls-certs\") pod \"simulator-single-main-89af8e-kserve-756c5cc577-qph4k\" (UID: \"22330f0d-0188-4060-a692-86686cfd517a\") " pod="e2e-models-e2e-worker-single-main-89af8e/simulator-single-main-89af8e-kserve-756c5cc577-qph4k"
Oct 30 21:34:27.145890 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:27.145852    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22330f0d-0188-4060-a692-86686cfd517a-dshm\") pod \"simulator-single-main-89af8e-kserve-756c5cc577-qph4k\" (UID: \"22330f0d-0188-4060-a692-86686cfd517a\") " pod="e2e-models-e2e-worker-single-main-89af8e/simulator-single-main-89af8e-kserve-756c5cc577-qph4k"
Oct 30 21:34:27.145998 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:27.145892    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22330f0d-0188-4060-a692-86686cfd517a-tmp-dir\") pod \"simulator-single-main-89af8e-kserve-756c5cc577-qph4k\" (UID: \"22330f0d-0188-4060-a692-86686cfd517a\") " pod="e2e-models-e2e-worker-single-main-89af8e/simulator-single-main-89af8e-kserve-756c5cc577-qph4k"
Oct 30 21:34:27.146052 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:27.146003    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22330f0d-0188-4060-a692-86686cfd517a-model-cache\") pod \"simulator-single-main-89af8e-kserve-756c5cc577-qph4k\" (UID: \"22330f0d-0188-4060-a692-86686cfd517a\") " pod="e2e-models-e2e-worker-single-main-89af8e/simulator-single-main-89af8e-kserve-756c5cc577-qph4k"
Oct 30 21:34:27.247009 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:27.246970    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22330f0d-0188-4060-a692-86686cfd517a-tls-certs\") pod \"simulator-single-main-89af8e-kserve-756c5cc577-qph4k\" (UID: \"22330f0d-0188-4060-a692-86686cfd517a\") " pod="e2e-models-e2e-worker-single-main-89af8e/simulator-single-main-89af8e-kserve-756c5cc577-qph4k"
Oct 30 21:34:27.247009 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:27.247012    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22330f0d-0188-4060-a692-86686cfd517a-dshm\") pod \"simulator-single-main-89af8e-kserve-756c5cc577-qph4k\" (UID: \"22330f0d-0188-4060-a692-86686cfd517a\") " pod="e2e-models-e2e-worker-single-main-89af8e/simulator-single-main-89af8e-kserve-756c5cc577-qph4k"
Oct 30 21:34:27.247275 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:27.247030    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22330f0d-0188-4060-a692-86686cfd517a-tmp-dir\") pod \"simulator-single-main-89af8e-kserve-756c5cc577-qph4k\" (UID: \"22330f0d-0188-4060-a692-86686cfd517a\") " pod="e2e-models-e2e-worker-single-main-89af8e/simulator-single-main-89af8e-kserve-756c5cc577-qph4k"
Oct 30 21:34:27.247275 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:27.247057    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22330f0d-0188-4060-a692-86686cfd517a-model-cache\") pod \"simulator-single-main-89af8e-kserve-756c5cc577-qph4k\" (UID: \"22330f0d-0188-4060-a692-86686cfd517a\") " pod="e2e-models-e2e-worker-single-main-89af8e/simulator-single-main-89af8e-kserve-756c5cc577-qph4k"
Oct 30 21:34:27.247275 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:27.247115    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84mph\" (UniqueName: \"kubernetes.io/projected/22330f0d-0188-4060-a692-86686cfd517a-kube-api-access-84mph\") pod \"simulator-single-main-89af8e-kserve-756c5cc577-qph4k\" (UID: \"22330f0d-0188-4060-a692-86686cfd517a\") " pod="e2e-models-e2e-worker-single-main-89af8e/simulator-single-main-89af8e-kserve-756c5cc577-qph4k"
Oct 30 21:34:27.247275 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:27.247232    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22330f0d-0188-4060-a692-86686cfd517a-home\") pod \"simulator-single-main-89af8e-kserve-756c5cc577-qph4k\" (UID: \"22330f0d-0188-4060-a692-86686cfd517a\") " pod="e2e-models-e2e-worker-single-main-89af8e/simulator-single-main-89af8e-kserve-756c5cc577-qph4k"
Oct 30 21:34:27.247561 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:27.247481    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22330f0d-0188-4060-a692-86686cfd517a-tmp-dir\") pod \"simulator-single-main-89af8e-kserve-756c5cc577-qph4k\" (UID: \"22330f0d-0188-4060-a692-86686cfd517a\") " pod="e2e-models-e2e-worker-single-main-89af8e/simulator-single-main-89af8e-kserve-756c5cc577-qph4k"
Oct 30 21:34:27.247561 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:27.247519    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22330f0d-0188-4060-a692-86686cfd517a-model-cache\") pod \"simulator-single-main-89af8e-kserve-756c5cc577-qph4k\" (UID: \"22330f0d-0188-4060-a692-86686cfd517a\") " pod="e2e-models-e2e-worker-single-main-89af8e/simulator-single-main-89af8e-kserve-756c5cc577-qph4k"
Oct 30 21:34:27.247561 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:27.247556    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22330f0d-0188-4060-a692-86686cfd517a-home\") pod \"simulator-single-main-89af8e-kserve-756c5cc577-qph4k\" (UID: \"22330f0d-0188-4060-a692-86686cfd517a\") " pod="e2e-models-e2e-worker-single-main-89af8e/simulator-single-main-89af8e-kserve-756c5cc577-qph4k"
Oct 30 21:34:27.249596 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:27.249576    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22330f0d-0188-4060-a692-86686cfd517a-dshm\") pod \"simulator-single-main-89af8e-kserve-756c5cc577-qph4k\" (UID: \"22330f0d-0188-4060-a692-86686cfd517a\") " pod="e2e-models-e2e-worker-single-main-89af8e/simulator-single-main-89af8e-kserve-756c5cc577-qph4k"
Oct 30 21:34:27.249909 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:27.249887    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22330f0d-0188-4060-a692-86686cfd517a-tls-certs\") pod \"simulator-single-main-89af8e-kserve-756c5cc577-qph4k\" (UID: \"22330f0d-0188-4060-a692-86686cfd517a\") " pod="e2e-models-e2e-worker-single-main-89af8e/simulator-single-main-89af8e-kserve-756c5cc577-qph4k"
Oct 30 21:34:27.307051 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:27.306977    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84mph\" (UniqueName: \"kubernetes.io/projected/22330f0d-0188-4060-a692-86686cfd517a-kube-api-access-84mph\") pod \"simulator-single-main-89af8e-kserve-756c5cc577-qph4k\" (UID: \"22330f0d-0188-4060-a692-86686cfd517a\") " pod="e2e-models-e2e-worker-single-main-89af8e/simulator-single-main-89af8e-kserve-756c5cc577-qph4k"
Oct 30 21:34:27.325866 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:27.325836    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-89af8e/simulator-single-main-89af8e-kserve-756c5cc577-qph4k"
Oct 30 21:34:27.702749 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:27.702720    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-89af8e/simulator-single-main-89af8e-kserve-756c5cc577-qph4k"]
Oct 30 21:34:27.704369 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:34:27.704338    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod22330f0d_0188_4060_a692_86686cfd517a.slice/crio-69155055e3d47c90f3e821c2e337556e0ee131c0b06cbf860e7d80f52d5f6f8f WatchSource:0}: Error finding container 69155055e3d47c90f3e821c2e337556e0ee131c0b06cbf860e7d80f52d5f6f8f: Status 404 returned error can't find the container with id 69155055e3d47c90f3e821c2e337556e0ee131c0b06cbf860e7d80f52d5f6f8f
Oct 30 21:34:28.015557 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:28.015469    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-89af8e/simulator-single-main-89af8e-kserve-756c5cc577-qph4k" event={"ID":"22330f0d-0188-4060-a692-86686cfd517a","Type":"ContainerStarted","Data":"2136a30cf50518a0e92756b121bbf05a49da39dcc9fa87f784e2635c5b5de450"}
Oct 30 21:34:28.015557 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:28.015524    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-89af8e/simulator-single-main-89af8e-kserve-756c5cc577-qph4k" event={"ID":"22330f0d-0188-4060-a692-86686cfd517a","Type":"ContainerStarted","Data":"69155055e3d47c90f3e821c2e337556e0ee131c0b06cbf860e7d80f52d5f6f8f"}
Oct 30 21:34:28.130055 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:28.129994    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-89af8e/simulator-single-main-89af8e-kserve-756c5cc577-qph4k" podStartSLOduration=2.129980866 podStartE2EDuration="2.129980866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:34:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:34:28.129336002 +0000 UTC m=+3351.194112400" watchObservedRunningTime="2026-10-30 21:34:28.129980866 +0000 UTC m=+3351.194757263"
Oct 30 21:34:28.327182 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:28.327101    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-89af8e/simulator-single-main-89af8e-kserve-756c5cc577-qph4k"
Oct 30 21:34:37.326486 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:37.326449    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-89af8e/simulator-single-main-89af8e-kserve-756c5cc577-qph4k"
Oct 30 21:34:37.338868 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:37.338847    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-89af8e/simulator-single-main-89af8e-kserve-756c5cc577-qph4k"
Oct 30 21:34:38.086139 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:34:38.086108    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-89af8e/simulator-single-main-89af8e-kserve-756c5cc577-qph4k"
Oct 30 21:34:54.458932 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:34:54.458889    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:35:01.045054 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:01.045018    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7vjdw"]
Oct 30 21:35:01.094435 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:01.094407    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7vjdw"]
Oct 30 21:35:03.061749 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:03.061718    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98ee900c-6775-47d5-8b32-cd71c283e14b" path="/var/lib/kubelet/pods/98ee900c-6775-47d5-8b32-cd71c283e14b/volumes"
Oct 30 21:35:12.048797 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:12.048764    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mgz9h"]
Oct 30 21:35:12.098374 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:12.098343    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mgz9h"]
Oct 30 21:35:12.153598 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:12.153568    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-864e0c4b-pch6l"]
Oct 30 21:35:12.156983 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:12.156966    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-864e0c4b-pch6l"
Oct 30 21:35:12.211244 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:12.211178    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfc0a108-5acc-4346-9942-50b15095ed2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-864e0c4b-pch6l\" (UID: \"cfc0a108-5acc-4346-9942-50b15095ed2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-864e0c4b-pch6l"
Oct 30 21:35:12.211468 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:12.211256    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvhhf\" (UniqueName: \"kubernetes.io/projected/cfc0a108-5acc-4346-9942-50b15095ed2a-kube-api-access-qvhhf\") pod \"maas-api-revoke-sub-864e0c4b-pch6l\" (UID: \"cfc0a108-5acc-4346-9942-50b15095ed2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-864e0c4b-pch6l"
Oct 30 21:35:12.211468 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:12.211291    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfc0a108-5acc-4346-9942-50b15095ed2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-864e0c4b-pch6l\" (UID: \"cfc0a108-5acc-4346-9942-50b15095ed2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-864e0c4b-pch6l"
Oct 30 21:35:12.211628 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:12.211564    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:35:12.211882 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:12.211855    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:35:12.212385 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:12.212365    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:35:12.214989 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:12.214964    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-864e0c4b-pch6l"]
Oct 30 21:35:12.260844 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:12.260818    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:35:12.312823 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:12.312730    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvhhf\" (UniqueName: \"kubernetes.io/projected/cfc0a108-5acc-4346-9942-50b15095ed2a-kube-api-access-qvhhf\") pod \"maas-api-revoke-sub-864e0c4b-pch6l\" (UID: \"cfc0a108-5acc-4346-9942-50b15095ed2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-864e0c4b-pch6l"
Oct 30 21:35:12.312823 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:12.312766    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfc0a108-5acc-4346-9942-50b15095ed2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-864e0c4b-pch6l\" (UID: \"cfc0a108-5acc-4346-9942-50b15095ed2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-864e0c4b-pch6l"
Oct 30 21:35:12.313013 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:12.312869    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfc0a108-5acc-4346-9942-50b15095ed2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-864e0c4b-pch6l\" (UID: \"cfc0a108-5acc-4346-9942-50b15095ed2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-864e0c4b-pch6l"
Oct 30 21:35:12.313457 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:12.313432    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfc0a108-5acc-4346-9942-50b15095ed2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-864e0c4b-pch6l\" (UID: \"cfc0a108-5acc-4346-9942-50b15095ed2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-864e0c4b-pch6l"
Oct 30 21:35:12.315337 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:12.315316    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfc0a108-5acc-4346-9942-50b15095ed2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-864e0c4b-pch6l\" (UID: \"cfc0a108-5acc-4346-9942-50b15095ed2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-864e0c4b-pch6l"
Oct 30 21:35:12.369889 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:12.369862    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvhhf\" (UniqueName: \"kubernetes.io/projected/cfc0a108-5acc-4346-9942-50b15095ed2a-kube-api-access-qvhhf\") pod \"maas-api-revoke-sub-864e0c4b-pch6l\" (UID: \"cfc0a108-5acc-4346-9942-50b15095ed2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-864e0c4b-pch6l"
Oct 30 21:35:12.469939 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:12.469904    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-864e0c4b-pch6l"
Oct 30 21:35:12.848688 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:12.848663    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-864e0c4b-pch6l"]
Oct 30 21:35:12.850728 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:35:12.850699    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfc0a108_5acc_4346_9942_50b15095ed2a.slice/crio-eb892517a3b821349ba75e47999f7c728cbebce65d4fd45134295623cb48643a WatchSource:0}: Error finding container eb892517a3b821349ba75e47999f7c728cbebce65d4fd45134295623cb48643a: Status 404 returned error can't find the container with id eb892517a3b821349ba75e47999f7c728cbebce65d4fd45134295623cb48643a
Oct 30 21:35:13.062355 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:13.062327    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f3ed3e1-ad8e-4605-9007-85d4016c19cd" path="/var/lib/kubelet/pods/1f3ed3e1-ad8e-4605-9007-85d4016c19cd/volumes"
Oct 30 21:35:13.287809 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:13.287739    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-864e0c4b-pch6l" event={"ID":"cfc0a108-5acc-4346-9942-50b15095ed2a","Type":"ContainerStarted","Data":"66216310782a7ed6845e0171fa288c42804cc106c2ab9ea5f7d0746638da9e37"}
Oct 30 21:35:13.287809 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:13.287810    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-864e0c4b-pch6l" event={"ID":"cfc0a108-5acc-4346-9942-50b15095ed2a","Type":"ContainerStarted","Data":"eb892517a3b821349ba75e47999f7c728cbebce65d4fd45134295623cb48643a"}
Oct 30 21:35:13.403144 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:13.403041    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-864e0c4b-pch6l" podStartSLOduration=1.403021372 podStartE2EDuration="1.403021372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:13.401555271 +0000 UTC m=+3396.466331668" watchObservedRunningTime="2026-10-30 21:35:13.403021372 +0000 UTC m=+3396.467797770"
Oct 30 21:35:14.294864 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:14.294828    5713 generic.go:356] "Generic (PLEG): container finished" podID="cfc0a108-5acc-4346-9942-50b15095ed2a" containerID="66216310782a7ed6845e0171fa288c42804cc106c2ab9ea5f7d0746638da9e37" exitCode=0
Oct 30 21:35:14.295298 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:14.294922    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-864e0c4b-pch6l" event={"ID":"cfc0a108-5acc-4346-9942-50b15095ed2a","Type":"ContainerDied","Data":"66216310782a7ed6845e0171fa288c42804cc106c2ab9ea5f7d0746638da9e37"}
Oct 30 21:35:15.434410 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:15.434387    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-864e0c4b-pch6l"
Oct 30 21:35:15.544069 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:15.544038    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfc0a108-5acc-4346-9942-50b15095ed2a-kube-api-access-qvhhf\" (UniqueName: \"kubernetes.io/projected/cfc0a108-5acc-4346-9942-50b15095ed2a-kube-api-access-qvhhf\") pod \"cfc0a108-5acc-4346-9942-50b15095ed2a\" (UID: \"cfc0a108-5acc-4346-9942-50b15095ed2a\") "
Oct 30 21:35:15.544240 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:15.544102    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfc0a108-5acc-4346-9942-50b15095ed2a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfc0a108-5acc-4346-9942-50b15095ed2a-maas-api-xks-ca\") pod \"cfc0a108-5acc-4346-9942-50b15095ed2a\" (UID: \"cfc0a108-5acc-4346-9942-50b15095ed2a\") "
Oct 30 21:35:15.544240 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:15.544152    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfc0a108-5acc-4346-9942-50b15095ed2a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfc0a108-5acc-4346-9942-50b15095ed2a-maas-api-openshift-ca\") pod \"cfc0a108-5acc-4346-9942-50b15095ed2a\" (UID: \"cfc0a108-5acc-4346-9942-50b15095ed2a\") "
Oct 30 21:35:15.544562 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:15.544537    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfc0a108-5acc-4346-9942-50b15095ed2a-maas-api-openshift-ca" pod "cfc0a108-5acc-4346-9942-50b15095ed2a" (UID: "cfc0a108-5acc-4346-9942-50b15095ed2a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:35:15.546372 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:15.546348    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfc0a108-5acc-4346-9942-50b15095ed2a-kube-api-access-qvhhf" pod "cfc0a108-5acc-4346-9942-50b15095ed2a" (UID: "cfc0a108-5acc-4346-9942-50b15095ed2a"). InnerVolumeSpecName "kube-api-access-qvhhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:35:15.546449 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:15.546412    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfc0a108-5acc-4346-9942-50b15095ed2a-maas-api-xks-ca" pod "cfc0a108-5acc-4346-9942-50b15095ed2a" (UID: "cfc0a108-5acc-4346-9942-50b15095ed2a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:35:15.645295 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:15.645249    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvhhf\" (UniqueName: \"kubernetes.io/projected/cfc0a108-5acc-4346-9942-50b15095ed2a-kube-api-access-qvhhf\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:35:15.645295 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:15.645291    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfc0a108-5acc-4346-9942-50b15095ed2a-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:35:15.645295 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:15.645301    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfc0a108-5acc-4346-9942-50b15095ed2a-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:35:16.306978 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:16.306947    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-864e0c4b-pch6l"
Oct 30 21:35:16.307164 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:16.306942    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-864e0c4b-pch6l" event={"ID":"cfc0a108-5acc-4346-9942-50b15095ed2a","Type":"ContainerDied","Data":"eb892517a3b821349ba75e47999f7c728cbebce65d4fd45134295623cb48643a"}
Oct 30 21:35:16.307164 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:16.307074    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb892517a3b821349ba75e47999f7c728cbebce65d4fd45134295623cb48643a"
Oct 30 21:35:24.366331 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:24.366296    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-89af8e-7vhj7"]
Oct 30 21:35:24.366728 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:24.366698    5713 cpu_manager.go:414] "containerMap: removing container" podUID="cfc0a108-5acc-4346-9942-50b15095ed2a" containerName="revoke-keys"
Oct 30 21:35:24.366728 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:24.366709    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfc0a108-5acc-4346-9942-50b15095ed2a" containerName="revoke-keys"
Oct 30 21:35:24.366848 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:24.366823    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfc0a108-5acc-4346-9942-50b15095ed2a" containerName="revoke-keys"
Oct 30 21:35:24.370057 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:24.370040    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-89af8e-7vhj7"
Oct 30 21:35:24.424645 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:24.424619    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:35:24.437909 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:24.437876    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2265ea30-96a0-4fb4-949f-152a5efce34a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-89af8e-7vhj7\" (UID: \"2265ea30-96a0-4fb4-949f-152a5efce34a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-89af8e-7vhj7"
Oct 30 21:35:24.438082 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:24.438059    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t468b\" (UniqueName: \"kubernetes.io/projected/2265ea30-96a0-4fb4-949f-152a5efce34a-kube-api-access-t468b\") pod \"maas-api-revoke-keys-e2e-worker-single-main-89af8e-7vhj7\" (UID: \"2265ea30-96a0-4fb4-949f-152a5efce34a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-89af8e-7vhj7"
Oct 30 21:35:24.438136 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:24.438101    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2265ea30-96a0-4fb4-949f-152a5efce34a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-89af8e-7vhj7\" (UID: \"2265ea30-96a0-4fb4-949f-152a5efce34a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-89af8e-7vhj7"
Oct 30 21:35:24.458944 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:35:24.458907    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:35:24.473980 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:24.473958    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:35:24.474076 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:24.473998    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:35:24.474076 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:24.473998    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:35:24.476387 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:24.476366    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-89af8e-7vhj7"]
Oct 30 21:35:24.539303 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:24.539275    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t468b\" (UniqueName: \"kubernetes.io/projected/2265ea30-96a0-4fb4-949f-152a5efce34a-kube-api-access-t468b\") pod \"maas-api-revoke-keys-e2e-worker-single-main-89af8e-7vhj7\" (UID: \"2265ea30-96a0-4fb4-949f-152a5efce34a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-89af8e-7vhj7"
Oct 30 21:35:24.539452 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:24.539312    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2265ea30-96a0-4fb4-949f-152a5efce34a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-89af8e-7vhj7\" (UID: \"2265ea30-96a0-4fb4-949f-152a5efce34a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-89af8e-7vhj7"
Oct 30 21:35:24.539452 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:24.539342    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2265ea30-96a0-4fb4-949f-152a5efce34a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-89af8e-7vhj7\" (UID: \"2265ea30-96a0-4fb4-949f-152a5efce34a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-89af8e-7vhj7"
Oct 30 21:35:24.539944 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:24.539925    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2265ea30-96a0-4fb4-949f-152a5efce34a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-89af8e-7vhj7\" (UID: \"2265ea30-96a0-4fb4-949f-152a5efce34a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-89af8e-7vhj7"
Oct 30 21:35:24.541759 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:24.541742    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2265ea30-96a0-4fb4-949f-152a5efce34a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-89af8e-7vhj7\" (UID: \"2265ea30-96a0-4fb4-949f-152a5efce34a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-89af8e-7vhj7"
Oct 30 21:35:24.596052 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:24.596024    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t468b\" (UniqueName: \"kubernetes.io/projected/2265ea30-96a0-4fb4-949f-152a5efce34a-kube-api-access-t468b\") pod \"maas-api-revoke-keys-e2e-worker-single-main-89af8e-7vhj7\" (UID: \"2265ea30-96a0-4fb4-949f-152a5efce34a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-89af8e-7vhj7"
Oct 30 21:35:24.682927 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:24.682902    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-89af8e-7vhj7"
Oct 30 21:35:25.053519 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:25.053497    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-89af8e-7vhj7"]
Oct 30 21:35:25.055535 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:35:25.055500    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2265ea30_96a0_4fb4_949f_152a5efce34a.slice/crio-39c43b619b7cc6cd9b161a7bc64de4237115428336be397fcd2229a04e0de1d7 WatchSource:0}: Error finding container 39c43b619b7cc6cd9b161a7bc64de4237115428336be397fcd2229a04e0de1d7: Status 404 returned error can't find the container with id 39c43b619b7cc6cd9b161a7bc64de4237115428336be397fcd2229a04e0de1d7
Oct 30 21:35:25.360808 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:25.360758    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-89af8e-7vhj7" event={"ID":"2265ea30-96a0-4fb4-949f-152a5efce34a","Type":"ContainerStarted","Data":"a02b127601dd231f1d1449f48b6a7786ed1edc7f48d386f9529a3e62676dfd70"}
Oct 30 21:35:25.360948 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:25.360828    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-89af8e-7vhj7" event={"ID":"2265ea30-96a0-4fb4-949f-152a5efce34a","Type":"ContainerStarted","Data":"39c43b619b7cc6cd9b161a7bc64de4237115428336be397fcd2229a04e0de1d7"}
Oct 30 21:35:25.475560 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:25.475509    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-89af8e-7vhj7" podStartSLOduration=1.475495102 podStartE2EDuration="1.475495102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:25.473171534 +0000 UTC m=+3408.537947933" watchObservedRunningTime="2026-10-30 21:35:25.475495102 +0000 UTC m=+3408.540271527"
Oct 30 21:35:26.369529 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:26.369497    5713 generic.go:356] "Generic (PLEG): container finished" podID="2265ea30-96a0-4fb4-949f-152a5efce34a" containerID="a02b127601dd231f1d1449f48b6a7786ed1edc7f48d386f9529a3e62676dfd70" exitCode=0
Oct 30 21:35:26.369717 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:26.369589    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-89af8e-7vhj7" event={"ID":"2265ea30-96a0-4fb4-949f-152a5efce34a","Type":"ContainerDied","Data":"a02b127601dd231f1d1449f48b6a7786ed1edc7f48d386f9529a3e62676dfd70"}
Oct 30 21:35:27.508190 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:27.508164    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-89af8e-7vhj7"
Oct 30 21:35:27.573676 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:27.573642    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2265ea30-96a0-4fb4-949f-152a5efce34a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2265ea30-96a0-4fb4-949f-152a5efce34a-maas-api-openshift-ca\") pod \"2265ea30-96a0-4fb4-949f-152a5efce34a\" (UID: \"2265ea30-96a0-4fb4-949f-152a5efce34a\") "
Oct 30 21:35:27.573859 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:27.573691    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2265ea30-96a0-4fb4-949f-152a5efce34a-kube-api-access-t468b\" (UniqueName: \"kubernetes.io/projected/2265ea30-96a0-4fb4-949f-152a5efce34a-kube-api-access-t468b\") pod \"2265ea30-96a0-4fb4-949f-152a5efce34a\" (UID: \"2265ea30-96a0-4fb4-949f-152a5efce34a\") "
Oct 30 21:35:27.573859 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:27.573742    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2265ea30-96a0-4fb4-949f-152a5efce34a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2265ea30-96a0-4fb4-949f-152a5efce34a-maas-api-xks-ca\") pod \"2265ea30-96a0-4fb4-949f-152a5efce34a\" (UID: \"2265ea30-96a0-4fb4-949f-152a5efce34a\") "
Oct 30 21:35:27.574050 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:27.574025    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2265ea30-96a0-4fb4-949f-152a5efce34a-maas-api-openshift-ca" pod "2265ea30-96a0-4fb4-949f-152a5efce34a" (UID: "2265ea30-96a0-4fb4-949f-152a5efce34a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:35:27.575920 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:27.575893    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2265ea30-96a0-4fb4-949f-152a5efce34a-kube-api-access-t468b" pod "2265ea30-96a0-4fb4-949f-152a5efce34a" (UID: "2265ea30-96a0-4fb4-949f-152a5efce34a"). InnerVolumeSpecName "kube-api-access-t468b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:35:27.576029 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:27.575946    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2265ea30-96a0-4fb4-949f-152a5efce34a-maas-api-xks-ca" pod "2265ea30-96a0-4fb4-949f-152a5efce34a" (UID: "2265ea30-96a0-4fb4-949f-152a5efce34a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:35:27.675103 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:27.675010    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2265ea30-96a0-4fb4-949f-152a5efce34a-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:35:27.675103 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:27.675044    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t468b\" (UniqueName: \"kubernetes.io/projected/2265ea30-96a0-4fb4-949f-152a5efce34a-kube-api-access-t468b\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:35:27.675103 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:27.675055    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2265ea30-96a0-4fb4-949f-152a5efce34a-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:35:28.383458 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:28.383420    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-89af8e-7vhj7" event={"ID":"2265ea30-96a0-4fb4-949f-152a5efce34a","Type":"ContainerDied","Data":"39c43b619b7cc6cd9b161a7bc64de4237115428336be397fcd2229a04e0de1d7"}
Oct 30 21:35:28.383661 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:28.383471    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39c43b619b7cc6cd9b161a7bc64de4237115428336be397fcd2229a04e0de1d7"
Oct 30 21:35:28.383661 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:28.383434    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-89af8e-7vhj7"
Oct 30 21:35:28.774272 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:28.774233    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-89af8e/simulator-single-main-89af8e-kserve-756c5cc577-qph4k"]
Oct 30 21:35:35.886931 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:35.886893    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-89af8e-7bd9b978bfbfhx"]
Oct 30 21:35:36.024921 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:36.024880    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-89af8e-54695b5qtr"]
Oct 30 21:35:40.300716 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:40.300680    5713 scope.go:122] "RemoveContainer" containerID="44ef0706b52c385bd6a3376aa98fd12ea67c08f5190cefbf7187552f29f4f9a4"
Oct 30 21:35:40.310369 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:40.310343    5713 scope.go:122] "RemoveContainer" containerID="8b12f5269b8ee2a9d14eae71cdc8bdf77f492054181adb2da9d551929bd7df74"
Oct 30 21:35:40.887489 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:40.887397    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-89af8e-7bd9b978bfbfhx" podUID="cf9fde33-48d1-462c-be89-ef10584fd6a5" containerName="payload-processing" containerID="cri-o://8d20f8671667fe024a685a16f20cff23712bdfa781c9b843aef67638c22c211c" gracePeriod=25
Oct 30 21:35:41.025780 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:41.025735    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-89af8e-54695b5qtr" podUID="1ea5a200-89c8-4cd6-b319-a438e0c9e553" containerName="payload-pre-processing" containerID="cri-o://8a15ae4d8e29507d27b86543a7c3ad1ec05d19c96e1fea4d040eb46aa404e24a" gracePeriod=25
Oct 30 21:35:43.794804 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:43.794757    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-89af8e/simulator-single-main-89af8e-kserve-756c5cc577-qph4k" podUID="22330f0d-0188-4060-a692-86686cfd517a" containerName="main" containerID="cri-o://2136a30cf50518a0e92756b121bbf05a49da39dcc9fa87f784e2635c5b5de450" gracePeriod=45
Oct 30 21:35:44.050612 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:44.050553    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-89af8e/simulator-single-main-89af8e-kserve-756c5cc577-qph4k"
Oct 30 21:35:44.147649 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:44.147620    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22330f0d-0188-4060-a692-86686cfd517a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/22330f0d-0188-4060-a692-86686cfd517a-dshm\") pod \"22330f0d-0188-4060-a692-86686cfd517a\" (UID: \"22330f0d-0188-4060-a692-86686cfd517a\") "
Oct 30 21:35:44.147862 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:44.147748    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22330f0d-0188-4060-a692-86686cfd517a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22330f0d-0188-4060-a692-86686cfd517a-tmp-dir\") pod \"22330f0d-0188-4060-a692-86686cfd517a\" (UID: \"22330f0d-0188-4060-a692-86686cfd517a\") "
Oct 30 21:35:44.147912 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:44.147877    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22330f0d-0188-4060-a692-86686cfd517a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22330f0d-0188-4060-a692-86686cfd517a-model-cache\") pod \"22330f0d-0188-4060-a692-86686cfd517a\" (UID: \"22330f0d-0188-4060-a692-86686cfd517a\") "
Oct 30 21:35:44.147950 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:44.147913    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22330f0d-0188-4060-a692-86686cfd517a-tls-certs\" (UniqueName: \"kubernetes.io/secret/22330f0d-0188-4060-a692-86686cfd517a-tls-certs\") pod \"22330f0d-0188-4060-a692-86686cfd517a\" (UID: \"22330f0d-0188-4060-a692-86686cfd517a\") "
Oct 30 21:35:44.148003 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:44.147950    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22330f0d-0188-4060-a692-86686cfd517a-kube-api-access-84mph\" (UniqueName: \"kubernetes.io/projected/22330f0d-0188-4060-a692-86686cfd517a-kube-api-access-84mph\") pod \"22330f0d-0188-4060-a692-86686cfd517a\" (UID: \"22330f0d-0188-4060-a692-86686cfd517a\") "
Oct 30 21:35:44.148003 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:44.147993    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22330f0d-0188-4060-a692-86686cfd517a-home\" (UniqueName: \"kubernetes.io/empty-dir/22330f0d-0188-4060-a692-86686cfd517a-home\") pod \"22330f0d-0188-4060-a692-86686cfd517a\" (UID: \"22330f0d-0188-4060-a692-86686cfd517a\") "
Oct 30 21:35:44.148119 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:44.148046    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22330f0d-0188-4060-a692-86686cfd517a-tmp-dir" pod "22330f0d-0188-4060-a692-86686cfd517a" (UID: "22330f0d-0188-4060-a692-86686cfd517a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:35:44.148186 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:44.148159    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22330f0d-0188-4060-a692-86686cfd517a-model-cache" pod "22330f0d-0188-4060-a692-86686cfd517a" (UID: "22330f0d-0188-4060-a692-86686cfd517a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:35:44.148330 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:44.148305    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22330f0d-0188-4060-a692-86686cfd517a-home" pod "22330f0d-0188-4060-a692-86686cfd517a" (UID: "22330f0d-0188-4060-a692-86686cfd517a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:35:44.148429 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:44.148412    5713 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22330f0d-0188-4060-a692-86686cfd517a-model-cache\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:35:44.148429 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:44.148425    5713 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22330f0d-0188-4060-a692-86686cfd517a-home\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:35:44.148505 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:44.148437    5713 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22330f0d-0188-4060-a692-86686cfd517a-tmp-dir\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:35:44.150272 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:44.150238    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22330f0d-0188-4060-a692-86686cfd517a-tls-certs" pod "22330f0d-0188-4060-a692-86686cfd517a" (UID: "22330f0d-0188-4060-a692-86686cfd517a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:35:44.150272 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:44.150243    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22330f0d-0188-4060-a692-86686cfd517a-dshm" pod "22330f0d-0188-4060-a692-86686cfd517a" (UID: "22330f0d-0188-4060-a692-86686cfd517a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:35:44.150418 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:44.150322    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22330f0d-0188-4060-a692-86686cfd517a-kube-api-access-84mph" pod "22330f0d-0188-4060-a692-86686cfd517a" (UID: "22330f0d-0188-4060-a692-86686cfd517a"). InnerVolumeSpecName "kube-api-access-84mph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:35:44.249595 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:44.249564    5713 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22330f0d-0188-4060-a692-86686cfd517a-tls-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:35:44.249595 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:44.249593    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84mph\" (UniqueName: \"kubernetes.io/projected/22330f0d-0188-4060-a692-86686cfd517a-kube-api-access-84mph\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:35:44.249595 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:44.249604    5713 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22330f0d-0188-4060-a692-86686cfd517a-dshm\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:35:44.268034 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:44.268000    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-89af8e-7vhj7"]
Oct 30 21:35:44.270511 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:44.270474    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-89af8e-7vhj7"]
Oct 30 21:35:44.476641 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:44.476602    5713 generic.go:356] "Generic (PLEG): container finished" podID="22330f0d-0188-4060-a692-86686cfd517a" containerID="2136a30cf50518a0e92756b121bbf05a49da39dcc9fa87f784e2635c5b5de450" exitCode=0
Oct 30 21:35:44.476878 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:44.476651    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-89af8e/simulator-single-main-89af8e-kserve-756c5cc577-qph4k"
Oct 30 21:35:44.476878 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:44.476692    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-89af8e/simulator-single-main-89af8e-kserve-756c5cc577-qph4k" event={"ID":"22330f0d-0188-4060-a692-86686cfd517a","Type":"ContainerDied","Data":"2136a30cf50518a0e92756b121bbf05a49da39dcc9fa87f784e2635c5b5de450"}
Oct 30 21:35:44.476878 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:44.476738    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-89af8e/simulator-single-main-89af8e-kserve-756c5cc577-qph4k" event={"ID":"22330f0d-0188-4060-a692-86686cfd517a","Type":"ContainerDied","Data":"69155055e3d47c90f3e821c2e337556e0ee131c0b06cbf860e7d80f52d5f6f8f"}
Oct 30 21:35:44.476878 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:44.476762    5713 scope.go:122] "RemoveContainer" containerID="2136a30cf50518a0e92756b121bbf05a49da39dcc9fa87f784e2635c5b5de450"
Oct 30 21:35:44.487847 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:44.487827    5713 scope.go:122] "RemoveContainer" containerID="2136a30cf50518a0e92756b121bbf05a49da39dcc9fa87f784e2635c5b5de450"
Oct 30 21:35:44.488112 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:35:44.488085    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2136a30cf50518a0e92756b121bbf05a49da39dcc9fa87f784e2635c5b5de450\": container with ID starting with 2136a30cf50518a0e92756b121bbf05a49da39dcc9fa87f784e2635c5b5de450 not found: ID does not exist" containerID="2136a30cf50518a0e92756b121bbf05a49da39dcc9fa87f784e2635c5b5de450"
Oct 30 21:35:44.488183 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:44.488125    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2136a30cf50518a0e92756b121bbf05a49da39dcc9fa87f784e2635c5b5de450"} err="failed to get container status \"2136a30cf50518a0e92756b121bbf05a49da39dcc9fa87f784e2635c5b5de450\": rpc error: code = NotFound desc = could not find container \"2136a30cf50518a0e92756b121bbf05a49da39dcc9fa87f784e2635c5b5de450\": container with ID starting with 2136a30cf50518a0e92756b121bbf05a49da39dcc9fa87f784e2635c5b5de450 not found: ID does not exist"
Oct 30 21:35:44.640972 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:44.640937    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-89af8e/simulator-single-main-89af8e-kserve-756c5cc577-qph4k"]
Oct 30 21:35:44.645906 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:44.645881    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-89af8e/simulator-single-main-89af8e-kserve-756c5cc577-qph4k"]
Oct 30 21:35:44.884192 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:44.884101    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-89af8e-7bd9b978bfbfhx" podUID="cf9fde33-48d1-462c-be89-ef10584fd6a5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:35:44.884192 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:44.884116    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-89af8e-54695b5qtr" podUID="1ea5a200-89c8-4cd6-b319-a438e0c9e553" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:35:45.061501 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:45.061466    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22330f0d-0188-4060-a692-86686cfd517a" path="/var/lib/kubelet/pods/22330f0d-0188-4060-a692-86686cfd517a/volumes"
Oct 30 21:35:45.061932 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:45.061916    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2265ea30-96a0-4fb4-949f-152a5efce34a" path="/var/lib/kubelet/pods/2265ea30-96a0-4fb4-949f-152a5efce34a/volumes"
Oct 30 21:35:46.049030 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:46.048985    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5625c"]
Oct 30 21:35:46.098875 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:46.098840    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5625c"]
Oct 30 21:35:47.062197 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:47.062117    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63ca2987-423e-43ed-90a4-f1cdb5b058c1" path="/var/lib/kubelet/pods/63ca2987-423e-43ed-90a4-f1cdb5b058c1/volumes"
Oct 30 21:35:52.365972 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:52.365932    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-slbn6"]
Oct 30 21:35:52.415608 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:52.415571    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-slbn6"]
Oct 30 21:35:52.490446 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:52.490411    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6q6fz"]
Oct 30 21:35:52.490912 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:52.490898    5713 cpu_manager.go:414] "containerMap: removing container" podUID="2265ea30-96a0-4fb4-949f-152a5efce34a" containerName="revoke-keys"
Oct 30 21:35:52.490981 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:52.490913    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2265ea30-96a0-4fb4-949f-152a5efce34a" containerName="revoke-keys"
Oct 30 21:35:52.490981 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:52.490944    5713 cpu_manager.go:414] "containerMap: removing container" podUID="22330f0d-0188-4060-a692-86686cfd517a" containerName="main"
Oct 30 21:35:52.490981 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:52.490950    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22330f0d-0188-4060-a692-86686cfd517a" containerName="main"
Oct 30 21:35:52.491097 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:52.491008    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="22330f0d-0188-4060-a692-86686cfd517a" containerName="main"
Oct 30 21:35:52.491097 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:52.491019    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="2265ea30-96a0-4fb4-949f-152a5efce34a" containerName="revoke-keys"
Oct 30 21:35:52.495709 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:52.495693    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6q6fz"
Oct 30 21:35:52.549776 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:52.549743    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:35:52.550009 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:52.549865    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:35:52.550009 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:52.549750    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:35:52.550142 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:52.550042    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:35:52.552859 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:52.552827    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6q6fz"]
Oct 30 21:35:52.636008 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:52.635925    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dptbk\" (UniqueName: \"kubernetes.io/projected/a6b9cfa2-a0ad-415f-903e-9f96b2d00bb1-kube-api-access-dptbk\") pod \"maas-api-revoke-sub-3d4156f1-6q6fz\" (UID: \"a6b9cfa2-a0ad-415f-903e-9f96b2d00bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6q6fz"
Oct 30 21:35:52.636008 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:52.635988    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6b9cfa2-a0ad-415f-903e-9f96b2d00bb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-6q6fz\" (UID: \"a6b9cfa2-a0ad-415f-903e-9f96b2d00bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6q6fz"
Oct 30 21:35:52.636232 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:52.636129    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6b9cfa2-a0ad-415f-903e-9f96b2d00bb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-6q6fz\" (UID: \"a6b9cfa2-a0ad-415f-903e-9f96b2d00bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6q6fz"
Oct 30 21:35:52.737628 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:52.737587    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6b9cfa2-a0ad-415f-903e-9f96b2d00bb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-6q6fz\" (UID: \"a6b9cfa2-a0ad-415f-903e-9f96b2d00bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6q6fz"
Oct 30 21:35:52.737816 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:52.737676    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dptbk\" (UniqueName: \"kubernetes.io/projected/a6b9cfa2-a0ad-415f-903e-9f96b2d00bb1-kube-api-access-dptbk\") pod \"maas-api-revoke-sub-3d4156f1-6q6fz\" (UID: \"a6b9cfa2-a0ad-415f-903e-9f96b2d00bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6q6fz"
Oct 30 21:35:52.737816 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:52.737748    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6b9cfa2-a0ad-415f-903e-9f96b2d00bb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-6q6fz\" (UID: \"a6b9cfa2-a0ad-415f-903e-9f96b2d00bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6q6fz"
Oct 30 21:35:52.738336 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:52.738311    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6b9cfa2-a0ad-415f-903e-9f96b2d00bb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-6q6fz\" (UID: \"a6b9cfa2-a0ad-415f-903e-9f96b2d00bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6q6fz"
Oct 30 21:35:52.740346 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:52.740324    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6b9cfa2-a0ad-415f-903e-9f96b2d00bb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-6q6fz\" (UID: \"a6b9cfa2-a0ad-415f-903e-9f96b2d00bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6q6fz"
Oct 30 21:35:52.795317 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:52.795267    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dptbk\" (UniqueName: \"kubernetes.io/projected/a6b9cfa2-a0ad-415f-903e-9f96b2d00bb1-kube-api-access-dptbk\") pod \"maas-api-revoke-sub-3d4156f1-6q6fz\" (UID: \"a6b9cfa2-a0ad-415f-903e-9f96b2d00bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6q6fz"
Oct 30 21:35:52.808049 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:52.808013    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6q6fz"
Oct 30 21:35:53.062330 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:53.062297    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dae11545-f309-491f-a41f-5154396960b1" path="/var/lib/kubelet/pods/dae11545-f309-491f-a41f-5154396960b1/volumes"
Oct 30 21:35:53.182817 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:53.182775    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6q6fz"]
Oct 30 21:35:53.185584 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:35:53.185557    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6b9cfa2_a0ad_415f_903e_9f96b2d00bb1.slice/crio-d579363015e0cdf9096f5e0b024cb4b3fc207fe1f53a371eab1500df4d45cffa WatchSource:0}: Error finding container d579363015e0cdf9096f5e0b024cb4b3fc207fe1f53a371eab1500df4d45cffa: Status 404 returned error can't find the container with id d579363015e0cdf9096f5e0b024cb4b3fc207fe1f53a371eab1500df4d45cffa
Oct 30 21:35:53.531767 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:53.531731    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6q6fz" event={"ID":"a6b9cfa2-a0ad-415f-903e-9f96b2d00bb1","Type":"ContainerStarted","Data":"ec2c7a76297de4069d750ed88598b9c1caf6b2875a7f7a4f06dfb67b2e1d0c3f"}
Oct 30 21:35:53.532213 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:53.531782    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6q6fz" event={"ID":"a6b9cfa2-a0ad-415f-903e-9f96b2d00bb1","Type":"ContainerStarted","Data":"d579363015e0cdf9096f5e0b024cb4b3fc207fe1f53a371eab1500df4d45cffa"}
Oct 30 21:35:53.649173 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:53.649069    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6q6fz" podStartSLOduration=1.649056793 podStartE2EDuration="1.649056793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:53.646766371 +0000 UTC m=+3436.711542769" watchObservedRunningTime="2026-10-30 21:35:53.649056793 +0000 UTC m=+3436.713833191"
Oct 30 21:35:54.458408 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:35:54.458365    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:35:54.539832 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:54.539773    5713 generic.go:356] "Generic (PLEG): container finished" podID="a6b9cfa2-a0ad-415f-903e-9f96b2d00bb1" containerID="ec2c7a76297de4069d750ed88598b9c1caf6b2875a7f7a4f06dfb67b2e1d0c3f" exitCode=0
Oct 30 21:35:54.539832 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:54.539822    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6q6fz" event={"ID":"a6b9cfa2-a0ad-415f-903e-9f96b2d00bb1","Type":"ContainerDied","Data":"ec2c7a76297de4069d750ed88598b9c1caf6b2875a7f7a4f06dfb67b2e1d0c3f"}
Oct 30 21:35:54.883913 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:54.883829    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-89af8e-7bd9b978bfbfhx" podUID="cf9fde33-48d1-462c-be89-ef10584fd6a5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:35:54.884070 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:54.883831    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-89af8e-54695b5qtr" podUID="1ea5a200-89c8-4cd6-b319-a438e0c9e553" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:35:55.672021 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:55.671995    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6q6fz"
Oct 30 21:35:55.768360 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:55.768332    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6b9cfa2-a0ad-415f-903e-9f96b2d00bb1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6b9cfa2-a0ad-415f-903e-9f96b2d00bb1-maas-api-openshift-ca\") pod \"a6b9cfa2-a0ad-415f-903e-9f96b2d00bb1\" (UID: \"a6b9cfa2-a0ad-415f-903e-9f96b2d00bb1\") "
Oct 30 21:35:55.768523 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:55.768449    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6b9cfa2-a0ad-415f-903e-9f96b2d00bb1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6b9cfa2-a0ad-415f-903e-9f96b2d00bb1-maas-api-xks-ca\") pod \"a6b9cfa2-a0ad-415f-903e-9f96b2d00bb1\" (UID: \"a6b9cfa2-a0ad-415f-903e-9f96b2d00bb1\") "
Oct 30 21:35:55.768523 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:55.768475    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6b9cfa2-a0ad-415f-903e-9f96b2d00bb1-kube-api-access-dptbk\" (UniqueName: \"kubernetes.io/projected/a6b9cfa2-a0ad-415f-903e-9f96b2d00bb1-kube-api-access-dptbk\") pod \"a6b9cfa2-a0ad-415f-903e-9f96b2d00bb1\" (UID: \"a6b9cfa2-a0ad-415f-903e-9f96b2d00bb1\") "
Oct 30 21:35:55.768707 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:55.768685    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6b9cfa2-a0ad-415f-903e-9f96b2d00bb1-maas-api-openshift-ca" pod "a6b9cfa2-a0ad-415f-903e-9f96b2d00bb1" (UID: "a6b9cfa2-a0ad-415f-903e-9f96b2d00bb1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:35:55.768834 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:55.768819    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6b9cfa2-a0ad-415f-903e-9f96b2d00bb1-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:35:55.770725 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:55.770699    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6b9cfa2-a0ad-415f-903e-9f96b2d00bb1-maas-api-xks-ca" pod "a6b9cfa2-a0ad-415f-903e-9f96b2d00bb1" (UID: "a6b9cfa2-a0ad-415f-903e-9f96b2d00bb1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:35:55.770815 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:55.770773    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6b9cfa2-a0ad-415f-903e-9f96b2d00bb1-kube-api-access-dptbk" pod "a6b9cfa2-a0ad-415f-903e-9f96b2d00bb1" (UID: "a6b9cfa2-a0ad-415f-903e-9f96b2d00bb1"). InnerVolumeSpecName "kube-api-access-dptbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:35:55.870015 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:55.869931    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6b9cfa2-a0ad-415f-903e-9f96b2d00bb1-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:35:55.870015 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:55.869962    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dptbk\" (UniqueName: \"kubernetes.io/projected/a6b9cfa2-a0ad-415f-903e-9f96b2d00bb1-kube-api-access-dptbk\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:35:56.551687 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:56.551646    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6q6fz" event={"ID":"a6b9cfa2-a0ad-415f-903e-9f96b2d00bb1","Type":"ContainerDied","Data":"d579363015e0cdf9096f5e0b024cb4b3fc207fe1f53a371eab1500df4d45cffa"}
Oct 30 21:35:56.551687 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:56.551671    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6q6fz"
Oct 30 21:35:56.552032 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:35:56.551704    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d579363015e0cdf9096f5e0b024cb4b3fc207fe1f53a371eab1500df4d45cffa"
Oct 30 21:36:01.151520 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:36:01.151464    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-89af8e-7bd9b978bfbfhx"
Oct 30 21:36:01.223119 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:36:01.223085    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf9fde33-48d1-462c-be89-ef10584fd6a5-kube-api-access-vrsfd\" (UniqueName: \"kubernetes.io/projected/cf9fde33-48d1-462c-be89-ef10584fd6a5-kube-api-access-vrsfd\") pod \"cf9fde33-48d1-462c-be89-ef10584fd6a5\" (UID: \"cf9fde33-48d1-462c-be89-ef10584fd6a5\") "
Oct 30 21:36:01.223355 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:36:01.223227    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf9fde33-48d1-462c-be89-ef10584fd6a5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf9fde33-48d1-462c-be89-ef10584fd6a5-plugins-config-volume\") pod \"cf9fde33-48d1-462c-be89-ef10584fd6a5\" (UID: \"cf9fde33-48d1-462c-be89-ef10584fd6a5\") "
Oct 30 21:36:01.223635 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:36:01.223612    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf9fde33-48d1-462c-be89-ef10584fd6a5-plugins-config-volume" pod "cf9fde33-48d1-462c-be89-ef10584fd6a5" (UID: "cf9fde33-48d1-462c-be89-ef10584fd6a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:36:01.225339 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:36:01.225312    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf9fde33-48d1-462c-be89-ef10584fd6a5-kube-api-access-vrsfd" pod "cf9fde33-48d1-462c-be89-ef10584fd6a5" (UID: "cf9fde33-48d1-462c-be89-ef10584fd6a5"). InnerVolumeSpecName "kube-api-access-vrsfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:36:01.261420 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:36:01.261397    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-89af8e-54695b5qtr"
Oct 30 21:36:01.325106 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:36:01.325072    5713 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf9fde33-48d1-462c-be89-ef10584fd6a5-plugins-config-volume\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:36:01.325106 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:36:01.325102    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrsfd\" (UniqueName: \"kubernetes.io/projected/cf9fde33-48d1-462c-be89-ef10584fd6a5-kube-api-access-vrsfd\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:36:01.426154 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:36:01.426070    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ea5a200-89c8-4cd6-b319-a438e0c9e553-kube-api-access-ch46l\" (UniqueName: \"kubernetes.io/projected/1ea5a200-89c8-4cd6-b319-a438e0c9e553-kube-api-access-ch46l\") pod \"1ea5a200-89c8-4cd6-b319-a438e0c9e553\" (UID: \"1ea5a200-89c8-4cd6-b319-a438e0c9e553\") "
Oct 30 21:36:01.426308 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:36:01.426178    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ea5a200-89c8-4cd6-b319-a438e0c9e553-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ea5a200-89c8-4cd6-b319-a438e0c9e553-plugins-config-volume\") pod \"1ea5a200-89c8-4cd6-b319-a438e0c9e553\" (UID: \"1ea5a200-89c8-4cd6-b319-a438e0c9e553\") "
Oct 30 21:36:01.426545 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:36:01.426517    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ea5a200-89c8-4cd6-b319-a438e0c9e553-plugins-config-volume" pod "1ea5a200-89c8-4cd6-b319-a438e0c9e553" (UID: "1ea5a200-89c8-4cd6-b319-a438e0c9e553"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:36:01.428368 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:36:01.428347    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ea5a200-89c8-4cd6-b319-a438e0c9e553-kube-api-access-ch46l" pod "1ea5a200-89c8-4cd6-b319-a438e0c9e553" (UID: "1ea5a200-89c8-4cd6-b319-a438e0c9e553"). InnerVolumeSpecName "kube-api-access-ch46l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:36:01.527445 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:36:01.527409    5713 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ea5a200-89c8-4cd6-b319-a438e0c9e553-plugins-config-volume\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:36:01.527445 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:36:01.527439    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch46l\" (UniqueName: \"kubernetes.io/projected/1ea5a200-89c8-4cd6-b319-a438e0c9e553-kube-api-access-ch46l\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:36:01.581704 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:36:01.581667    5713 generic.go:356] "Generic (PLEG): container finished" podID="cf9fde33-48d1-462c-be89-ef10584fd6a5" containerID="8d20f8671667fe024a685a16f20cff23712bdfa781c9b843aef67638c22c211c" exitCode=0
Oct 30 21:36:01.581916 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:36:01.581722    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-89af8e-7bd9b978bfbfhx"
Oct 30 21:36:01.581916 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:36:01.581745    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-89af8e-7bd9b978bfbfhx" event={"ID":"cf9fde33-48d1-462c-be89-ef10584fd6a5","Type":"ContainerDied","Data":"8d20f8671667fe024a685a16f20cff23712bdfa781c9b843aef67638c22c211c"}
Oct 30 21:36:01.581916 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:36:01.581848    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-89af8e-7bd9b978bfbfhx" event={"ID":"cf9fde33-48d1-462c-be89-ef10584fd6a5","Type":"ContainerDied","Data":"f2539bf5219bd81072ccd175c3ef7f26ab0171a1660bb553f87a40f678448775"}
Oct 30 21:36:01.581916 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:36:01.581881    5713 scope.go:122] "RemoveContainer" containerID="8d20f8671667fe024a685a16f20cff23712bdfa781c9b843aef67638c22c211c"
Oct 30 21:36:01.583143 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:36:01.583118    5713 generic.go:356] "Generic (PLEG): container finished" podID="1ea5a200-89c8-4cd6-b319-a438e0c9e553" containerID="8a15ae4d8e29507d27b86543a7c3ad1ec05d19c96e1fea4d040eb46aa404e24a" exitCode=0
Oct 30 21:36:01.583251 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:36:01.583153    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-89af8e-54695b5qtr" event={"ID":"1ea5a200-89c8-4cd6-b319-a438e0c9e553","Type":"ContainerDied","Data":"8a15ae4d8e29507d27b86543a7c3ad1ec05d19c96e1fea4d040eb46aa404e24a"}
Oct 30 21:36:01.583251 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:36:01.583185    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-89af8e-54695b5qtr" event={"ID":"1ea5a200-89c8-4cd6-b319-a438e0c9e553","Type":"ContainerDied","Data":"952f7c4666c63747e14aa2f3d1c80cadab90d748e2321ed87da0dbdb8389421a"}
Oct 30 21:36:01.583251 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:36:01.583185    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-89af8e-54695b5qtr"
Oct 30 21:36:01.593359 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:36:01.593333    5713 scope.go:122] "RemoveContainer" containerID="8d20f8671667fe024a685a16f20cff23712bdfa781c9b843aef67638c22c211c"
Oct 30 21:36:01.594041 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:36:01.594015    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d20f8671667fe024a685a16f20cff23712bdfa781c9b843aef67638c22c211c\": container with ID starting with 8d20f8671667fe024a685a16f20cff23712bdfa781c9b843aef67638c22c211c not found: ID does not exist" containerID="8d20f8671667fe024a685a16f20cff23712bdfa781c9b843aef67638c22c211c"
Oct 30 21:36:01.594172 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:36:01.594055    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d20f8671667fe024a685a16f20cff23712bdfa781c9b843aef67638c22c211c"} err="failed to get container status \"8d20f8671667fe024a685a16f20cff23712bdfa781c9b843aef67638c22c211c\": rpc error: code = NotFound desc = could not find container \"8d20f8671667fe024a685a16f20cff23712bdfa781c9b843aef67638c22c211c\": container with ID starting with 8d20f8671667fe024a685a16f20cff23712bdfa781c9b843aef67638c22c211c not found: ID does not exist"
Oct 30 21:36:01.594172 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:36:01.594074    5713 scope.go:122] "RemoveContainer" containerID="8a15ae4d8e29507d27b86543a7c3ad1ec05d19c96e1fea4d040eb46aa404e24a"
Oct 30 21:36:01.604881 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:36:01.604860    5713 scope.go:122] "RemoveContainer" containerID="8a15ae4d8e29507d27b86543a7c3ad1ec05d19c96e1fea4d040eb46aa404e24a"
Oct 30 21:36:01.605221 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:36:01.605189    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a15ae4d8e29507d27b86543a7c3ad1ec05d19c96e1fea4d040eb46aa404e24a\": container with ID starting with 8a15ae4d8e29507d27b86543a7c3ad1ec05d19c96e1fea4d040eb46aa404e24a not found: ID does not exist" containerID="8a15ae4d8e29507d27b86543a7c3ad1ec05d19c96e1fea4d040eb46aa404e24a"
Oct 30 21:36:01.605338 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:36:01.605231    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a15ae4d8e29507d27b86543a7c3ad1ec05d19c96e1fea4d040eb46aa404e24a"} err="failed to get container status \"8a15ae4d8e29507d27b86543a7c3ad1ec05d19c96e1fea4d040eb46aa404e24a\": rpc error: code = NotFound desc = could not find container \"8a15ae4d8e29507d27b86543a7c3ad1ec05d19c96e1fea4d040eb46aa404e24a\": container with ID starting with 8a15ae4d8e29507d27b86543a7c3ad1ec05d19c96e1fea4d040eb46aa404e24a not found: ID does not exist"
Oct 30 21:36:01.744730 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:36:01.744692    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-89af8e-7bd9b978bfbfhx"]
Oct 30 21:36:01.748710 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:36:01.748682    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-89af8e-7bd9b978bfbfhx"]
Oct 30 21:36:01.859210 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:36:01.859175    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-89af8e-54695b5qtr"]
Oct 30 21:36:01.861103 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:36:01.861084    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-89af8e-54695b5qtr"]
Oct 30 21:36:03.061679 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:36:03.061637    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ea5a200-89c8-4cd6-b319-a438e0c9e553" path="/var/lib/kubelet/pods/1ea5a200-89c8-4cd6-b319-a438e0c9e553/volumes"
Oct 30 21:36:03.062232 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:36:03.062212    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf9fde33-48d1-462c-be89-ef10584fd6a5" path="/var/lib/kubelet/pods/cf9fde33-48d1-462c-be89-ef10584fd6a5/volumes"
Oct 30 21:36:24.459557 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:36:24.459520    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:36:40.337418 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:36:40.337388    5713 scope.go:122] "RemoveContainer" containerID="1d73186fbbd7c1c39025681de1b15c3609dabb952c53433dae21c8561a116a2e"
Oct 30 21:36:40.346390 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:36:40.346367    5713 scope.go:122] "RemoveContainer" containerID="c569612ff49af3e90d9fecb6c92a1a072a575683dd91aca88afc63a287497606"
Oct 30 21:36:46.045292 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:36:46.045255    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vjc66"]
Oct 30 21:36:46.094977 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:36:46.094939    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vjc66"]
Oct 30 21:36:47.063198 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:36:47.063153    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad47d82f-6b21-4a15-9f2a-3ebb3d4134dc" path="/var/lib/kubelet/pods/ad47d82f-6b21-4a15-9f2a-3ebb3d4134dc/volumes"
Oct 30 21:36:54.457854 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:36:54.457812    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:37:06.045533 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:37:06.045493    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-hzt4v"]
Oct 30 21:37:06.094975 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:37:06.094947    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-hzt4v"]
Oct 30 21:37:07.061859 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:37:07.061818    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="204b868f-aba0-4de3-86ed-7a16f4382dfc" path="/var/lib/kubelet/pods/204b868f-aba0-4de3-86ed-7a16f4382dfc/volumes"
Oct 30 21:37:24.457257 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:37:24.457219    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:37:40.384845 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:37:40.384784    5713 scope.go:122] "RemoveContainer" containerID="d99a0dd3e22b8bdde846557083fa3a43a5ea4cff68ed0a13b55038dbc386ebca"
Oct 30 21:37:54.458271 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:37:54.458235    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:38:05.145048 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:05.144978    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-8nzgh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:38:05.145636 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:05.145094    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8nzgh" podUID="d91e8f40-c2d6-455c-8d73-9f83b615f3dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:38:05.206765 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:05.206715    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-fdp2g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:38:05.207003 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:05.206808    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-fdp2g" podUID="b905ceb0-d3ed-426a-90fe-3717158cee72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:38:13.771047 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:13.771011    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4z97z"]
Oct 30 21:38:13.771457 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:13.771405    5713 cpu_manager.go:414] "containerMap: removing container" podUID="1ea5a200-89c8-4cd6-b319-a438e0c9e553" containerName="payload-pre-processing"
Oct 30 21:38:13.771457 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:13.771415    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ea5a200-89c8-4cd6-b319-a438e0c9e553" containerName="payload-pre-processing"
Oct 30 21:38:13.771457 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:13.771426    5713 cpu_manager.go:414] "containerMap: removing container" podUID="cf9fde33-48d1-462c-be89-ef10584fd6a5" containerName="payload-processing"
Oct 30 21:38:13.771457 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:13.771431    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf9fde33-48d1-462c-be89-ef10584fd6a5" containerName="payload-processing"
Oct 30 21:38:13.771457 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:13.771451    5713 cpu_manager.go:414] "containerMap: removing container" podUID="a6b9cfa2-a0ad-415f-903e-9f96b2d00bb1" containerName="revoke-keys"
Oct 30 21:38:13.771457 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:13.771456    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6b9cfa2-a0ad-415f-903e-9f96b2d00bb1" containerName="revoke-keys"
Oct 30 21:38:13.771664 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:13.771520    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6b9cfa2-a0ad-415f-903e-9f96b2d00bb1" containerName="revoke-keys"
Oct 30 21:38:13.771664 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:13.771532    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ea5a200-89c8-4cd6-b319-a438e0c9e553" containerName="payload-pre-processing"
Oct 30 21:38:13.771664 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:13.771538    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf9fde33-48d1-462c-be89-ef10584fd6a5" containerName="payload-processing"
Oct 30 21:38:13.774361 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:13.774338    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4z97z"
Oct 30 21:38:13.878690 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:13.878650    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:38:13.878922 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:13.878843    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:38:13.878922 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:13.878905    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:38:13.879051 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:13.878949    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:38:13.881087 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:13.881067    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4z97z"]
Oct 30 21:38:13.932255 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:13.932221    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/580dd8f4-d053-4878-b779-675505987f89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-4z97z\" (UID: \"580dd8f4-d053-4878-b779-675505987f89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4z97z"
Oct 30 21:38:13.932427 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:13.932265    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k84qn\" (UniqueName: \"kubernetes.io/projected/580dd8f4-d053-4878-b779-675505987f89-kube-api-access-k84qn\") pod \"maas-api-revoke-sub-d2bc93bc-4z97z\" (UID: \"580dd8f4-d053-4878-b779-675505987f89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4z97z"
Oct 30 21:38:13.932427 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:13.932416    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/580dd8f4-d053-4878-b779-675505987f89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-4z97z\" (UID: \"580dd8f4-d053-4878-b779-675505987f89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4z97z"
Oct 30 21:38:14.033698 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:14.033610    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/580dd8f4-d053-4878-b779-675505987f89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-4z97z\" (UID: \"580dd8f4-d053-4878-b779-675505987f89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4z97z"
Oct 30 21:38:14.033698 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:14.033689    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/580dd8f4-d053-4878-b779-675505987f89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-4z97z\" (UID: \"580dd8f4-d053-4878-b779-675505987f89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4z97z"
Oct 30 21:38:14.033941 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:14.033723    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k84qn\" (UniqueName: \"kubernetes.io/projected/580dd8f4-d053-4878-b779-675505987f89-kube-api-access-k84qn\") pod \"maas-api-revoke-sub-d2bc93bc-4z97z\" (UID: \"580dd8f4-d053-4878-b779-675505987f89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4z97z"
Oct 30 21:38:14.034467 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:14.034443    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/580dd8f4-d053-4878-b779-675505987f89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-4z97z\" (UID: \"580dd8f4-d053-4878-b779-675505987f89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4z97z"
Oct 30 21:38:14.036293 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:14.036274    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/580dd8f4-d053-4878-b779-675505987f89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-4z97z\" (UID: \"580dd8f4-d053-4878-b779-675505987f89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4z97z"
Oct 30 21:38:14.071407 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:14.071372    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rvj4r"]
Oct 30 21:38:14.077238 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:14.077204    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rvj4r"]
Oct 30 21:38:14.097984 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:14.097954    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k84qn\" (UniqueName: \"kubernetes.io/projected/580dd8f4-d053-4878-b779-675505987f89-kube-api-access-k84qn\") pod \"maas-api-revoke-sub-d2bc93bc-4z97z\" (UID: \"580dd8f4-d053-4878-b779-675505987f89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4z97z"
Oct 30 21:38:14.387354 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:14.387271    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4z97z"
Oct 30 21:38:14.759699 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:14.759674    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4z97z"]
Oct 30 21:38:14.762239 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:38:14.762208    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod580dd8f4_d053_4878_b779_675505987f89.slice/crio-51530c4818a0e012ae315e821b12217505c6dc9361cabb5c7ec20af86839d3fd WatchSource:0}: Error finding container 51530c4818a0e012ae315e821b12217505c6dc9361cabb5c7ec20af86839d3fd: Status 404 returned error can't find the container with id 51530c4818a0e012ae315e821b12217505c6dc9361cabb5c7ec20af86839d3fd
Oct 30 21:38:15.062708 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:15.062677    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb0e8b23-8074-49f5-b460-dcf5f750cb63" path="/var/lib/kubelet/pods/eb0e8b23-8074-49f5-b460-dcf5f750cb63/volumes"
Oct 30 21:38:15.369677 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:15.369583    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4z97z" event={"ID":"580dd8f4-d053-4878-b779-675505987f89","Type":"ContainerStarted","Data":"595ab2f0b78d6d34ba0a63607f8cd23bb284bab412a74651af88284dc71dcd24"}
Oct 30 21:38:15.369677 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:15.369645    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4z97z" event={"ID":"580dd8f4-d053-4878-b779-675505987f89","Type":"ContainerStarted","Data":"51530c4818a0e012ae315e821b12217505c6dc9361cabb5c7ec20af86839d3fd"}
Oct 30 21:38:15.484821 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:15.484742    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4z97z" podStartSLOduration=2.484727294 podStartE2EDuration="2.484727294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:15.483770661 +0000 UTC m=+3578.548547061" watchObservedRunningTime="2026-10-30 21:38:15.484727294 +0000 UTC m=+3578.549503756"
Oct 30 21:38:15.667101 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:38:15.667066    5713 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod580dd8f4_d053_4878_b779_675505987f89.slice/crio-595ab2f0b78d6d34ba0a63607f8cd23bb284bab412a74651af88284dc71dcd24.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod580dd8f4_d053_4878_b779_675505987f89.slice/crio-conmon-595ab2f0b78d6d34ba0a63607f8cd23bb284bab412a74651af88284dc71dcd24.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 21:38:16.377463 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:16.377426    5713 generic.go:356] "Generic (PLEG): container finished" podID="580dd8f4-d053-4878-b779-675505987f89" containerID="595ab2f0b78d6d34ba0a63607f8cd23bb284bab412a74651af88284dc71dcd24" exitCode=0
Oct 30 21:38:16.377911 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:16.377495    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4z97z" event={"ID":"580dd8f4-d053-4878-b779-675505987f89","Type":"ContainerDied","Data":"595ab2f0b78d6d34ba0a63607f8cd23bb284bab412a74651af88284dc71dcd24"}
Oct 30 21:38:17.526661 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:17.526634    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4z97z"
Oct 30 21:38:17.667922 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:17.667847    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/580dd8f4-d053-4878-b779-675505987f89-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/580dd8f4-d053-4878-b779-675505987f89-maas-api-xks-ca\") pod \"580dd8f4-d053-4878-b779-675505987f89\" (UID: \"580dd8f4-d053-4878-b779-675505987f89\") "
Oct 30 21:38:17.668084 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:17.667967    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/580dd8f4-d053-4878-b779-675505987f89-kube-api-access-k84qn\" (UniqueName: \"kubernetes.io/projected/580dd8f4-d053-4878-b779-675505987f89-kube-api-access-k84qn\") pod \"580dd8f4-d053-4878-b779-675505987f89\" (UID: \"580dd8f4-d053-4878-b779-675505987f89\") "
Oct 30 21:38:17.668084 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:17.667987    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/580dd8f4-d053-4878-b779-675505987f89-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/580dd8f4-d053-4878-b779-675505987f89-maas-api-openshift-ca\") pod \"580dd8f4-d053-4878-b779-675505987f89\" (UID: \"580dd8f4-d053-4878-b779-675505987f89\") "
Oct 30 21:38:17.668433 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:17.668402    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/580dd8f4-d053-4878-b779-675505987f89-maas-api-openshift-ca" pod "580dd8f4-d053-4878-b779-675505987f89" (UID: "580dd8f4-d053-4878-b779-675505987f89"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:38:17.670140 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:17.670112    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/580dd8f4-d053-4878-b779-675505987f89-maas-api-xks-ca" pod "580dd8f4-d053-4878-b779-675505987f89" (UID: "580dd8f4-d053-4878-b779-675505987f89"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:38:17.670254 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:17.670234    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/580dd8f4-d053-4878-b779-675505987f89-kube-api-access-k84qn" pod "580dd8f4-d053-4878-b779-675505987f89" (UID: "580dd8f4-d053-4878-b779-675505987f89"). InnerVolumeSpecName "kube-api-access-k84qn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:38:17.769493 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:17.769459    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/580dd8f4-d053-4878-b779-675505987f89-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:38:17.769493 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:17.769490    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k84qn\" (UniqueName: \"kubernetes.io/projected/580dd8f4-d053-4878-b779-675505987f89-kube-api-access-k84qn\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:38:17.769493 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:17.769501    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/580dd8f4-d053-4878-b779-675505987f89-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:38:18.391417 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:18.391384    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4z97z"
Oct 30 21:38:18.391417 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:18.391399    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4z97z" event={"ID":"580dd8f4-d053-4878-b779-675505987f89","Type":"ContainerDied","Data":"51530c4818a0e012ae315e821b12217505c6dc9361cabb5c7ec20af86839d3fd"}
Oct 30 21:38:18.393579 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:18.391447    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51530c4818a0e012ae315e821b12217505c6dc9361cabb5c7ec20af86839d3fd"
Oct 30 21:38:24.457713 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:38:24.457676    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:38:34.961605 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:34.961566    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3f401f-68954qhgfh"]
Oct 30 21:38:34.962250 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:34.962175    5713 cpu_manager.go:414] "containerMap: removing container" podUID="580dd8f4-d053-4878-b779-675505987f89" containerName="revoke-keys"
Oct 30 21:38:34.962250 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:34.962196    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="580dd8f4-d053-4878-b779-675505987f89" containerName="revoke-keys"
Oct 30 21:38:34.962381 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:34.962342    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="580dd8f4-d053-4878-b779-675505987f89" containerName="revoke-keys"
Oct 30 21:38:34.965629 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:34.965609    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3f401f-68954qhgfh"
Oct 30 21:38:35.045348 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:35.045307    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6r47\" (UniqueName: \"kubernetes.io/projected/bff3da8e-9570-44cf-8496-79321d68dd72-kube-api-access-b6r47\") pod \"payload-pre-processing-e2e-worker-single-main-3f401f-68954qhgfh\" (UID: \"bff3da8e-9570-44cf-8496-79321d68dd72\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3f401f-68954qhgfh"
Oct 30 21:38:35.045602 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:35.045584    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bff3da8e-9570-44cf-8496-79321d68dd72-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-3f401f-68954qhgfh\" (UID: \"bff3da8e-9570-44cf-8496-79321d68dd72\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3f401f-68954qhgfh"
Oct 30 21:38:35.068958 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:35.068926    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-3f401f\""
Oct 30 21:38:35.069136 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:35.068986    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-93d73747-dockercfg-h8z6r\""
Oct 30 21:38:35.072316 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:35.072285    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3f401f-68954qhgfh"]
Oct 30 21:38:35.109701 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:35.109670    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-3f401f-74b4fd855rnq2q"]
Oct 30 21:38:35.113357 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:35.113341    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3f401f-74b4fd855rnq2q"
Oct 30 21:38:35.147241 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:35.147211    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b3b9543-13db-48a2-a385-e641f9096a73-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-3f401f-74b4fd855rnq2q\" (UID: \"3b3b9543-13db-48a2-a385-e641f9096a73\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-3f401f-74b4fd855rnq2q"
Oct 30 21:38:35.147475 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:35.147278    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6r47\" (UniqueName: \"kubernetes.io/projected/bff3da8e-9570-44cf-8496-79321d68dd72-kube-api-access-b6r47\") pod \"payload-pre-processing-e2e-worker-single-main-3f401f-68954qhgfh\" (UID: \"bff3da8e-9570-44cf-8496-79321d68dd72\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3f401f-68954qhgfh"
Oct 30 21:38:35.147475 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:35.147302    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bff3da8e-9570-44cf-8496-79321d68dd72-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-3f401f-68954qhgfh\" (UID: \"bff3da8e-9570-44cf-8496-79321d68dd72\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3f401f-68954qhgfh"
Oct 30 21:38:35.147475 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:35.147418    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8vzg\" (UniqueName: \"kubernetes.io/projected/3b3b9543-13db-48a2-a385-e641f9096a73-kube-api-access-f8vzg\") pod \"payload-processing-e2e-worker-single-main-3f401f-74b4fd855rnq2q\" (UID: \"3b3b9543-13db-48a2-a385-e641f9096a73\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-3f401f-74b4fd855rnq2q"
Oct 30 21:38:35.148077 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:35.148060    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bff3da8e-9570-44cf-8496-79321d68dd72-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-3f401f-68954qhgfh\" (UID: \"bff3da8e-9570-44cf-8496-79321d68dd72\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3f401f-68954qhgfh"
Oct 30 21:38:35.168991 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:35.168960    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-3f401f-74b4fd855rnq2q"]
Oct 30 21:38:35.204389 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:35.204362    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6r47\" (UniqueName: \"kubernetes.io/projected/bff3da8e-9570-44cf-8496-79321d68dd72-kube-api-access-b6r47\") pod \"payload-pre-processing-e2e-worker-single-main-3f401f-68954qhgfh\" (UID: \"bff3da8e-9570-44cf-8496-79321d68dd72\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3f401f-68954qhgfh"
Oct 30 21:38:35.248390 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:35.248284    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8vzg\" (UniqueName: \"kubernetes.io/projected/3b3b9543-13db-48a2-a385-e641f9096a73-kube-api-access-f8vzg\") pod \"payload-processing-e2e-worker-single-main-3f401f-74b4fd855rnq2q\" (UID: \"3b3b9543-13db-48a2-a385-e641f9096a73\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-3f401f-74b4fd855rnq2q"
Oct 30 21:38:35.248390 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:35.248361    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b3b9543-13db-48a2-a385-e641f9096a73-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-3f401f-74b4fd855rnq2q\" (UID: \"3b3b9543-13db-48a2-a385-e641f9096a73\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-3f401f-74b4fd855rnq2q"
Oct 30 21:38:35.249066 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:35.249044    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b3b9543-13db-48a2-a385-e641f9096a73-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-3f401f-74b4fd855rnq2q\" (UID: \"3b3b9543-13db-48a2-a385-e641f9096a73\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-3f401f-74b4fd855rnq2q"
Oct 30 21:38:35.280209 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:35.280173    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3f401f-68954qhgfh"
Oct 30 21:38:35.305418 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:35.305389    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8vzg\" (UniqueName: \"kubernetes.io/projected/3b3b9543-13db-48a2-a385-e641f9096a73-kube-api-access-f8vzg\") pod \"payload-processing-e2e-worker-single-main-3f401f-74b4fd855rnq2q\" (UID: \"3b3b9543-13db-48a2-a385-e641f9096a73\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-3f401f-74b4fd855rnq2q"
Oct 30 21:38:35.426392 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:35.426352    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3f401f-74b4fd855rnq2q"
Oct 30 21:38:35.655862 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:35.655710    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3f401f-68954qhgfh"]
Oct 30 21:38:35.659403 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:38:35.659351    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbff3da8e_9570_44cf_8496_79321d68dd72.slice/crio-fbe7b29eedd5b719fd53744a2b213c7d561ed96949636553298a9d90815bc324 WatchSource:0}: Error finding container fbe7b29eedd5b719fd53744a2b213c7d561ed96949636553298a9d90815bc324: Status 404 returned error can't find the container with id fbe7b29eedd5b719fd53744a2b213c7d561ed96949636553298a9d90815bc324
Oct 30 21:38:35.799438 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:35.799412    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-3f401f-74b4fd855rnq2q"]
Oct 30 21:38:35.801590 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:38:35.801561    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b3b9543_13db_48a2_a385_e641f9096a73.slice/crio-8be315772b152bd1622602d11e9bf8567e1576057c31292a0688acee9755ff7e WatchSource:0}: Error finding container 8be315772b152bd1622602d11e9bf8567e1576057c31292a0688acee9755ff7e: Status 404 returned error can't find the container with id 8be315772b152bd1622602d11e9bf8567e1576057c31292a0688acee9755ff7e
Oct 30 21:38:36.491720 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:36.491681    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3f401f-68954qhgfh" event={"ID":"bff3da8e-9570-44cf-8496-79321d68dd72","Type":"ContainerStarted","Data":"4e1941dacf422ed5d07155b952e8432ad9eae19f0698e585fce723863fed6e7f"}
Oct 30 21:38:36.492204 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:36.491740    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3f401f-68954qhgfh" event={"ID":"bff3da8e-9570-44cf-8496-79321d68dd72","Type":"ContainerStarted","Data":"fbe7b29eedd5b719fd53744a2b213c7d561ed96949636553298a9d90815bc324"}
Oct 30 21:38:36.492204 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:36.491820    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3f401f-68954qhgfh"
Oct 30 21:38:36.493580 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:36.493557    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3f401f-74b4fd855rnq2q" event={"ID":"3b3b9543-13db-48a2-a385-e641f9096a73","Type":"ContainerStarted","Data":"94de3ff7ef3941f326dff9763975c5dd6b665809df07d7a93a59091aa545b725"}
Oct 30 21:38:36.493705 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:36.493591    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3f401f-74b4fd855rnq2q" event={"ID":"3b3b9543-13db-48a2-a385-e641f9096a73","Type":"ContainerStarted","Data":"8be315772b152bd1622602d11e9bf8567e1576057c31292a0688acee9755ff7e"}
Oct 30 21:38:36.493705 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:36.493702    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3f401f-74b4fd855rnq2q"
Oct 30 21:38:36.608043 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:36.607987    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3f401f-68954qhgfh" podStartSLOduration=2.607972091 podStartE2EDuration="2.607972091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:36.605549101 +0000 UTC m=+3599.670325505" watchObservedRunningTime="2026-10-30 21:38:36.607972091 +0000 UTC m=+3599.672748488"
Oct 30 21:38:36.720875 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:36.720809    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3f401f-74b4fd855rnq2q" podStartSLOduration=1.720775156 podStartE2EDuration="1.720775156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:36.720238581 +0000 UTC m=+3599.785014981" watchObservedRunningTime="2026-10-30 21:38:36.720775156 +0000 UTC m=+3599.785551551"
Oct 30 21:38:37.840868 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:37.840840    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d4597073-8ff0-4a4e-933c-c64e934fe977/prometheus/0.log"
Oct 30 21:38:37.854939 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:37.854891    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d4597073-8ff0-4a4e-933c-c64e934fe977/prometheus/0.log"
Oct 30 21:38:37.864434 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:37.864407    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r7lpf_764461d5-e0b2-4856-a97e-0c5d3d34a497/ovn-acl-logging/0.log"
Oct 30 21:38:37.872843 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:37.872820    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r7lpf_764461d5-e0b2-4856-a97e-0c5d3d34a497/ovn-acl-logging/0.log"
Oct 30 21:38:40.408670 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:40.408631    5713 scope.go:122] "RemoveContainer" containerID="63e01e8f6e7f3e32afe9d0b677e235f0ee37da75597593fca4e030be2b52713f"
Oct 30 21:38:40.417691 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:40.417669    5713 scope.go:122] "RemoveContainer" containerID="27302790f9275276e40c5347387ce22cb77124cc266ba129a1273e8b50c5095e"
Oct 30 21:38:40.425908 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:40.425888    5713 scope.go:122] "RemoveContainer" containerID="aa42a1bc8842228412c6428d702baabaf40824e42168b8605c01f84cd7ef9f73"
Oct 30 21:38:40.435145 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:40.435109    5713 scope.go:122] "RemoveContainer" containerID="bab809122973d6ad4da20dd816e16c5bc4e03f9d67549c825a91905e83ef1256"
Oct 30 21:38:47.502632 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:47.502551    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3f401f-74b4fd855rnq2q"
Oct 30 21:38:47.503091 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:47.502733    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3f401f-68954qhgfh"
Oct 30 21:38:54.458183 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:38:54.458137    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:38:59.559229 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:59.559198    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-3f401f/simulator-single-main-3f401f-kserve-5cb85bc544-rp4b9"]
Oct 30 21:38:59.564424 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:59.564403    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-3f401f/simulator-single-main-3f401f-kserve-5cb85bc544-rp4b9"
Oct 30 21:38:59.666670 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:59.666640    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-3f401f\"/\"openshift-service-ca.crt\""
Oct 30 21:38:59.666895 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:59.666699    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-3f401f\"/\"kube-root-ca.crt\""
Oct 30 21:38:59.667106 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:59.667087    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-3f401f\"/\"default-dockercfg-jhcts\""
Oct 30 21:38:59.667238 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:59.667223    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-3f401f\"/\"simulator-single-main-3f401f-kserve-self-signed-certs\""
Oct 30 21:38:59.668819 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:59.668799    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-3f401f/simulator-single-main-3f401f-kserve-5cb85bc544-rp4b9"]
Oct 30 21:38:59.696743 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:59.696716    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a572561-a9e9-4dcb-8214-73ddd01d7fae-home\") pod \"simulator-single-main-3f401f-kserve-5cb85bc544-rp4b9\" (UID: \"5a572561-a9e9-4dcb-8214-73ddd01d7fae\") " pod="e2e-models-e2e-worker-single-main-3f401f/simulator-single-main-3f401f-kserve-5cb85bc544-rp4b9"
Oct 30 21:38:59.696890 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:59.696800    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a572561-a9e9-4dcb-8214-73ddd01d7fae-model-cache\") pod \"simulator-single-main-3f401f-kserve-5cb85bc544-rp4b9\" (UID: \"5a572561-a9e9-4dcb-8214-73ddd01d7fae\") " pod="e2e-models-e2e-worker-single-main-3f401f/simulator-single-main-3f401f-kserve-5cb85bc544-rp4b9"
Oct 30 21:38:59.696960 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:59.696905    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a572561-a9e9-4dcb-8214-73ddd01d7fae-tls-certs\") pod \"simulator-single-main-3f401f-kserve-5cb85bc544-rp4b9\" (UID: \"5a572561-a9e9-4dcb-8214-73ddd01d7fae\") " pod="e2e-models-e2e-worker-single-main-3f401f/simulator-single-main-3f401f-kserve-5cb85bc544-rp4b9"
Oct 30 21:38:59.697061 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:59.696978    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a572561-a9e9-4dcb-8214-73ddd01d7fae-tmp-dir\") pod \"simulator-single-main-3f401f-kserve-5cb85bc544-rp4b9\" (UID: \"5a572561-a9e9-4dcb-8214-73ddd01d7fae\") " pod="e2e-models-e2e-worker-single-main-3f401f/simulator-single-main-3f401f-kserve-5cb85bc544-rp4b9"
Oct 30 21:38:59.697110 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:59.697054    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a572561-a9e9-4dcb-8214-73ddd01d7fae-dshm\") pod \"simulator-single-main-3f401f-kserve-5cb85bc544-rp4b9\" (UID: \"5a572561-a9e9-4dcb-8214-73ddd01d7fae\") " pod="e2e-models-e2e-worker-single-main-3f401f/simulator-single-main-3f401f-kserve-5cb85bc544-rp4b9"
Oct 30 21:38:59.697170 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:59.697116    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2f4w\" (UniqueName: \"kubernetes.io/projected/5a572561-a9e9-4dcb-8214-73ddd01d7fae-kube-api-access-w2f4w\") pod \"simulator-single-main-3f401f-kserve-5cb85bc544-rp4b9\" (UID: \"5a572561-a9e9-4dcb-8214-73ddd01d7fae\") " pod="e2e-models-e2e-worker-single-main-3f401f/simulator-single-main-3f401f-kserve-5cb85bc544-rp4b9"
Oct 30 21:38:59.798525 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:59.798495    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a572561-a9e9-4dcb-8214-73ddd01d7fae-model-cache\") pod \"simulator-single-main-3f401f-kserve-5cb85bc544-rp4b9\" (UID: \"5a572561-a9e9-4dcb-8214-73ddd01d7fae\") " pod="e2e-models-e2e-worker-single-main-3f401f/simulator-single-main-3f401f-kserve-5cb85bc544-rp4b9"
Oct 30 21:38:59.798738 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:59.798538    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a572561-a9e9-4dcb-8214-73ddd01d7fae-tls-certs\") pod \"simulator-single-main-3f401f-kserve-5cb85bc544-rp4b9\" (UID: \"5a572561-a9e9-4dcb-8214-73ddd01d7fae\") " pod="e2e-models-e2e-worker-single-main-3f401f/simulator-single-main-3f401f-kserve-5cb85bc544-rp4b9"
Oct 30 21:38:59.798738 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:59.798583    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a572561-a9e9-4dcb-8214-73ddd01d7fae-tmp-dir\") pod \"simulator-single-main-3f401f-kserve-5cb85bc544-rp4b9\" (UID: \"5a572561-a9e9-4dcb-8214-73ddd01d7fae\") " pod="e2e-models-e2e-worker-single-main-3f401f/simulator-single-main-3f401f-kserve-5cb85bc544-rp4b9"
Oct 30 21:38:59.798738 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:59.798600    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a572561-a9e9-4dcb-8214-73ddd01d7fae-dshm\") pod \"simulator-single-main-3f401f-kserve-5cb85bc544-rp4b9\" (UID: \"5a572561-a9e9-4dcb-8214-73ddd01d7fae\") " pod="e2e-models-e2e-worker-single-main-3f401f/simulator-single-main-3f401f-kserve-5cb85bc544-rp4b9"
Oct 30 21:38:59.798738 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:59.798634    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2f4w\" (UniqueName: \"kubernetes.io/projected/5a572561-a9e9-4dcb-8214-73ddd01d7fae-kube-api-access-w2f4w\") pod \"simulator-single-main-3f401f-kserve-5cb85bc544-rp4b9\" (UID: \"5a572561-a9e9-4dcb-8214-73ddd01d7fae\") " pod="e2e-models-e2e-worker-single-main-3f401f/simulator-single-main-3f401f-kserve-5cb85bc544-rp4b9"
Oct 30 21:38:59.798738 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:59.798677    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a572561-a9e9-4dcb-8214-73ddd01d7fae-home\") pod \"simulator-single-main-3f401f-kserve-5cb85bc544-rp4b9\" (UID: \"5a572561-a9e9-4dcb-8214-73ddd01d7fae\") " pod="e2e-models-e2e-worker-single-main-3f401f/simulator-single-main-3f401f-kserve-5cb85bc544-rp4b9"
Oct 30 21:38:59.799070 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:59.798938    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a572561-a9e9-4dcb-8214-73ddd01d7fae-model-cache\") pod \"simulator-single-main-3f401f-kserve-5cb85bc544-rp4b9\" (UID: \"5a572561-a9e9-4dcb-8214-73ddd01d7fae\") " pod="e2e-models-e2e-worker-single-main-3f401f/simulator-single-main-3f401f-kserve-5cb85bc544-rp4b9"
Oct 30 21:38:59.799070 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:59.798999    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a572561-a9e9-4dcb-8214-73ddd01d7fae-tmp-dir\") pod \"simulator-single-main-3f401f-kserve-5cb85bc544-rp4b9\" (UID: \"5a572561-a9e9-4dcb-8214-73ddd01d7fae\") " pod="e2e-models-e2e-worker-single-main-3f401f/simulator-single-main-3f401f-kserve-5cb85bc544-rp4b9"
Oct 30 21:38:59.799155 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:59.799076    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a572561-a9e9-4dcb-8214-73ddd01d7fae-home\") pod \"simulator-single-main-3f401f-kserve-5cb85bc544-rp4b9\" (UID: \"5a572561-a9e9-4dcb-8214-73ddd01d7fae\") " pod="e2e-models-e2e-worker-single-main-3f401f/simulator-single-main-3f401f-kserve-5cb85bc544-rp4b9"
Oct 30 21:38:59.801134 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:59.801113    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a572561-a9e9-4dcb-8214-73ddd01d7fae-dshm\") pod \"simulator-single-main-3f401f-kserve-5cb85bc544-rp4b9\" (UID: \"5a572561-a9e9-4dcb-8214-73ddd01d7fae\") " pod="e2e-models-e2e-worker-single-main-3f401f/simulator-single-main-3f401f-kserve-5cb85bc544-rp4b9"
Oct 30 21:38:59.801284 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:59.801267    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a572561-a9e9-4dcb-8214-73ddd01d7fae-tls-certs\") pod \"simulator-single-main-3f401f-kserve-5cb85bc544-rp4b9\" (UID: \"5a572561-a9e9-4dcb-8214-73ddd01d7fae\") " pod="e2e-models-e2e-worker-single-main-3f401f/simulator-single-main-3f401f-kserve-5cb85bc544-rp4b9"
Oct 30 21:38:59.855616 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:59.855532    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2f4w\" (UniqueName: \"kubernetes.io/projected/5a572561-a9e9-4dcb-8214-73ddd01d7fae-kube-api-access-w2f4w\") pod \"simulator-single-main-3f401f-kserve-5cb85bc544-rp4b9\" (UID: \"5a572561-a9e9-4dcb-8214-73ddd01d7fae\") " pod="e2e-models-e2e-worker-single-main-3f401f/simulator-single-main-3f401f-kserve-5cb85bc544-rp4b9"
Oct 30 21:38:59.876451 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:38:59.876417    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-3f401f/simulator-single-main-3f401f-kserve-5cb85bc544-rp4b9"
Oct 30 21:39:00.250429 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:00.250399    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-3f401f/simulator-single-main-3f401f-kserve-5cb85bc544-rp4b9"]
Oct 30 21:39:00.252645 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:39:00.252620    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a572561_a9e9_4dcb_8214_73ddd01d7fae.slice/crio-569a0e3e291fcfd114464363b6f40d9795cfa7c1ac4f0bb4436356701b48bb21 WatchSource:0}: Error finding container 569a0e3e291fcfd114464363b6f40d9795cfa7c1ac4f0bb4436356701b48bb21: Status 404 returned error can't find the container with id 569a0e3e291fcfd114464363b6f40d9795cfa7c1ac4f0bb4436356701b48bb21
Oct 30 21:39:00.635925 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:00.635726    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-3f401f/simulator-single-main-3f401f-kserve-5cb85bc544-rp4b9" event={"ID":"5a572561-a9e9-4dcb-8214-73ddd01d7fae","Type":"ContainerStarted","Data":"ef08cc3d235a8c6949da33d05c563a4ad886654c4b5ef888832191dfd9b5a9f6"}
Oct 30 21:39:00.635925 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:00.635770    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-3f401f/simulator-single-main-3f401f-kserve-5cb85bc544-rp4b9" event={"ID":"5a572561-a9e9-4dcb-8214-73ddd01d7fae","Type":"ContainerStarted","Data":"569a0e3e291fcfd114464363b6f40d9795cfa7c1ac4f0bb4436356701b48bb21"}
Oct 30 21:39:00.750729 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:00.750671    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-3f401f/simulator-single-main-3f401f-kserve-5cb85bc544-rp4b9" podStartSLOduration=1.750656471 podStartE2EDuration="1.750656471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:00.749827285 +0000 UTC m=+3623.814603680" watchObservedRunningTime="2026-10-30 21:39:00.750656471 +0000 UTC m=+3623.815432869"
Oct 30 21:39:00.876930 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:00.876892    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-3f401f/simulator-single-main-3f401f-kserve-5cb85bc544-rp4b9"
Oct 30 21:39:09.876710 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:09.876673    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-3f401f/simulator-single-main-3f401f-kserve-5cb85bc544-rp4b9"
Oct 30 21:39:09.889269 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:09.889246    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-3f401f/simulator-single-main-3f401f-kserve-5cb85bc544-rp4b9"
Oct 30 21:39:10.702546 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:10.702515    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-3f401f/simulator-single-main-3f401f-kserve-5cb85bc544-rp4b9"
Oct 30 21:39:24.457553 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:39:24.457508    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:39:40.483575 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:40.483541    5713 scope.go:122] "RemoveContainer" containerID="7881b4912c5143d50ebcb0e26a0e14ea6ee10c4e9cb64dd783ddbc6624f566b5"
Oct 30 21:39:40.492718 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:40.492699    5713 scope.go:122] "RemoveContainer" containerID="d66068f65643fdf4c2431984d9bf02947093e2d5914850f5333712a979262d11"
Oct 30 21:39:45.162496 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:45.162461    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16fe7dab-ftjwv"]
Oct 30 21:39:45.167351 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:45.167333    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fe7dab-ftjwv"
Oct 30 21:39:45.220102 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:45.220054    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:39:45.269644 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:45.269619    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:39:45.269847 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:45.269682    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:39:45.269847 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:45.269682    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:39:45.271929 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:45.271911    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16fe7dab-ftjwv"]
Oct 30 21:39:45.355096 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:45.355064    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8xrt\" (UniqueName: \"kubernetes.io/projected/f40a5e4f-c02f-4d81-9041-11d3947d26cf-kube-api-access-t8xrt\") pod \"maas-api-revoke-sub-16fe7dab-ftjwv\" (UID: \"f40a5e4f-c02f-4d81-9041-11d3947d26cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fe7dab-ftjwv"
Oct 30 21:39:45.355290 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:45.355204    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f40a5e4f-c02f-4d81-9041-11d3947d26cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16fe7dab-ftjwv\" (UID: \"f40a5e4f-c02f-4d81-9041-11d3947d26cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fe7dab-ftjwv"
Oct 30 21:39:45.355290 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:45.355259    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f40a5e4f-c02f-4d81-9041-11d3947d26cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16fe7dab-ftjwv\" (UID: \"f40a5e4f-c02f-4d81-9041-11d3947d26cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fe7dab-ftjwv"
Oct 30 21:39:45.455778 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:45.455740    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8xrt\" (UniqueName: \"kubernetes.io/projected/f40a5e4f-c02f-4d81-9041-11d3947d26cf-kube-api-access-t8xrt\") pod \"maas-api-revoke-sub-16fe7dab-ftjwv\" (UID: \"f40a5e4f-c02f-4d81-9041-11d3947d26cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fe7dab-ftjwv"
Oct 30 21:39:45.456000 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:45.455866    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f40a5e4f-c02f-4d81-9041-11d3947d26cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16fe7dab-ftjwv\" (UID: \"f40a5e4f-c02f-4d81-9041-11d3947d26cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fe7dab-ftjwv"
Oct 30 21:39:45.456000 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:45.455932    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f40a5e4f-c02f-4d81-9041-11d3947d26cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16fe7dab-ftjwv\" (UID: \"f40a5e4f-c02f-4d81-9041-11d3947d26cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fe7dab-ftjwv"
Oct 30 21:39:45.456613 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:45.456547    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f40a5e4f-c02f-4d81-9041-11d3947d26cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16fe7dab-ftjwv\" (UID: \"f40a5e4f-c02f-4d81-9041-11d3947d26cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fe7dab-ftjwv"
Oct 30 21:39:45.458370 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:45.458349    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f40a5e4f-c02f-4d81-9041-11d3947d26cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16fe7dab-ftjwv\" (UID: \"f40a5e4f-c02f-4d81-9041-11d3947d26cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fe7dab-ftjwv"
Oct 30 21:39:45.512723 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:45.512686    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8xrt\" (UniqueName: \"kubernetes.io/projected/f40a5e4f-c02f-4d81-9041-11d3947d26cf-kube-api-access-t8xrt\") pod \"maas-api-revoke-sub-16fe7dab-ftjwv\" (UID: \"f40a5e4f-c02f-4d81-9041-11d3947d26cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fe7dab-ftjwv"
Oct 30 21:39:45.780360 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:45.780268    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fe7dab-ftjwv"
Oct 30 21:39:46.150994 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:46.150970    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16fe7dab-ftjwv"]
Oct 30 21:39:46.153707 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:39:46.153680    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf40a5e4f_c02f_4d81_9041_11d3947d26cf.slice/crio-c951cfe9d2a07c92c1ccacb2c98fbc3f761a421e8f61a25d1e36dd7a6c7c802c WatchSource:0}: Error finding container c951cfe9d2a07c92c1ccacb2c98fbc3f761a421e8f61a25d1e36dd7a6c7c802c: Status 404 returned error can't find the container with id c951cfe9d2a07c92c1ccacb2c98fbc3f761a421e8f61a25d1e36dd7a6c7c802c
Oct 30 21:39:46.897076 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:46.896983    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fe7dab-ftjwv" event={"ID":"f40a5e4f-c02f-4d81-9041-11d3947d26cf","Type":"ContainerStarted","Data":"92425b4f6d7eb2ee00fb195316b990f56038940c34e6fa64a567a3c18f8f5c77"}
Oct 30 21:39:46.897076 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:46.897041    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fe7dab-ftjwv" event={"ID":"f40a5e4f-c02f-4d81-9041-11d3947d26cf","Type":"ContainerStarted","Data":"c951cfe9d2a07c92c1ccacb2c98fbc3f761a421e8f61a25d1e36dd7a6c7c802c"}
Oct 30 21:39:47.012819 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:47.012697    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fe7dab-ftjwv" podStartSLOduration=2.012674413 podStartE2EDuration="2.012674413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:47.010991806 +0000 UTC m=+3670.075768206" watchObservedRunningTime="2026-10-30 21:39:47.012674413 +0000 UTC m=+3670.077450811"
Oct 30 21:39:47.904326 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:47.904289    5713 generic.go:356] "Generic (PLEG): container finished" podID="f40a5e4f-c02f-4d81-9041-11d3947d26cf" containerID="92425b4f6d7eb2ee00fb195316b990f56038940c34e6fa64a567a3c18f8f5c77" exitCode=0
Oct 30 21:39:47.904751 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:47.904374    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fe7dab-ftjwv" event={"ID":"f40a5e4f-c02f-4d81-9041-11d3947d26cf","Type":"ContainerDied","Data":"92425b4f6d7eb2ee00fb195316b990f56038940c34e6fa64a567a3c18f8f5c77"}
Oct 30 21:39:49.044139 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:49.044112    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fe7dab-ftjwv"
Oct 30 21:39:49.087105 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:49.087073    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f40a5e4f-c02f-4d81-9041-11d3947d26cf-kube-api-access-t8xrt\" (UniqueName: \"kubernetes.io/projected/f40a5e4f-c02f-4d81-9041-11d3947d26cf-kube-api-access-t8xrt\") pod \"f40a5e4f-c02f-4d81-9041-11d3947d26cf\" (UID: \"f40a5e4f-c02f-4d81-9041-11d3947d26cf\") "
Oct 30 21:39:49.087255 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:49.087126    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f40a5e4f-c02f-4d81-9041-11d3947d26cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f40a5e4f-c02f-4d81-9041-11d3947d26cf-maas-api-openshift-ca\") pod \"f40a5e4f-c02f-4d81-9041-11d3947d26cf\" (UID: \"f40a5e4f-c02f-4d81-9041-11d3947d26cf\") "
Oct 30 21:39:49.087255 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:49.087203    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f40a5e4f-c02f-4d81-9041-11d3947d26cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f40a5e4f-c02f-4d81-9041-11d3947d26cf-maas-api-xks-ca\") pod \"f40a5e4f-c02f-4d81-9041-11d3947d26cf\" (UID: \"f40a5e4f-c02f-4d81-9041-11d3947d26cf\") "
Oct 30 21:39:49.087441 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:49.087416    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f40a5e4f-c02f-4d81-9041-11d3947d26cf-maas-api-openshift-ca" pod "f40a5e4f-c02f-4d81-9041-11d3947d26cf" (UID: "f40a5e4f-c02f-4d81-9041-11d3947d26cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:39:49.087520 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:49.087507    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f40a5e4f-c02f-4d81-9041-11d3947d26cf-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:39:49.089275 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:49.089251    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f40a5e4f-c02f-4d81-9041-11d3947d26cf-kube-api-access-t8xrt" pod "f40a5e4f-c02f-4d81-9041-11d3947d26cf" (UID: "f40a5e4f-c02f-4d81-9041-11d3947d26cf"). InnerVolumeSpecName "kube-api-access-t8xrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:39:49.089388 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:49.089326    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f40a5e4f-c02f-4d81-9041-11d3947d26cf-maas-api-xks-ca" pod "f40a5e4f-c02f-4d81-9041-11d3947d26cf" (UID: "f40a5e4f-c02f-4d81-9041-11d3947d26cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:39:49.188166 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:49.188137    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f40a5e4f-c02f-4d81-9041-11d3947d26cf-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:39:49.188166 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:49.188163    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8xrt\" (UniqueName: \"kubernetes.io/projected/f40a5e4f-c02f-4d81-9041-11d3947d26cf-kube-api-access-t8xrt\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:39:49.918291 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:49.918264    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fe7dab-ftjwv"
Oct 30 21:39:49.918475 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:49.918284    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16fe7dab-ftjwv" event={"ID":"f40a5e4f-c02f-4d81-9041-11d3947d26cf","Type":"ContainerDied","Data":"c951cfe9d2a07c92c1ccacb2c98fbc3f761a421e8f61a25d1e36dd7a6c7c802c"}
Oct 30 21:39:49.918475 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:49.918333    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c951cfe9d2a07c92c1ccacb2c98fbc3f761a421e8f61a25d1e36dd7a6c7c802c"
Oct 30 21:39:54.458391 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:39:54.458356    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:39:56.812445 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:56.812409    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3f401f-4b55c"]
Oct 30 21:39:56.813058 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:56.813037    5713 cpu_manager.go:414] "containerMap: removing container" podUID="f40a5e4f-c02f-4d81-9041-11d3947d26cf" containerName="revoke-keys"
Oct 30 21:39:56.813142 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:56.813060    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f40a5e4f-c02f-4d81-9041-11d3947d26cf" containerName="revoke-keys"
Oct 30 21:39:56.813211 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:56.813181    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="f40a5e4f-c02f-4d81-9041-11d3947d26cf" containerName="revoke-keys"
Oct 30 21:39:56.816374 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:56.816354    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3f401f-4b55c"
Oct 30 21:39:56.855405 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:56.855368    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wptsx\" (UniqueName: \"kubernetes.io/projected/cb198023-e257-4920-936c-5fffbb231e1a-kube-api-access-wptsx\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3f401f-4b55c\" (UID: \"cb198023-e257-4920-936c-5fffbb231e1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3f401f-4b55c"
Oct 30 21:39:56.855578 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:56.855419    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb198023-e257-4920-936c-5fffbb231e1a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3f401f-4b55c\" (UID: \"cb198023-e257-4920-936c-5fffbb231e1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3f401f-4b55c"
Oct 30 21:39:56.855641 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:56.855574    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb198023-e257-4920-936c-5fffbb231e1a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3f401f-4b55c\" (UID: \"cb198023-e257-4920-936c-5fffbb231e1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3f401f-4b55c"
Oct 30 21:39:56.868814 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:56.868748    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:39:56.918852 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:56.918781    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:39:56.919060 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:56.918911    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:39:56.919060 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:56.918925    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:39:56.921467 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:56.921441    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3f401f-4b55c"]
Oct 30 21:39:56.956963 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:56.956934    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wptsx\" (UniqueName: \"kubernetes.io/projected/cb198023-e257-4920-936c-5fffbb231e1a-kube-api-access-wptsx\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3f401f-4b55c\" (UID: \"cb198023-e257-4920-936c-5fffbb231e1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3f401f-4b55c"
Oct 30 21:39:56.957121 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:56.956974    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb198023-e257-4920-936c-5fffbb231e1a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3f401f-4b55c\" (UID: \"cb198023-e257-4920-936c-5fffbb231e1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3f401f-4b55c"
Oct 30 21:39:56.957121 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:56.957074    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb198023-e257-4920-936c-5fffbb231e1a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3f401f-4b55c\" (UID: \"cb198023-e257-4920-936c-5fffbb231e1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3f401f-4b55c"
Oct 30 21:39:56.957859 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:56.957831    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb198023-e257-4920-936c-5fffbb231e1a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3f401f-4b55c\" (UID: \"cb198023-e257-4920-936c-5fffbb231e1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3f401f-4b55c"
Oct 30 21:39:56.960125 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:56.960091    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb198023-e257-4920-936c-5fffbb231e1a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3f401f-4b55c\" (UID: \"cb198023-e257-4920-936c-5fffbb231e1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3f401f-4b55c"
Oct 30 21:39:57.013516 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:57.013488    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wptsx\" (UniqueName: \"kubernetes.io/projected/cb198023-e257-4920-936c-5fffbb231e1a-kube-api-access-wptsx\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3f401f-4b55c\" (UID: \"cb198023-e257-4920-936c-5fffbb231e1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3f401f-4b55c"
Oct 30 21:39:57.129559 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:57.129469    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3f401f-4b55c"
Oct 30 21:39:57.502605 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:57.502582    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3f401f-4b55c"]
Oct 30 21:39:57.505021 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:39:57.504993    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb198023_e257_4920_936c_5fffbb231e1a.slice/crio-25ce0ec6bd2d4ec3d1bfbb471ed6bf1ca2fe0d651d6405625997d691bee05863 WatchSource:0}: Error finding container 25ce0ec6bd2d4ec3d1bfbb471ed6bf1ca2fe0d651d6405625997d691bee05863: Status 404 returned error can't find the container with id 25ce0ec6bd2d4ec3d1bfbb471ed6bf1ca2fe0d651d6405625997d691bee05863
Oct 30 21:39:57.969674 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:57.969638    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3f401f-4b55c" event={"ID":"cb198023-e257-4920-936c-5fffbb231e1a","Type":"ContainerStarted","Data":"f5a657577e4aa9e4d5d77a38eb9bf2167043d414f8964201eef61cd4b2c5fd25"}
Oct 30 21:39:57.970151 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:57.969696    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3f401f-4b55c" event={"ID":"cb198023-e257-4920-936c-5fffbb231e1a","Type":"ContainerStarted","Data":"25ce0ec6bd2d4ec3d1bfbb471ed6bf1ca2fe0d651d6405625997d691bee05863"}
Oct 30 21:39:58.084021 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:58.083969    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3f401f-4b55c" podStartSLOduration=2.083956795 podStartE2EDuration="2.083956795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:58.081262786 +0000 UTC m=+3681.146039198" watchObservedRunningTime="2026-10-30 21:39:58.083956795 +0000 UTC m=+3681.148733192"
Oct 30 21:39:58.976521 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:58.976479    5713 generic.go:356] "Generic (PLEG): container finished" podID="cb198023-e257-4920-936c-5fffbb231e1a" containerID="f5a657577e4aa9e4d5d77a38eb9bf2167043d414f8964201eef61cd4b2c5fd25" exitCode=0
Oct 30 21:39:58.976963 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:39:58.976560    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3f401f-4b55c" event={"ID":"cb198023-e257-4920-936c-5fffbb231e1a","Type":"ContainerDied","Data":"f5a657577e4aa9e4d5d77a38eb9bf2167043d414f8964201eef61cd4b2c5fd25"}
Oct 30 21:40:00.120201 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:00.120175    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3f401f-4b55c"
Oct 30 21:40:00.186149 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:00.186118    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb198023-e257-4920-936c-5fffbb231e1a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb198023-e257-4920-936c-5fffbb231e1a-maas-api-openshift-ca\") pod \"cb198023-e257-4920-936c-5fffbb231e1a\" (UID: \"cb198023-e257-4920-936c-5fffbb231e1a\") "
Oct 30 21:40:00.186306 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:00.186192    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb198023-e257-4920-936c-5fffbb231e1a-kube-api-access-wptsx\" (UniqueName: \"kubernetes.io/projected/cb198023-e257-4920-936c-5fffbb231e1a-kube-api-access-wptsx\") pod \"cb198023-e257-4920-936c-5fffbb231e1a\" (UID: \"cb198023-e257-4920-936c-5fffbb231e1a\") "
Oct 30 21:40:00.186373 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:00.186321    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb198023-e257-4920-936c-5fffbb231e1a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb198023-e257-4920-936c-5fffbb231e1a-maas-api-xks-ca\") pod \"cb198023-e257-4920-936c-5fffbb231e1a\" (UID: \"cb198023-e257-4920-936c-5fffbb231e1a\") "
Oct 30 21:40:00.186462 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:00.186434    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb198023-e257-4920-936c-5fffbb231e1a-maas-api-openshift-ca" pod "cb198023-e257-4920-936c-5fffbb231e1a" (UID: "cb198023-e257-4920-936c-5fffbb231e1a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:40:00.186658 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:00.186641    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb198023-e257-4920-936c-5fffbb231e1a-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:40:00.188329 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:00.188307    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb198023-e257-4920-936c-5fffbb231e1a-kube-api-access-wptsx" pod "cb198023-e257-4920-936c-5fffbb231e1a" (UID: "cb198023-e257-4920-936c-5fffbb231e1a"). InnerVolumeSpecName "kube-api-access-wptsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:40:00.188499 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:00.188486    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb198023-e257-4920-936c-5fffbb231e1a-maas-api-xks-ca" pod "cb198023-e257-4920-936c-5fffbb231e1a" (UID: "cb198023-e257-4920-936c-5fffbb231e1a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:40:00.287413 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:00.287343    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wptsx\" (UniqueName: \"kubernetes.io/projected/cb198023-e257-4920-936c-5fffbb231e1a-kube-api-access-wptsx\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:40:00.287413 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:00.287369    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb198023-e257-4920-936c-5fffbb231e1a-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:40:00.988618 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:00.988583    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3f401f-4b55c" event={"ID":"cb198023-e257-4920-936c-5fffbb231e1a","Type":"ContainerDied","Data":"25ce0ec6bd2d4ec3d1bfbb471ed6bf1ca2fe0d651d6405625997d691bee05863"}
Oct 30 21:40:00.988618 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:00.988634    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25ce0ec6bd2d4ec3d1bfbb471ed6bf1ca2fe0d651d6405625997d691bee05863"
Oct 30 21:40:00.988886 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:00.988598    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3f401f-4b55c"
Oct 30 21:40:01.278129 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:01.278047    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-3f401f/simulator-single-main-3f401f-kserve-5cb85bc544-rp4b9"]
Oct 30 21:40:08.310279 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:08.310243    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-3f401f-74b4fd855rnq2q"]
Oct 30 21:40:08.420212 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:08.420178    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3f401f-68954qhgfh"]
Oct 30 21:40:13.310914 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:13.310846    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3f401f-74b4fd855rnq2q" podUID="3b3b9543-13db-48a2-a385-e641f9096a73" containerName="payload-processing" containerID="cri-o://94de3ff7ef3941f326dff9763975c5dd6b665809df07d7a93a59091aa545b725" gracePeriod=25
Oct 30 21:40:13.421401 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:13.421324    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3f401f-68954qhgfh" podUID="bff3da8e-9570-44cf-8496-79321d68dd72" containerName="payload-pre-processing" containerID="cri-o://4e1941dacf422ed5d07155b952e8432ad9eae19f0698e585fce723863fed6e7f" gracePeriod=25
Oct 30 21:40:16.045642 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:16.045605    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-864e0c4b-pch6l"]
Oct 30 21:40:16.095415 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:16.095383    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-864e0c4b-pch6l"]
Oct 30 21:40:16.298868 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:16.298703    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-3f401f/simulator-single-main-3f401f-kserve-5cb85bc544-rp4b9" podUID="5a572561-a9e9-4dcb-8214-73ddd01d7fae" containerName="main" containerID="cri-o://ef08cc3d235a8c6949da33d05c563a4ad886654c4b5ef888832191dfd9b5a9f6" gracePeriod=45
Oct 30 21:40:16.552519 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:16.552458    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-3f401f/simulator-single-main-3f401f-kserve-5cb85bc544-rp4b9"
Oct 30 21:40:16.656564 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:16.656533    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3f401f-4b55c"]
Oct 30 21:40:16.658465 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:16.658442    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3f401f-4b55c"]
Oct 30 21:40:16.668517 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:16.668488    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a572561-a9e9-4dcb-8214-73ddd01d7fae-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a572561-a9e9-4dcb-8214-73ddd01d7fae-model-cache\") pod \"5a572561-a9e9-4dcb-8214-73ddd01d7fae\" (UID: \"5a572561-a9e9-4dcb-8214-73ddd01d7fae\") "
Oct 30 21:40:16.668687 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:16.668564    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a572561-a9e9-4dcb-8214-73ddd01d7fae-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a572561-a9e9-4dcb-8214-73ddd01d7fae-tmp-dir\") pod \"5a572561-a9e9-4dcb-8214-73ddd01d7fae\" (UID: \"5a572561-a9e9-4dcb-8214-73ddd01d7fae\") "
Oct 30 21:40:16.668687 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:16.668634    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a572561-a9e9-4dcb-8214-73ddd01d7fae-tls-certs\" (UniqueName: \"kubernetes.io/secret/5a572561-a9e9-4dcb-8214-73ddd01d7fae-tls-certs\") pod \"5a572561-a9e9-4dcb-8214-73ddd01d7fae\" (UID: \"5a572561-a9e9-4dcb-8214-73ddd01d7fae\") "
Oct 30 21:40:16.668687 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:16.668671    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a572561-a9e9-4dcb-8214-73ddd01d7fae-home\" (UniqueName: \"kubernetes.io/empty-dir/5a572561-a9e9-4dcb-8214-73ddd01d7fae-home\") pod \"5a572561-a9e9-4dcb-8214-73ddd01d7fae\" (UID: \"5a572561-a9e9-4dcb-8214-73ddd01d7fae\") "
Oct 30 21:40:16.668911 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:16.668751    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a572561-a9e9-4dcb-8214-73ddd01d7fae-kube-api-access-w2f4w\" (UniqueName: \"kubernetes.io/projected/5a572561-a9e9-4dcb-8214-73ddd01d7fae-kube-api-access-w2f4w\") pod \"5a572561-a9e9-4dcb-8214-73ddd01d7fae\" (UID: \"5a572561-a9e9-4dcb-8214-73ddd01d7fae\") "
Oct 30 21:40:16.668911 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:16.668801    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a572561-a9e9-4dcb-8214-73ddd01d7fae-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a572561-a9e9-4dcb-8214-73ddd01d7fae-dshm\") pod \"5a572561-a9e9-4dcb-8214-73ddd01d7fae\" (UID: \"5a572561-a9e9-4dcb-8214-73ddd01d7fae\") "
Oct 30 21:40:16.668911 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:16.668816    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a572561-a9e9-4dcb-8214-73ddd01d7fae-model-cache" pod "5a572561-a9e9-4dcb-8214-73ddd01d7fae" (UID: "5a572561-a9e9-4dcb-8214-73ddd01d7fae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:40:16.668911 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:16.668895    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a572561-a9e9-4dcb-8214-73ddd01d7fae-tmp-dir" pod "5a572561-a9e9-4dcb-8214-73ddd01d7fae" (UID: "5a572561-a9e9-4dcb-8214-73ddd01d7fae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:40:16.669136 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:16.669062    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a572561-a9e9-4dcb-8214-73ddd01d7fae-home" pod "5a572561-a9e9-4dcb-8214-73ddd01d7fae" (UID: "5a572561-a9e9-4dcb-8214-73ddd01d7fae"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:40:16.669236 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:16.669215    5713 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a572561-a9e9-4dcb-8214-73ddd01d7fae-model-cache\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:40:16.669295 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:16.669242    5713 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a572561-a9e9-4dcb-8214-73ddd01d7fae-tmp-dir\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:40:16.669295 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:16.669255    5713 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a572561-a9e9-4dcb-8214-73ddd01d7fae-home\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:40:16.671629 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:16.671598    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a572561-a9e9-4dcb-8214-73ddd01d7fae-tls-certs" pod "5a572561-a9e9-4dcb-8214-73ddd01d7fae" (UID: "5a572561-a9e9-4dcb-8214-73ddd01d7fae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:40:16.671756 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:16.671681    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a572561-a9e9-4dcb-8214-73ddd01d7fae-dshm" pod "5a572561-a9e9-4dcb-8214-73ddd01d7fae" (UID: "5a572561-a9e9-4dcb-8214-73ddd01d7fae"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:40:16.671978 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:16.671944    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a572561-a9e9-4dcb-8214-73ddd01d7fae-kube-api-access-w2f4w" pod "5a572561-a9e9-4dcb-8214-73ddd01d7fae" (UID: "5a572561-a9e9-4dcb-8214-73ddd01d7fae"). InnerVolumeSpecName "kube-api-access-w2f4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:40:16.769842 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:16.769807    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2f4w\" (UniqueName: \"kubernetes.io/projected/5a572561-a9e9-4dcb-8214-73ddd01d7fae-kube-api-access-w2f4w\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:40:16.769842 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:16.769840    5713 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a572561-a9e9-4dcb-8214-73ddd01d7fae-dshm\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:40:16.769842 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:16.769851    5713 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a572561-a9e9-4dcb-8214-73ddd01d7fae-tls-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:40:17.061933 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:17.061891    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb198023-e257-4920-936c-5fffbb231e1a" path="/var/lib/kubelet/pods/cb198023-e257-4920-936c-5fffbb231e1a/volumes"
Oct 30 21:40:17.062495 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:17.062475    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfc0a108-5acc-4346-9942-50b15095ed2a" path="/var/lib/kubelet/pods/cfc0a108-5acc-4346-9942-50b15095ed2a/volumes"
Oct 30 21:40:17.077051 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:17.077018    5713 generic.go:356] "Generic (PLEG): container finished" podID="5a572561-a9e9-4dcb-8214-73ddd01d7fae" containerID="ef08cc3d235a8c6949da33d05c563a4ad886654c4b5ef888832191dfd9b5a9f6" exitCode=0
Oct 30 21:40:17.077201 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:17.077066    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-3f401f/simulator-single-main-3f401f-kserve-5cb85bc544-rp4b9"
Oct 30 21:40:17.077201 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:17.077116    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-3f401f/simulator-single-main-3f401f-kserve-5cb85bc544-rp4b9" event={"ID":"5a572561-a9e9-4dcb-8214-73ddd01d7fae","Type":"ContainerDied","Data":"ef08cc3d235a8c6949da33d05c563a4ad886654c4b5ef888832191dfd9b5a9f6"}
Oct 30 21:40:17.077201 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:17.077165    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-3f401f/simulator-single-main-3f401f-kserve-5cb85bc544-rp4b9" event={"ID":"5a572561-a9e9-4dcb-8214-73ddd01d7fae","Type":"ContainerDied","Data":"569a0e3e291fcfd114464363b6f40d9795cfa7c1ac4f0bb4436356701b48bb21"}
Oct 30 21:40:17.077201 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:17.077182    5713 scope.go:122] "RemoveContainer" containerID="ef08cc3d235a8c6949da33d05c563a4ad886654c4b5ef888832191dfd9b5a9f6"
Oct 30 21:40:17.087666 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:17.087647    5713 scope.go:122] "RemoveContainer" containerID="ef08cc3d235a8c6949da33d05c563a4ad886654c4b5ef888832191dfd9b5a9f6"
Oct 30 21:40:17.088016 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:40:17.087995    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef08cc3d235a8c6949da33d05c563a4ad886654c4b5ef888832191dfd9b5a9f6\": container with ID starting with ef08cc3d235a8c6949da33d05c563a4ad886654c4b5ef888832191dfd9b5a9f6 not found: ID does not exist" containerID="ef08cc3d235a8c6949da33d05c563a4ad886654c4b5ef888832191dfd9b5a9f6"
Oct 30 21:40:17.088081 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:17.088028    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef08cc3d235a8c6949da33d05c563a4ad886654c4b5ef888832191dfd9b5a9f6"} err="failed to get container status \"ef08cc3d235a8c6949da33d05c563a4ad886654c4b5ef888832191dfd9b5a9f6\": rpc error: code = NotFound desc = could not find container \"ef08cc3d235a8c6949da33d05c563a4ad886654c4b5ef888832191dfd9b5a9f6\": container with ID starting with ef08cc3d235a8c6949da33d05c563a4ad886654c4b5ef888832191dfd9b5a9f6 not found: ID does not exist"
Oct 30 21:40:17.239582 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:17.239545    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-3f401f/simulator-single-main-3f401f-kserve-5cb85bc544-rp4b9"]
Oct 30 21:40:17.243194 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:17.243165    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-3f401f/simulator-single-main-3f401f-kserve-5cb85bc544-rp4b9"]
Oct 30 21:40:17.501290 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:17.501251    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3f401f-68954qhgfh" podUID="bff3da8e-9570-44cf-8496-79321d68dd72" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:40:17.501507 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:17.501362    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3f401f-74b4fd855rnq2q" podUID="3b3b9543-13db-48a2-a385-e641f9096a73" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:40:19.062433 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:19.062400    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a572561-a9e9-4dcb-8214-73ddd01d7fae" path="/var/lib/kubelet/pods/5a572561-a9e9-4dcb-8214-73ddd01d7fae/volumes"
Oct 30 21:40:24.459128 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:40:24.459080    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:40:27.500580 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:27.500547    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3f401f-68954qhgfh" podUID="bff3da8e-9570-44cf-8496-79321d68dd72" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:40:27.501044 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:27.500557    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3f401f-74b4fd855rnq2q" podUID="3b3b9543-13db-48a2-a385-e641f9096a73" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:40:30.141644 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:30.141609    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-2a553e41-kserve-849fbb4cdf-d26cm"]
Oct 30 21:40:30.142129 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:30.142045    5713 cpu_manager.go:414] "containerMap: removing container" podUID="5a572561-a9e9-4dcb-8214-73ddd01d7fae" containerName="main"
Oct 30 21:40:30.142129 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:30.142058    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a572561-a9e9-4dcb-8214-73ddd01d7fae" containerName="main"
Oct 30 21:40:30.142129 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:30.142080    5713 cpu_manager.go:414] "containerMap: removing container" podUID="cb198023-e257-4920-936c-5fffbb231e1a" containerName="revoke-keys"
Oct 30 21:40:30.142129 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:30.142085    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb198023-e257-4920-936c-5fffbb231e1a" containerName="revoke-keys"
Oct 30 21:40:30.142299 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:30.142162    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a572561-a9e9-4dcb-8214-73ddd01d7fae" containerName="main"
Oct 30 21:40:30.142299 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:30.142173    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb198023-e257-4920-936c-5fffbb231e1a" containerName="revoke-keys"
Oct 30 21:40:30.146572 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:30.146548    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2a553e41-kserve-849fbb4cdf-d26cm"
Oct 30 21:40:30.203540 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:30.203487    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e-tmp-dir\") pod \"e2e-same-id-a-2a553e41-kserve-849fbb4cdf-d26cm\" (UID: \"07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e\") " pod="llm/e2e-same-id-a-2a553e41-kserve-849fbb4cdf-d26cm"
Oct 30 21:40:30.203751 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:30.203551    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-2a553e41-kserve-849fbb4cdf-d26cm"]
Oct 30 21:40:30.203751 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:30.203570    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e-tls-certs\") pod \"e2e-same-id-a-2a553e41-kserve-849fbb4cdf-d26cm\" (UID: \"07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e\") " pod="llm/e2e-same-id-a-2a553e41-kserve-849fbb4cdf-d26cm"
Oct 30 21:40:30.203751 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:30.203674    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88mcq\" (UniqueName: \"kubernetes.io/projected/07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e-kube-api-access-88mcq\") pod \"e2e-same-id-a-2a553e41-kserve-849fbb4cdf-d26cm\" (UID: \"07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e\") " pod="llm/e2e-same-id-a-2a553e41-kserve-849fbb4cdf-d26cm"
Oct 30 21:40:30.203751 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:30.203706    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e-dshm\") pod \"e2e-same-id-a-2a553e41-kserve-849fbb4cdf-d26cm\" (UID: \"07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e\") " pod="llm/e2e-same-id-a-2a553e41-kserve-849fbb4cdf-d26cm"
Oct 30 21:40:30.203751 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:30.203743    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e-model-cache\") pod \"e2e-same-id-a-2a553e41-kserve-849fbb4cdf-d26cm\" (UID: \"07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e\") " pod="llm/e2e-same-id-a-2a553e41-kserve-849fbb4cdf-d26cm"
Oct 30 21:40:30.204079 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:30.203822    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e-home\") pod \"e2e-same-id-a-2a553e41-kserve-849fbb4cdf-d26cm\" (UID: \"07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e\") " pod="llm/e2e-same-id-a-2a553e41-kserve-849fbb4cdf-d26cm"
Oct 30 21:40:30.250549 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:30.250514    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-2a553e41-kserve-self-signed-certs\""
Oct 30 21:40:30.305270 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:30.305232    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e-home\") pod \"e2e-same-id-a-2a553e41-kserve-849fbb4cdf-d26cm\" (UID: \"07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e\") " pod="llm/e2e-same-id-a-2a553e41-kserve-849fbb4cdf-d26cm"
Oct 30 21:40:30.305476 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:30.305300    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e-tmp-dir\") pod \"e2e-same-id-a-2a553e41-kserve-849fbb4cdf-d26cm\" (UID: \"07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e\") " pod="llm/e2e-same-id-a-2a553e41-kserve-849fbb4cdf-d26cm"
Oct 30 21:40:30.305476 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:30.305336    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e-tls-certs\") pod \"e2e-same-id-a-2a553e41-kserve-849fbb4cdf-d26cm\" (UID: \"07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e\") " pod="llm/e2e-same-id-a-2a553e41-kserve-849fbb4cdf-d26cm"
Oct 30 21:40:30.305476 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:30.305368    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88mcq\" (UniqueName: \"kubernetes.io/projected/07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e-kube-api-access-88mcq\") pod \"e2e-same-id-a-2a553e41-kserve-849fbb4cdf-d26cm\" (UID: \"07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e\") " pod="llm/e2e-same-id-a-2a553e41-kserve-849fbb4cdf-d26cm"
Oct 30 21:40:30.305476 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:30.305435    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e-dshm\") pod \"e2e-same-id-a-2a553e41-kserve-849fbb4cdf-d26cm\" (UID: \"07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e\") " pod="llm/e2e-same-id-a-2a553e41-kserve-849fbb4cdf-d26cm"
Oct 30 21:40:30.305727 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:30.305505    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e-model-cache\") pod \"e2e-same-id-a-2a553e41-kserve-849fbb4cdf-d26cm\" (UID: \"07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e\") " pod="llm/e2e-same-id-a-2a553e41-kserve-849fbb4cdf-d26cm"
Oct 30 21:40:30.305727 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:30.305693    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e-home\") pod \"e2e-same-id-a-2a553e41-kserve-849fbb4cdf-d26cm\" (UID: \"07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e\") " pod="llm/e2e-same-id-a-2a553e41-kserve-849fbb4cdf-d26cm"
Oct 30 21:40:30.305850 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:30.305751    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e-tmp-dir\") pod \"e2e-same-id-a-2a553e41-kserve-849fbb4cdf-d26cm\" (UID: \"07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e\") " pod="llm/e2e-same-id-a-2a553e41-kserve-849fbb4cdf-d26cm"
Oct 30 21:40:30.305899 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:30.305854    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e-model-cache\") pod \"e2e-same-id-a-2a553e41-kserve-849fbb4cdf-d26cm\" (UID: \"07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e\") " pod="llm/e2e-same-id-a-2a553e41-kserve-849fbb4cdf-d26cm"
Oct 30 21:40:30.307911 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:30.307890    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e-dshm\") pod \"e2e-same-id-a-2a553e41-kserve-849fbb4cdf-d26cm\" (UID: \"07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e\") " pod="llm/e2e-same-id-a-2a553e41-kserve-849fbb4cdf-d26cm"
Oct 30 21:40:30.308066 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:30.308047    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e-tls-certs\") pod \"e2e-same-id-a-2a553e41-kserve-849fbb4cdf-d26cm\" (UID: \"07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e\") " pod="llm/e2e-same-id-a-2a553e41-kserve-849fbb4cdf-d26cm"
Oct 30 21:40:30.362255 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:30.362219    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88mcq\" (UniqueName: \"kubernetes.io/projected/07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e-kube-api-access-88mcq\") pod \"e2e-same-id-a-2a553e41-kserve-849fbb4cdf-d26cm\" (UID: \"07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e\") " pod="llm/e2e-same-id-a-2a553e41-kserve-849fbb4cdf-d26cm"
Oct 30 21:40:30.458995 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:30.458952    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2a553e41-kserve-849fbb4cdf-d26cm"
Oct 30 21:40:30.849163 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:30.849135    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-2a553e41-kserve-849fbb4cdf-d26cm"]
Oct 30 21:40:30.851107 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:40:30.851075    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod07f7a4ed_8b7b_4812_a2b2_21d5f902ad1e.slice/crio-0bc9eb4bab8e7c1092f17baa93f62102a2816f57949c7341bbde4dc89edbc31d WatchSource:0}: Error finding container 0bc9eb4bab8e7c1092f17baa93f62102a2816f57949c7341bbde4dc89edbc31d: Status 404 returned error can't find the container with id 0bc9eb4bab8e7c1092f17baa93f62102a2816f57949c7341bbde4dc89edbc31d
Oct 30 21:40:31.158832 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:31.158711    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2a553e41-kserve-849fbb4cdf-d26cm" event={"ID":"07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e","Type":"ContainerStarted","Data":"110d96c3593510de4bf870e85d246d6e1908e758dae1b82ebc00fb3cc006f5f0"}
Oct 30 21:40:31.158832 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:31.158756    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2a553e41-kserve-849fbb4cdf-d26cm" event={"ID":"07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e","Type":"ContainerStarted","Data":"0bc9eb4bab8e7c1092f17baa93f62102a2816f57949c7341bbde4dc89edbc31d"}
Oct 30 21:40:31.272067 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:31.272012    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-2a553e41-kserve-849fbb4cdf-d26cm" podStartSLOduration=1.27199153 podStartE2EDuration="1.27199153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:31.271709782 +0000 UTC m=+3714.336486181" watchObservedRunningTime="2026-10-30 21:40:31.27199153 +0000 UTC m=+3714.336767930"
Oct 30 21:40:31.459544 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:31.459509    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-2a553e41-kserve-849fbb4cdf-d26cm"
Oct 30 21:40:33.566926 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:33.566859    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3f401f-74b4fd855rnq2q"
Oct 30 21:40:33.642539 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:33.642493    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b3b9543-13db-48a2-a385-e641f9096a73-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b3b9543-13db-48a2-a385-e641f9096a73-plugins-config-volume\") pod \"3b3b9543-13db-48a2-a385-e641f9096a73\" (UID: \"3b3b9543-13db-48a2-a385-e641f9096a73\") "
Oct 30 21:40:33.642771 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:33.642755    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b3b9543-13db-48a2-a385-e641f9096a73-kube-api-access-f8vzg\" (UniqueName: \"kubernetes.io/projected/3b3b9543-13db-48a2-a385-e641f9096a73-kube-api-access-f8vzg\") pod \"3b3b9543-13db-48a2-a385-e641f9096a73\" (UID: \"3b3b9543-13db-48a2-a385-e641f9096a73\") "
Oct 30 21:40:33.642861 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:33.642838    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b3b9543-13db-48a2-a385-e641f9096a73-plugins-config-volume" pod "3b3b9543-13db-48a2-a385-e641f9096a73" (UID: "3b3b9543-13db-48a2-a385-e641f9096a73"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:40:33.643062 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:33.643042    5713 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b3b9543-13db-48a2-a385-e641f9096a73-plugins-config-volume\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:40:33.645007 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:33.644979    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b3b9543-13db-48a2-a385-e641f9096a73-kube-api-access-f8vzg" pod "3b3b9543-13db-48a2-a385-e641f9096a73" (UID: "3b3b9543-13db-48a2-a385-e641f9096a73"). InnerVolumeSpecName "kube-api-access-f8vzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:40:33.660091 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:33.660067    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3f401f-68954qhgfh"
Oct 30 21:40:33.744000 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:33.743968    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bff3da8e-9570-44cf-8496-79321d68dd72-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bff3da8e-9570-44cf-8496-79321d68dd72-plugins-config-volume\") pod \"bff3da8e-9570-44cf-8496-79321d68dd72\" (UID: \"bff3da8e-9570-44cf-8496-79321d68dd72\") "
Oct 30 21:40:33.744184 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:33.744063    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bff3da8e-9570-44cf-8496-79321d68dd72-kube-api-access-b6r47\" (UniqueName: \"kubernetes.io/projected/bff3da8e-9570-44cf-8496-79321d68dd72-kube-api-access-b6r47\") pod \"bff3da8e-9570-44cf-8496-79321d68dd72\" (UID: \"bff3da8e-9570-44cf-8496-79321d68dd72\") "
Oct 30 21:40:33.744312 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:33.744299    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8vzg\" (UniqueName: \"kubernetes.io/projected/3b3b9543-13db-48a2-a385-e641f9096a73-kube-api-access-f8vzg\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:40:33.744382 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:33.744323    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bff3da8e-9570-44cf-8496-79321d68dd72-plugins-config-volume" pod "bff3da8e-9570-44cf-8496-79321d68dd72" (UID: "bff3da8e-9570-44cf-8496-79321d68dd72"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:40:33.746415 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:33.746394    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bff3da8e-9570-44cf-8496-79321d68dd72-kube-api-access-b6r47" pod "bff3da8e-9570-44cf-8496-79321d68dd72" (UID: "bff3da8e-9570-44cf-8496-79321d68dd72"). InnerVolumeSpecName "kube-api-access-b6r47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:40:33.845761 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:33.845691    5713 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bff3da8e-9570-44cf-8496-79321d68dd72-plugins-config-volume\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:40:33.845761 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:33.845717    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6r47\" (UniqueName: \"kubernetes.io/projected/bff3da8e-9570-44cf-8496-79321d68dd72-kube-api-access-b6r47\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:40:34.176439 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:34.176351    5713 generic.go:356] "Generic (PLEG): container finished" podID="bff3da8e-9570-44cf-8496-79321d68dd72" containerID="4e1941dacf422ed5d07155b952e8432ad9eae19f0698e585fce723863fed6e7f" exitCode=0
Oct 30 21:40:34.176439 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:34.176420    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3f401f-68954qhgfh"
Oct 30 21:40:34.176662 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:34.176439    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3f401f-68954qhgfh" event={"ID":"bff3da8e-9570-44cf-8496-79321d68dd72","Type":"ContainerDied","Data":"4e1941dacf422ed5d07155b952e8432ad9eae19f0698e585fce723863fed6e7f"}
Oct 30 21:40:34.176662 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:34.176535    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3f401f-68954qhgfh" event={"ID":"bff3da8e-9570-44cf-8496-79321d68dd72","Type":"ContainerDied","Data":"fbe7b29eedd5b719fd53744a2b213c7d561ed96949636553298a9d90815bc324"}
Oct 30 21:40:34.176662 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:34.176570    5713 scope.go:122] "RemoveContainer" containerID="4e1941dacf422ed5d07155b952e8432ad9eae19f0698e585fce723863fed6e7f"
Oct 30 21:40:34.177997 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:34.177972    5713 generic.go:356] "Generic (PLEG): container finished" podID="3b3b9543-13db-48a2-a385-e641f9096a73" containerID="94de3ff7ef3941f326dff9763975c5dd6b665809df07d7a93a59091aa545b725" exitCode=0
Oct 30 21:40:34.178106 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:34.178037    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3f401f-74b4fd855rnq2q"
Oct 30 21:40:34.178106 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:34.178050    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3f401f-74b4fd855rnq2q" event={"ID":"3b3b9543-13db-48a2-a385-e641f9096a73","Type":"ContainerDied","Data":"94de3ff7ef3941f326dff9763975c5dd6b665809df07d7a93a59091aa545b725"}
Oct 30 21:40:34.178106 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:34.178101    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3f401f-74b4fd855rnq2q" event={"ID":"3b3b9543-13db-48a2-a385-e641f9096a73","Type":"ContainerDied","Data":"8be315772b152bd1622602d11e9bf8567e1576057c31292a0688acee9755ff7e"}
Oct 30 21:40:34.188266 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:34.188246    5713 scope.go:122] "RemoveContainer" containerID="4e1941dacf422ed5d07155b952e8432ad9eae19f0698e585fce723863fed6e7f"
Oct 30 21:40:34.188655 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:40:34.188628    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e1941dacf422ed5d07155b952e8432ad9eae19f0698e585fce723863fed6e7f\": container with ID starting with 4e1941dacf422ed5d07155b952e8432ad9eae19f0698e585fce723863fed6e7f not found: ID does not exist" containerID="4e1941dacf422ed5d07155b952e8432ad9eae19f0698e585fce723863fed6e7f"
Oct 30 21:40:34.188705 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:34.188666    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e1941dacf422ed5d07155b952e8432ad9eae19f0698e585fce723863fed6e7f"} err="failed to get container status \"4e1941dacf422ed5d07155b952e8432ad9eae19f0698e585fce723863fed6e7f\": rpc error: code = NotFound desc = could not find container \"4e1941dacf422ed5d07155b952e8432ad9eae19f0698e585fce723863fed6e7f\": container with ID starting with 4e1941dacf422ed5d07155b952e8432ad9eae19f0698e585fce723863fed6e7f not found: ID does not exist"
Oct 30 21:40:34.188705 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:34.188686    5713 scope.go:122] "RemoveContainer" containerID="94de3ff7ef3941f326dff9763975c5dd6b665809df07d7a93a59091aa545b725"
Oct 30 21:40:34.198393 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:34.198371    5713 scope.go:122] "RemoveContainer" containerID="94de3ff7ef3941f326dff9763975c5dd6b665809df07d7a93a59091aa545b725"
Oct 30 21:40:34.198706 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:40:34.198684    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94de3ff7ef3941f326dff9763975c5dd6b665809df07d7a93a59091aa545b725\": container with ID starting with 94de3ff7ef3941f326dff9763975c5dd6b665809df07d7a93a59091aa545b725 not found: ID does not exist" containerID="94de3ff7ef3941f326dff9763975c5dd6b665809df07d7a93a59091aa545b725"
Oct 30 21:40:34.198766 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:34.198715    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94de3ff7ef3941f326dff9763975c5dd6b665809df07d7a93a59091aa545b725"} err="failed to get container status \"94de3ff7ef3941f326dff9763975c5dd6b665809df07d7a93a59091aa545b725\": rpc error: code = NotFound desc = could not find container \"94de3ff7ef3941f326dff9763975c5dd6b665809df07d7a93a59091aa545b725\": container with ID starting with 94de3ff7ef3941f326dff9763975c5dd6b665809df07d7a93a59091aa545b725 not found: ID does not exist"
Oct 30 21:40:34.339249 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:34.339217    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3f401f-68954qhgfh"]
Oct 30 21:40:34.388750 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:34.388710    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3f401f-68954qhgfh"]
Oct 30 21:40:34.450462 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:34.450428    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-3f401f-74b4fd855rnq2q"]
Oct 30 21:40:34.456016 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:34.455990    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-3f401f-74b4fd855rnq2q"]
Oct 30 21:40:35.069384 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:35.069342    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b3b9543-13db-48a2-a385-e641f9096a73" path="/var/lib/kubelet/pods/3b3b9543-13db-48a2-a385-e641f9096a73/volumes"
Oct 30 21:40:35.069908 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:35.069776    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bff3da8e-9570-44cf-8496-79321d68dd72" path="/var/lib/kubelet/pods/bff3da8e-9570-44cf-8496-79321d68dd72/volumes"
Oct 30 21:40:40.460069 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:40.460030    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-2a553e41-kserve-849fbb4cdf-d26cm"
Oct 30 21:40:40.472652 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:40.472629    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-2a553e41-kserve-849fbb4cdf-d26cm"
Oct 30 21:40:41.253533 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:41.253502    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-2a553e41-kserve-849fbb4cdf-d26cm"
Oct 30 21:40:54.457771 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:40:54.457725    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:40:56.044282 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:56.044245    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6q6fz"]
Oct 30 21:40:56.093623 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:56.093594    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6q6fz"]
Oct 30 21:40:57.070551 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:40:57.070512    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6b9cfa2-a0ad-415f-903e-9f96b2d00bb1" path="/var/lib/kubelet/pods/a6b9cfa2-a0ad-415f-903e-9f96b2d00bb1/volumes"
Oct 30 21:41:12.614367 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:41:12.614331    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-2a553e41-kserve-5d44b5487c-627bp"]
Oct 30 21:41:12.614774 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:41:12.614719    5713 cpu_manager.go:414] "containerMap: removing container" podUID="bff3da8e-9570-44cf-8496-79321d68dd72" containerName="payload-pre-processing"
Oct 30 21:41:12.614774 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:41:12.614730    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bff3da8e-9570-44cf-8496-79321d68dd72" containerName="payload-pre-processing"
Oct 30 21:41:12.614774 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:41:12.614739    5713 cpu_manager.go:414] "containerMap: removing container" podUID="3b3b9543-13db-48a2-a385-e641f9096a73" containerName="payload-processing"
Oct 30 21:41:12.614774 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:41:12.614745    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b3b9543-13db-48a2-a385-e641f9096a73" containerName="payload-processing"
Oct 30 21:41:12.614984 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:41:12.614855    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="bff3da8e-9570-44cf-8496-79321d68dd72" containerName="payload-pre-processing"
Oct 30 21:41:12.614984 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:41:12.614863    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b3b9543-13db-48a2-a385-e641f9096a73" containerName="payload-processing"
Oct 30 21:41:12.617603 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:41:12.617587    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-2a553e41-kserve-5d44b5487c-627bp"
Oct 30 21:41:12.661661 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:41:12.661622    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af52aa33-0c07-42c8-9d74-f83f31e3fc1a-dshm\") pod \"e2e-same-id-b-2a553e41-kserve-5d44b5487c-627bp\" (UID: \"af52aa33-0c07-42c8-9d74-f83f31e3fc1a\") " pod="llm/e2e-same-id-b-2a553e41-kserve-5d44b5487c-627bp"
Oct 30 21:41:12.661661 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:41:12.661662    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af52aa33-0c07-42c8-9d74-f83f31e3fc1a-tmp-dir\") pod \"e2e-same-id-b-2a553e41-kserve-5d44b5487c-627bp\" (UID: \"af52aa33-0c07-42c8-9d74-f83f31e3fc1a\") " pod="llm/e2e-same-id-b-2a553e41-kserve-5d44b5487c-627bp"
Oct 30 21:41:12.662144 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:41:12.661697    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af52aa33-0c07-42c8-9d74-f83f31e3fc1a-model-cache\") pod \"e2e-same-id-b-2a553e41-kserve-5d44b5487c-627bp\" (UID: \"af52aa33-0c07-42c8-9d74-f83f31e3fc1a\") " pod="llm/e2e-same-id-b-2a553e41-kserve-5d44b5487c-627bp"
Oct 30 21:41:12.662144 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:41:12.661852    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9tpv\" (UniqueName: \"kubernetes.io/projected/af52aa33-0c07-42c8-9d74-f83f31e3fc1a-kube-api-access-w9tpv\") pod \"e2e-same-id-b-2a553e41-kserve-5d44b5487c-627bp\" (UID: \"af52aa33-0c07-42c8-9d74-f83f31e3fc1a\") " pod="llm/e2e-same-id-b-2a553e41-kserve-5d44b5487c-627bp"
Oct 30 21:41:12.662144 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:41:12.661912    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af52aa33-0c07-42c8-9d74-f83f31e3fc1a-home\") pod \"e2e-same-id-b-2a553e41-kserve-5d44b5487c-627bp\" (UID: \"af52aa33-0c07-42c8-9d74-f83f31e3fc1a\") " pod="llm/e2e-same-id-b-2a553e41-kserve-5d44b5487c-627bp"
Oct 30 21:41:12.662144 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:41:12.661970    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af52aa33-0c07-42c8-9d74-f83f31e3fc1a-tls-certs\") pod \"e2e-same-id-b-2a553e41-kserve-5d44b5487c-627bp\" (UID: \"af52aa33-0c07-42c8-9d74-f83f31e3fc1a\") " pod="llm/e2e-same-id-b-2a553e41-kserve-5d44b5487c-627bp"
Oct 30 21:41:12.719125 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:41:12.719087    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-2a553e41-kserve-self-signed-certs\""
Oct 30 21:41:12.721325 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:41:12.721304    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-2a553e41-kserve-5d44b5487c-627bp"]
Oct 30 21:41:12.762809 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:41:12.762759    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af52aa33-0c07-42c8-9d74-f83f31e3fc1a-tls-certs\") pod \"e2e-same-id-b-2a553e41-kserve-5d44b5487c-627bp\" (UID: \"af52aa33-0c07-42c8-9d74-f83f31e3fc1a\") " pod="llm/e2e-same-id-b-2a553e41-kserve-5d44b5487c-627bp"
Oct 30 21:41:12.763029 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:41:12.762846    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af52aa33-0c07-42c8-9d74-f83f31e3fc1a-dshm\") pod \"e2e-same-id-b-2a553e41-kserve-5d44b5487c-627bp\" (UID: \"af52aa33-0c07-42c8-9d74-f83f31e3fc1a\") " pod="llm/e2e-same-id-b-2a553e41-kserve-5d44b5487c-627bp"
Oct 30 21:41:12.763029 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:41:12.762868    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af52aa33-0c07-42c8-9d74-f83f31e3fc1a-tmp-dir\") pod \"e2e-same-id-b-2a553e41-kserve-5d44b5487c-627bp\" (UID: \"af52aa33-0c07-42c8-9d74-f83f31e3fc1a\") " pod="llm/e2e-same-id-b-2a553e41-kserve-5d44b5487c-627bp"
Oct 30 21:41:12.763029 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:41:12.762906    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af52aa33-0c07-42c8-9d74-f83f31e3fc1a-model-cache\") pod \"e2e-same-id-b-2a553e41-kserve-5d44b5487c-627bp\" (UID: \"af52aa33-0c07-42c8-9d74-f83f31e3fc1a\") " pod="llm/e2e-same-id-b-2a553e41-kserve-5d44b5487c-627bp"
Oct 30 21:41:12.763029 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:41:12.762929    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9tpv\" (UniqueName: \"kubernetes.io/projected/af52aa33-0c07-42c8-9d74-f83f31e3fc1a-kube-api-access-w9tpv\") pod \"e2e-same-id-b-2a553e41-kserve-5d44b5487c-627bp\" (UID: \"af52aa33-0c07-42c8-9d74-f83f31e3fc1a\") " pod="llm/e2e-same-id-b-2a553e41-kserve-5d44b5487c-627bp"
Oct 30 21:41:12.763029 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:41:12.762962    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af52aa33-0c07-42c8-9d74-f83f31e3fc1a-home\") pod \"e2e-same-id-b-2a553e41-kserve-5d44b5487c-627bp\" (UID: \"af52aa33-0c07-42c8-9d74-f83f31e3fc1a\") " pod="llm/e2e-same-id-b-2a553e41-kserve-5d44b5487c-627bp"
Oct 30 21:41:12.763369 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:41:12.763345    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af52aa33-0c07-42c8-9d74-f83f31e3fc1a-model-cache\") pod \"e2e-same-id-b-2a553e41-kserve-5d44b5487c-627bp\" (UID: \"af52aa33-0c07-42c8-9d74-f83f31e3fc1a\") " pod="llm/e2e-same-id-b-2a553e41-kserve-5d44b5487c-627bp"
Oct 30 21:41:12.763440 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:41:12.763392    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af52aa33-0c07-42c8-9d74-f83f31e3fc1a-tmp-dir\") pod \"e2e-same-id-b-2a553e41-kserve-5d44b5487c-627bp\" (UID: \"af52aa33-0c07-42c8-9d74-f83f31e3fc1a\") " pod="llm/e2e-same-id-b-2a553e41-kserve-5d44b5487c-627bp"
Oct 30 21:41:12.763440 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:41:12.763415    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af52aa33-0c07-42c8-9d74-f83f31e3fc1a-home\") pod \"e2e-same-id-b-2a553e41-kserve-5d44b5487c-627bp\" (UID: \"af52aa33-0c07-42c8-9d74-f83f31e3fc1a\") " pod="llm/e2e-same-id-b-2a553e41-kserve-5d44b5487c-627bp"
Oct 30 21:41:12.765405 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:41:12.765379    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af52aa33-0c07-42c8-9d74-f83f31e3fc1a-dshm\") pod \"e2e-same-id-b-2a553e41-kserve-5d44b5487c-627bp\" (UID: \"af52aa33-0c07-42c8-9d74-f83f31e3fc1a\") " pod="llm/e2e-same-id-b-2a553e41-kserve-5d44b5487c-627bp"
Oct 30 21:41:12.766109 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:41:12.766079    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af52aa33-0c07-42c8-9d74-f83f31e3fc1a-tls-certs\") pod \"e2e-same-id-b-2a553e41-kserve-5d44b5487c-627bp\" (UID: \"af52aa33-0c07-42c8-9d74-f83f31e3fc1a\") " pod="llm/e2e-same-id-b-2a553e41-kserve-5d44b5487c-627bp"
Oct 30 21:41:12.822775 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:41:12.822743    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9tpv\" (UniqueName: \"kubernetes.io/projected/af52aa33-0c07-42c8-9d74-f83f31e3fc1a-kube-api-access-w9tpv\") pod \"e2e-same-id-b-2a553e41-kserve-5d44b5487c-627bp\" (UID: \"af52aa33-0c07-42c8-9d74-f83f31e3fc1a\") " pod="llm/e2e-same-id-b-2a553e41-kserve-5d44b5487c-627bp"
Oct 30 21:41:12.930928 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:41:12.930869    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-2a553e41-kserve-5d44b5487c-627bp"
Oct 30 21:41:13.306059 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:41:13.305980    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-2a553e41-kserve-5d44b5487c-627bp"]
Oct 30 21:41:13.309289 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:41:13.309258    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaf52aa33_0c07_42c8_9d74_f83f31e3fc1a.slice/crio-767d11d89b28a6f7ffcd8ec95d420551dd400862e6027bf0587074b8111dc200 WatchSource:0}: Error finding container 767d11d89b28a6f7ffcd8ec95d420551dd400862e6027bf0587074b8111dc200: Status 404 returned error can't find the container with id 767d11d89b28a6f7ffcd8ec95d420551dd400862e6027bf0587074b8111dc200
Oct 30 21:41:13.422958 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:41:13.422914    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-2a553e41-kserve-5d44b5487c-627bp" event={"ID":"af52aa33-0c07-42c8-9d74-f83f31e3fc1a","Type":"ContainerStarted","Data":"218f28e7181dd755390f4e8f4f6f34e09d26b9f5fc4847321a2ce9f6f06ae307"}
Oct 30 21:41:13.423164 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:41:13.422981    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-2a553e41-kserve-5d44b5487c-627bp" event={"ID":"af52aa33-0c07-42c8-9d74-f83f31e3fc1a","Type":"ContainerStarted","Data":"767d11d89b28a6f7ffcd8ec95d420551dd400862e6027bf0587074b8111dc200"}
Oct 30 21:41:13.536730 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:41:13.536675    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-2a553e41-kserve-5d44b5487c-627bp" podStartSLOduration=1.53666177 podStartE2EDuration="1.53666177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:13.535124833 +0000 UTC m=+3756.599901231" watchObservedRunningTime="2026-10-30 21:41:13.53666177 +0000 UTC m=+3756.601438168"
Oct 30 21:41:13.931436 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:41:13.931403    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-2a553e41-kserve-5d44b5487c-627bp"
Oct 30 21:41:20.147810 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:41:20.147287    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-8nzgh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:41:20.147810 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:41:20.147407    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8nzgh" podUID="d91e8f40-c2d6-455c-8d73-9f83b615f3dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:41:20.247925 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:41:20.247880    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-fdp2g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:41:20.248231 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:41:20.248201    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-fdp2g" podUID="b905ceb0-d3ed-426a-90fe-3717158cee72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:41:22.931680 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:41:22.931640    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-2a553e41-kserve-5d44b5487c-627bp"
Oct 30 21:41:22.944553 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:41:22.944522    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-2a553e41-kserve-5d44b5487c-627bp"
Oct 30 21:41:23.516701 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:41:23.516675    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-2a553e41-kserve-5d44b5487c-627bp"
Oct 30 21:41:24.457446 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:41:24.457406    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:41:40.539043 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:41:40.539010    5713 scope.go:122] "RemoveContainer" containerID="a02b127601dd231f1d1449f48b6a7786ed1edc7f48d386f9529a3e62676dfd70"
Oct 30 21:41:40.548107 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:41:40.548083    5713 scope.go:122] "RemoveContainer" containerID="66216310782a7ed6845e0171fa288c42804cc106c2ab9ea5f7d0746638da9e37"
Oct 30 21:41:54.458276 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:41:54.458191    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:41:54.668126 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:41:54.668083    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-2a553e41-kserve-849fbb4cdf-d26cm"]
Oct 30 21:42:07.115574 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:07.115532    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-2a553e41-kserve-5d44b5487c-627bp"]
Oct 30 21:42:09.688602 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:09.688541    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-2a553e41-kserve-849fbb4cdf-d26cm" podUID="07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e" containerName="main" containerID="cri-o://110d96c3593510de4bf870e85d246d6e1908e758dae1b82ebc00fb3cc006f5f0" gracePeriod=45
Oct 30 21:42:09.939507 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:09.939444    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2a553e41-kserve-849fbb4cdf-d26cm"
Oct 30 21:42:10.008293 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:10.008267    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e-home\" (UniqueName: \"kubernetes.io/empty-dir/07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e-home\") pod \"07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e\" (UID: \"07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e\") "
Oct 30 21:42:10.008431 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:10.008321    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e-tls-certs\" (UniqueName: \"kubernetes.io/secret/07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e-tls-certs\") pod \"07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e\" (UID: \"07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e\") "
Oct 30 21:42:10.008431 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:10.008363    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e-tmp-dir\") pod \"07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e\" (UID: \"07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e\") "
Oct 30 21:42:10.008583 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:10.008431    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e-kube-api-access-88mcq\" (UniqueName: \"kubernetes.io/projected/07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e-kube-api-access-88mcq\") pod \"07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e\" (UID: \"07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e\") "
Oct 30 21:42:10.008583 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:10.008481    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e-dshm\") pod \"07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e\" (UID: \"07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e\") "
Oct 30 21:42:10.008583 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:10.008527    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e-home" pod "07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e" (UID: "07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:42:10.008817 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:10.008558    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e-model-cache\") pod \"07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e\" (UID: \"07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e\") "
Oct 30 21:42:10.008817 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:10.008689    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e-tmp-dir" pod "07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e" (UID: "07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:42:10.008952 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:10.008876    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e-model-cache" pod "07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e" (UID: "07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:42:10.009044 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:10.009026    5713 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e-model-cache\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:42:10.009105 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:10.009050    5713 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e-home\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:42:10.009105 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:10.009065    5713 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e-tmp-dir\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:42:10.010726 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:10.010705    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e-tls-certs" pod "07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e" (UID: "07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:42:10.011152 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:10.011135    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e-dshm" pod "07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e" (UID: "07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:42:10.011220 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:10.011163    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e-kube-api-access-88mcq" pod "07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e" (UID: "07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e"). InnerVolumeSpecName "kube-api-access-88mcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:42:10.110059 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:10.110023    5713 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e-dshm\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:42:10.110059 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:10.110055    5713 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e-tls-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:42:10.110272 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:10.110070    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88mcq\" (UniqueName: \"kubernetes.io/projected/07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e-kube-api-access-88mcq\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:42:10.789578 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:10.789542    5713 generic.go:356] "Generic (PLEG): container finished" podID="07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e" containerID="110d96c3593510de4bf870e85d246d6e1908e758dae1b82ebc00fb3cc006f5f0" exitCode=0
Oct 30 21:42:10.790120 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:10.789589    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2a553e41-kserve-849fbb4cdf-d26cm"
Oct 30 21:42:10.790120 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:10.789632    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2a553e41-kserve-849fbb4cdf-d26cm" event={"ID":"07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e","Type":"ContainerDied","Data":"110d96c3593510de4bf870e85d246d6e1908e758dae1b82ebc00fb3cc006f5f0"}
Oct 30 21:42:10.790120 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:10.789682    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2a553e41-kserve-849fbb4cdf-d26cm" event={"ID":"07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e","Type":"ContainerDied","Data":"0bc9eb4bab8e7c1092f17baa93f62102a2816f57949c7341bbde4dc89edbc31d"}
Oct 30 21:42:10.790120 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:10.789707    5713 scope.go:122] "RemoveContainer" containerID="110d96c3593510de4bf870e85d246d6e1908e758dae1b82ebc00fb3cc006f5f0"
Oct 30 21:42:10.802029 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:10.802010    5713 scope.go:122] "RemoveContainer" containerID="110d96c3593510de4bf870e85d246d6e1908e758dae1b82ebc00fb3cc006f5f0"
Oct 30 21:42:10.802320 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:42:10.802299    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"110d96c3593510de4bf870e85d246d6e1908e758dae1b82ebc00fb3cc006f5f0\": container with ID starting with 110d96c3593510de4bf870e85d246d6e1908e758dae1b82ebc00fb3cc006f5f0 not found: ID does not exist" containerID="110d96c3593510de4bf870e85d246d6e1908e758dae1b82ebc00fb3cc006f5f0"
Oct 30 21:42:10.802398 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:10.802333    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"110d96c3593510de4bf870e85d246d6e1908e758dae1b82ebc00fb3cc006f5f0"} err="failed to get container status \"110d96c3593510de4bf870e85d246d6e1908e758dae1b82ebc00fb3cc006f5f0\": rpc error: code = NotFound desc = could not find container \"110d96c3593510de4bf870e85d246d6e1908e758dae1b82ebc00fb3cc006f5f0\": container with ID starting with 110d96c3593510de4bf870e85d246d6e1908e758dae1b82ebc00fb3cc006f5f0 not found: ID does not exist"
Oct 30 21:42:10.951231 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:10.951197    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-2a553e41-kserve-849fbb4cdf-d26cm"]
Oct 30 21:42:11.001025 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:11.000992    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-2a553e41-kserve-849fbb4cdf-d26cm"]
Oct 30 21:42:11.061577 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:11.061507    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e" path="/var/lib/kubelet/pods/07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e/volumes"
Oct 30 21:42:11.389566 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:11.389487    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xlb8h"]
Oct 30 21:42:11.389959 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:11.389943    5713 cpu_manager.go:414] "containerMap: removing container" podUID="07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e" containerName="main"
Oct 30 21:42:11.390014 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:11.389962    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e" containerName="main"
Oct 30 21:42:11.390088 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:11.390076    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="07f7a4ed-8b7b-4812-a2b2-21d5f902ad1e" containerName="main"
Oct 30 21:42:11.395427 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:11.395403    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xlb8h"
Oct 30 21:42:11.419855 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:11.419829    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z84m4\" (UniqueName: \"kubernetes.io/projected/4dbc5bdf-a04c-4786-8292-53ccd57fec92-kube-api-access-z84m4\") pod \"maas-api-revoke-sub-3e31a506-xlb8h\" (UID: \"4dbc5bdf-a04c-4786-8292-53ccd57fec92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xlb8h"
Oct 30 21:42:11.420027 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:11.419888    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dbc5bdf-a04c-4786-8292-53ccd57fec92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-xlb8h\" (UID: \"4dbc5bdf-a04c-4786-8292-53ccd57fec92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xlb8h"
Oct 30 21:42:11.420027 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:11.419953    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dbc5bdf-a04c-4786-8292-53ccd57fec92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-xlb8h\" (UID: \"4dbc5bdf-a04c-4786-8292-53ccd57fec92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xlb8h"
Oct 30 21:42:11.450882 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:11.450850    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xlb8h"]
Oct 30 21:42:11.451466 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:11.451008    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:42:11.487120 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:11.487086    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:42:11.487120 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:11.487119    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:42:11.487427 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:11.487122    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:42:11.520642 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:11.520602    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z84m4\" (UniqueName: \"kubernetes.io/projected/4dbc5bdf-a04c-4786-8292-53ccd57fec92-kube-api-access-z84m4\") pod \"maas-api-revoke-sub-3e31a506-xlb8h\" (UID: \"4dbc5bdf-a04c-4786-8292-53ccd57fec92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xlb8h"
Oct 30 21:42:11.520810 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:11.520691    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dbc5bdf-a04c-4786-8292-53ccd57fec92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-xlb8h\" (UID: \"4dbc5bdf-a04c-4786-8292-53ccd57fec92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xlb8h"
Oct 30 21:42:11.520810 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:11.520730    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dbc5bdf-a04c-4786-8292-53ccd57fec92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-xlb8h\" (UID: \"4dbc5bdf-a04c-4786-8292-53ccd57fec92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xlb8h"
Oct 30 21:42:11.521399 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:11.521375    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dbc5bdf-a04c-4786-8292-53ccd57fec92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-xlb8h\" (UID: \"4dbc5bdf-a04c-4786-8292-53ccd57fec92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xlb8h"
Oct 30 21:42:11.523097 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:11.523079    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dbc5bdf-a04c-4786-8292-53ccd57fec92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-xlb8h\" (UID: \"4dbc5bdf-a04c-4786-8292-53ccd57fec92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xlb8h"
Oct 30 21:42:11.582639 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:11.582606    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z84m4\" (UniqueName: \"kubernetes.io/projected/4dbc5bdf-a04c-4786-8292-53ccd57fec92-kube-api-access-z84m4\") pod \"maas-api-revoke-sub-3e31a506-xlb8h\" (UID: \"4dbc5bdf-a04c-4786-8292-53ccd57fec92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xlb8h"
Oct 30 21:42:11.713665 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:11.713629    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xlb8h"
Oct 30 21:42:12.088982 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:12.088858    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xlb8h"]
Oct 30 21:42:12.093964 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:42:12.093922    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dbc5bdf_a04c_4786_8292_53ccd57fec92.slice/crio-0014a596b3008ee408c27f37a7ca7c64874848007f2173291ed232d0d28543c0 WatchSource:0}: Error finding container 0014a596b3008ee408c27f37a7ca7c64874848007f2173291ed232d0d28543c0: Status 404 returned error can't find the container with id 0014a596b3008ee408c27f37a7ca7c64874848007f2173291ed232d0d28543c0
Oct 30 21:42:12.813706 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:12.813156    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xlb8h" event={"ID":"4dbc5bdf-a04c-4786-8292-53ccd57fec92","Type":"ContainerStarted","Data":"d06b8cca8eb4a4f538a552f2461253afe57a7bef880c7eb9f9abf8920e032724"}
Oct 30 21:42:12.813706 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:12.813216    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xlb8h" event={"ID":"4dbc5bdf-a04c-4786-8292-53ccd57fec92","Type":"ContainerStarted","Data":"0014a596b3008ee408c27f37a7ca7c64874848007f2173291ed232d0d28543c0"}
Oct 30 21:42:12.931124 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:12.931072    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xlb8h" podStartSLOduration=1.93105652 podStartE2EDuration="1.93105652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:12.929584431 +0000 UTC m=+3815.994360829" watchObservedRunningTime="2026-10-30 21:42:12.93105652 +0000 UTC m=+3815.995832918"
Oct 30 21:42:13.820999 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:13.820948    5713 generic.go:356] "Generic (PLEG): container finished" podID="4dbc5bdf-a04c-4786-8292-53ccd57fec92" containerID="d06b8cca8eb4a4f538a552f2461253afe57a7bef880c7eb9f9abf8920e032724" exitCode=0
Oct 30 21:42:13.821481 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:13.821019    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xlb8h" event={"ID":"4dbc5bdf-a04c-4786-8292-53ccd57fec92","Type":"ContainerDied","Data":"d06b8cca8eb4a4f538a552f2461253afe57a7bef880c7eb9f9abf8920e032724"}
Oct 30 21:42:14.956532 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:14.956507    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xlb8h"
Oct 30 21:42:14.997897 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:14.997872    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4dbc5bdf-a04c-4786-8292-53ccd57fec92-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dbc5bdf-a04c-4786-8292-53ccd57fec92-maas-api-openshift-ca\") pod \"4dbc5bdf-a04c-4786-8292-53ccd57fec92\" (UID: \"4dbc5bdf-a04c-4786-8292-53ccd57fec92\") "
Oct 30 21:42:14.998022 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:14.997910    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dbc5bdf-a04c-4786-8292-53ccd57fec92-kube-api-access-z84m4\" (UniqueName: \"kubernetes.io/projected/4dbc5bdf-a04c-4786-8292-53ccd57fec92-kube-api-access-z84m4\") pod \"4dbc5bdf-a04c-4786-8292-53ccd57fec92\" (UID: \"4dbc5bdf-a04c-4786-8292-53ccd57fec92\") "
Oct 30 21:42:14.998022 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:14.998017    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4dbc5bdf-a04c-4786-8292-53ccd57fec92-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dbc5bdf-a04c-4786-8292-53ccd57fec92-maas-api-xks-ca\") pod \"4dbc5bdf-a04c-4786-8292-53ccd57fec92\" (UID: \"4dbc5bdf-a04c-4786-8292-53ccd57fec92\") "
Oct 30 21:42:14.998264 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:14.998236    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dbc5bdf-a04c-4786-8292-53ccd57fec92-maas-api-openshift-ca" pod "4dbc5bdf-a04c-4786-8292-53ccd57fec92" (UID: "4dbc5bdf-a04c-4786-8292-53ccd57fec92"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:42:15.000219 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:15.000198    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dbc5bdf-a04c-4786-8292-53ccd57fec92-maas-api-xks-ca" pod "4dbc5bdf-a04c-4786-8292-53ccd57fec92" (UID: "4dbc5bdf-a04c-4786-8292-53ccd57fec92"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:42:15.000309 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:15.000279    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dbc5bdf-a04c-4786-8292-53ccd57fec92-kube-api-access-z84m4" pod "4dbc5bdf-a04c-4786-8292-53ccd57fec92" (UID: "4dbc5bdf-a04c-4786-8292-53ccd57fec92"). InnerVolumeSpecName "kube-api-access-z84m4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:42:15.099395 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:15.099329    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dbc5bdf-a04c-4786-8292-53ccd57fec92-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:42:15.099395 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:15.099356    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dbc5bdf-a04c-4786-8292-53ccd57fec92-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:42:15.099395 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:15.099365    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z84m4\" (UniqueName: \"kubernetes.io/projected/4dbc5bdf-a04c-4786-8292-53ccd57fec92-kube-api-access-z84m4\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:42:15.834581 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:15.834549    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xlb8h"
Oct 30 21:42:15.834581 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:15.834572    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xlb8h" event={"ID":"4dbc5bdf-a04c-4786-8292-53ccd57fec92","Type":"ContainerDied","Data":"0014a596b3008ee408c27f37a7ca7c64874848007f2173291ed232d0d28543c0"}
Oct 30 21:42:15.834822 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:15.834621    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0014a596b3008ee408c27f37a7ca7c64874848007f2173291ed232d0d28543c0"
Oct 30 21:42:22.135851 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:22.135776    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-2a553e41-kserve-5d44b5487c-627bp" podUID="af52aa33-0c07-42c8-9d74-f83f31e3fc1a" containerName="main" containerID="cri-o://218f28e7181dd755390f4e8f4f6f34e09d26b9f5fc4847321a2ce9f6f06ae307" gracePeriod=45
Oct 30 21:42:22.383677 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:22.383648    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-2a553e41-kserve-5d44b5487c-627bp"
Oct 30 21:42:22.469548 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:22.469509    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af52aa33-0c07-42c8-9d74-f83f31e3fc1a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af52aa33-0c07-42c8-9d74-f83f31e3fc1a-model-cache\") pod \"af52aa33-0c07-42c8-9d74-f83f31e3fc1a\" (UID: \"af52aa33-0c07-42c8-9d74-f83f31e3fc1a\") "
Oct 30 21:42:22.469548 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:22.469550    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af52aa33-0c07-42c8-9d74-f83f31e3fc1a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af52aa33-0c07-42c8-9d74-f83f31e3fc1a-tmp-dir\") pod \"af52aa33-0c07-42c8-9d74-f83f31e3fc1a\" (UID: \"af52aa33-0c07-42c8-9d74-f83f31e3fc1a\") "
Oct 30 21:42:22.469779 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:22.469617    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af52aa33-0c07-42c8-9d74-f83f31e3fc1a-kube-api-access-w9tpv\" (UniqueName: \"kubernetes.io/projected/af52aa33-0c07-42c8-9d74-f83f31e3fc1a-kube-api-access-w9tpv\") pod \"af52aa33-0c07-42c8-9d74-f83f31e3fc1a\" (UID: \"af52aa33-0c07-42c8-9d74-f83f31e3fc1a\") "
Oct 30 21:42:22.469779 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:22.469699    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af52aa33-0c07-42c8-9d74-f83f31e3fc1a-home\" (UniqueName: \"kubernetes.io/empty-dir/af52aa33-0c07-42c8-9d74-f83f31e3fc1a-home\") pod \"af52aa33-0c07-42c8-9d74-f83f31e3fc1a\" (UID: \"af52aa33-0c07-42c8-9d74-f83f31e3fc1a\") "
Oct 30 21:42:22.469779 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:22.469743    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af52aa33-0c07-42c8-9d74-f83f31e3fc1a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/af52aa33-0c07-42c8-9d74-f83f31e3fc1a-dshm\") pod \"af52aa33-0c07-42c8-9d74-f83f31e3fc1a\" (UID: \"af52aa33-0c07-42c8-9d74-f83f31e3fc1a\") "
Oct 30 21:42:22.470083 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:22.469862    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af52aa33-0c07-42c8-9d74-f83f31e3fc1a-tls-certs\" (UniqueName: \"kubernetes.io/secret/af52aa33-0c07-42c8-9d74-f83f31e3fc1a-tls-certs\") pod \"af52aa33-0c07-42c8-9d74-f83f31e3fc1a\" (UID: \"af52aa33-0c07-42c8-9d74-f83f31e3fc1a\") "
Oct 30 21:42:22.470083 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:22.469881    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af52aa33-0c07-42c8-9d74-f83f31e3fc1a-model-cache" pod "af52aa33-0c07-42c8-9d74-f83f31e3fc1a" (UID: "af52aa33-0c07-42c8-9d74-f83f31e3fc1a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:42:22.470083 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:22.469903    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af52aa33-0c07-42c8-9d74-f83f31e3fc1a-tmp-dir" pod "af52aa33-0c07-42c8-9d74-f83f31e3fc1a" (UID: "af52aa33-0c07-42c8-9d74-f83f31e3fc1a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:42:22.470083 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:22.470044    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af52aa33-0c07-42c8-9d74-f83f31e3fc1a-home" pod "af52aa33-0c07-42c8-9d74-f83f31e3fc1a" (UID: "af52aa33-0c07-42c8-9d74-f83f31e3fc1a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:42:22.470305 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:22.470284    5713 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af52aa33-0c07-42c8-9d74-f83f31e3fc1a-home\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:42:22.470362 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:22.470310    5713 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af52aa33-0c07-42c8-9d74-f83f31e3fc1a-model-cache\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:42:22.470362 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:22.470320    5713 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af52aa33-0c07-42c8-9d74-f83f31e3fc1a-tmp-dir\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:42:22.472098 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:22.472072    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af52aa33-0c07-42c8-9d74-f83f31e3fc1a-dshm" pod "af52aa33-0c07-42c8-9d74-f83f31e3fc1a" (UID: "af52aa33-0c07-42c8-9d74-f83f31e3fc1a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:42:22.472228 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:22.472133    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af52aa33-0c07-42c8-9d74-f83f31e3fc1a-tls-certs" pod "af52aa33-0c07-42c8-9d74-f83f31e3fc1a" (UID: "af52aa33-0c07-42c8-9d74-f83f31e3fc1a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:42:22.472228 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:22.472186    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af52aa33-0c07-42c8-9d74-f83f31e3fc1a-kube-api-access-w9tpv" pod "af52aa33-0c07-42c8-9d74-f83f31e3fc1a" (UID: "af52aa33-0c07-42c8-9d74-f83f31e3fc1a"). InnerVolumeSpecName "kube-api-access-w9tpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:42:22.571246 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:22.571195    5713 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af52aa33-0c07-42c8-9d74-f83f31e3fc1a-dshm\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:42:22.571246 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:22.571240    5713 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af52aa33-0c07-42c8-9d74-f83f31e3fc1a-tls-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:42:22.571246 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:22.571251    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9tpv\" (UniqueName: \"kubernetes.io/projected/af52aa33-0c07-42c8-9d74-f83f31e3fc1a-kube-api-access-w9tpv\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:42:22.872999 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:22.872912    5713 generic.go:356] "Generic (PLEG): container finished" podID="af52aa33-0c07-42c8-9d74-f83f31e3fc1a" containerID="218f28e7181dd755390f4e8f4f6f34e09d26b9f5fc4847321a2ce9f6f06ae307" exitCode=0
Oct 30 21:42:22.872999 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:22.872963    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-2a553e41-kserve-5d44b5487c-627bp" event={"ID":"af52aa33-0c07-42c8-9d74-f83f31e3fc1a","Type":"ContainerDied","Data":"218f28e7181dd755390f4e8f4f6f34e09d26b9f5fc4847321a2ce9f6f06ae307"}
Oct 30 21:42:22.872999 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:22.872982    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-2a553e41-kserve-5d44b5487c-627bp"
Oct 30 21:42:22.873260 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:22.873006    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-2a553e41-kserve-5d44b5487c-627bp" event={"ID":"af52aa33-0c07-42c8-9d74-f83f31e3fc1a","Type":"ContainerDied","Data":"767d11d89b28a6f7ffcd8ec95d420551dd400862e6027bf0587074b8111dc200"}
Oct 30 21:42:22.873260 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:22.873028    5713 scope.go:122] "RemoveContainer" containerID="218f28e7181dd755390f4e8f4f6f34e09d26b9f5fc4847321a2ce9f6f06ae307"
Oct 30 21:42:22.883039 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:22.883018    5713 scope.go:122] "RemoveContainer" containerID="218f28e7181dd755390f4e8f4f6f34e09d26b9f5fc4847321a2ce9f6f06ae307"
Oct 30 21:42:22.883296 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:42:22.883275    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"218f28e7181dd755390f4e8f4f6f34e09d26b9f5fc4847321a2ce9f6f06ae307\": container with ID starting with 218f28e7181dd755390f4e8f4f6f34e09d26b9f5fc4847321a2ce9f6f06ae307 not found: ID does not exist" containerID="218f28e7181dd755390f4e8f4f6f34e09d26b9f5fc4847321a2ce9f6f06ae307"
Oct 30 21:42:22.883361 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:22.883309    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"218f28e7181dd755390f4e8f4f6f34e09d26b9f5fc4847321a2ce9f6f06ae307"} err="failed to get container status \"218f28e7181dd755390f4e8f4f6f34e09d26b9f5fc4847321a2ce9f6f06ae307\": rpc error: code = NotFound desc = could not find container \"218f28e7181dd755390f4e8f4f6f34e09d26b9f5fc4847321a2ce9f6f06ae307\": container with ID starting with 218f28e7181dd755390f4e8f4f6f34e09d26b9f5fc4847321a2ce9f6f06ae307 not found: ID does not exist"
Oct 30 21:42:23.036591 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:23.036563    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-2a553e41-kserve-5d44b5487c-627bp"]
Oct 30 21:42:23.086152 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:23.086118    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-2a553e41-kserve-5d44b5487c-627bp"]
Oct 30 21:42:24.458523 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:42:24.458484    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:42:25.095989 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:25.093360    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af52aa33-0c07-42c8-9d74-f83f31e3fc1a" path="/var/lib/kubelet/pods/af52aa33-0c07-42c8-9d74-f83f31e3fc1a/volumes"
Oct 30 21:42:40.577252 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:40.577212    5713 scope.go:122] "RemoveContainer" containerID="ec2c7a76297de4069d750ed88598b9c1caf6b2875a7f7a4f06dfb67b2e1d0c3f"
Oct 30 21:42:45.178991 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:45.178960    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfe4e0bf-wxkhd"]
Oct 30 21:42:45.179437 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:45.179411    5713 cpu_manager.go:414] "containerMap: removing container" podUID="4dbc5bdf-a04c-4786-8292-53ccd57fec92" containerName="revoke-keys"
Oct 30 21:42:45.179437 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:45.179423    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dbc5bdf-a04c-4786-8292-53ccd57fec92" containerName="revoke-keys"
Oct 30 21:42:45.179527 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:45.179462    5713 cpu_manager.go:414] "containerMap: removing container" podUID="af52aa33-0c07-42c8-9d74-f83f31e3fc1a" containerName="main"
Oct 30 21:42:45.179527 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:45.179469    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af52aa33-0c07-42c8-9d74-f83f31e3fc1a" containerName="main"
Oct 30 21:42:45.179608 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:45.179538    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="af52aa33-0c07-42c8-9d74-f83f31e3fc1a" containerName="main"
Oct 30 21:42:45.179608 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:45.179550    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dbc5bdf-a04c-4786-8292-53ccd57fec92" containerName="revoke-keys"
Oct 30 21:42:45.182647 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:45.182631    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe4e0bf-wxkhd"
Oct 30 21:42:45.239100 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:45.239061    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfe4e0bf-wxkhd"]
Oct 30 21:42:45.241401 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:45.241368    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wgv7\" (UniqueName: \"kubernetes.io/projected/2bf696ed-fdbc-4776-8527-7fc89ca2b35b-kube-api-access-5wgv7\") pod \"maas-api-revoke-sub-dfe4e0bf-wxkhd\" (UID: \"2bf696ed-fdbc-4776-8527-7fc89ca2b35b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe4e0bf-wxkhd"
Oct 30 21:42:45.241531 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:45.241509    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bf696ed-fdbc-4776-8527-7fc89ca2b35b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dfe4e0bf-wxkhd\" (UID: \"2bf696ed-fdbc-4776-8527-7fc89ca2b35b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe4e0bf-wxkhd"
Oct 30 21:42:45.241576 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:45.241563    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bf696ed-fdbc-4776-8527-7fc89ca2b35b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dfe4e0bf-wxkhd\" (UID: \"2bf696ed-fdbc-4776-8527-7fc89ca2b35b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe4e0bf-wxkhd"
Oct 30 21:42:45.285645 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:45.285613    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:42:45.285841 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:45.285624    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:42:45.285841 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:45.285683    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:42:45.285841 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:45.285730    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q2mgc\""
Oct 30 21:42:45.342875 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:45.342850    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bf696ed-fdbc-4776-8527-7fc89ca2b35b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dfe4e0bf-wxkhd\" (UID: \"2bf696ed-fdbc-4776-8527-7fc89ca2b35b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe4e0bf-wxkhd"
Oct 30 21:42:45.343000 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:45.342885    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bf696ed-fdbc-4776-8527-7fc89ca2b35b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dfe4e0bf-wxkhd\" (UID: \"2bf696ed-fdbc-4776-8527-7fc89ca2b35b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe4e0bf-wxkhd"
Oct 30 21:42:45.343000 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:45.342963    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wgv7\" (UniqueName: \"kubernetes.io/projected/2bf696ed-fdbc-4776-8527-7fc89ca2b35b-kube-api-access-5wgv7\") pod \"maas-api-revoke-sub-dfe4e0bf-wxkhd\" (UID: \"2bf696ed-fdbc-4776-8527-7fc89ca2b35b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe4e0bf-wxkhd"
Oct 30 21:42:45.343577 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:45.343558    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bf696ed-fdbc-4776-8527-7fc89ca2b35b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dfe4e0bf-wxkhd\" (UID: \"2bf696ed-fdbc-4776-8527-7fc89ca2b35b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe4e0bf-wxkhd"
Oct 30 21:42:45.345852 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:45.345830    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bf696ed-fdbc-4776-8527-7fc89ca2b35b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dfe4e0bf-wxkhd\" (UID: \"2bf696ed-fdbc-4776-8527-7fc89ca2b35b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe4e0bf-wxkhd"
Oct 30 21:42:45.400048 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:45.400018    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wgv7\" (UniqueName: \"kubernetes.io/projected/2bf696ed-fdbc-4776-8527-7fc89ca2b35b-kube-api-access-5wgv7\") pod \"maas-api-revoke-sub-dfe4e0bf-wxkhd\" (UID: \"2bf696ed-fdbc-4776-8527-7fc89ca2b35b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe4e0bf-wxkhd"
Oct 30 21:42:45.495447 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:45.495421    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe4e0bf-wxkhd"
Oct 30 21:42:45.874684 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:45.874658    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfe4e0bf-wxkhd"]
Oct 30 21:42:45.876962 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:42:45.876928    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bf696ed_fdbc_4776_8527_7fc89ca2b35b.slice/crio-08b5ab2ad7f72d9155a948bb116ec0305a3fdb98e7f3e632709580fc54a532cb WatchSource:0}: Error finding container 08b5ab2ad7f72d9155a948bb116ec0305a3fdb98e7f3e632709580fc54a532cb: Status 404 returned error can't find the container with id 08b5ab2ad7f72d9155a948bb116ec0305a3fdb98e7f3e632709580fc54a532cb
Oct 30 21:42:46.060551 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:46.060523    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe4e0bf-wxkhd" event={"ID":"2bf696ed-fdbc-4776-8527-7fc89ca2b35b","Type":"ContainerStarted","Data":"08b5ab2ad7f72d9155a948bb116ec0305a3fdb98e7f3e632709580fc54a532cb"}
Oct 30 21:42:47.066682 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:47.066591    5713 generic.go:356] "Generic (PLEG): container finished" podID="2bf696ed-fdbc-4776-8527-7fc89ca2b35b" containerID="a7962d533bd7042abc78f0fa61f59bd14def8b8401de6c0c4f7d37acd633fedd" exitCode=0
Oct 30 21:42:47.066682 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:47.066666    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe4e0bf-wxkhd" event={"ID":"2bf696ed-fdbc-4776-8527-7fc89ca2b35b","Type":"ContainerDied","Data":"a7962d533bd7042abc78f0fa61f59bd14def8b8401de6c0c4f7d37acd633fedd"}
Oct 30 21:42:48.216637 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:48.216611    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe4e0bf-wxkhd"
Oct 30 21:42:48.373605 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:48.373521    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bf696ed-fdbc-4776-8527-7fc89ca2b35b-kube-api-access-5wgv7\" (UniqueName: \"kubernetes.io/projected/2bf696ed-fdbc-4776-8527-7fc89ca2b35b-kube-api-access-5wgv7\") pod \"2bf696ed-fdbc-4776-8527-7fc89ca2b35b\" (UID: \"2bf696ed-fdbc-4776-8527-7fc89ca2b35b\") "
Oct 30 21:42:48.373605 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:48.373572    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bf696ed-fdbc-4776-8527-7fc89ca2b35b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bf696ed-fdbc-4776-8527-7fc89ca2b35b-maas-api-openshift-ca\") pod \"2bf696ed-fdbc-4776-8527-7fc89ca2b35b\" (UID: \"2bf696ed-fdbc-4776-8527-7fc89ca2b35b\") "
Oct 30 21:42:48.373900 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:48.373614    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bf696ed-fdbc-4776-8527-7fc89ca2b35b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bf696ed-fdbc-4776-8527-7fc89ca2b35b-maas-api-xks-ca\") pod \"2bf696ed-fdbc-4776-8527-7fc89ca2b35b\" (UID: \"2bf696ed-fdbc-4776-8527-7fc89ca2b35b\") "
Oct 30 21:42:48.374044 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:48.374019    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bf696ed-fdbc-4776-8527-7fc89ca2b35b-maas-api-openshift-ca" pod "2bf696ed-fdbc-4776-8527-7fc89ca2b35b" (UID: "2bf696ed-fdbc-4776-8527-7fc89ca2b35b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:42:48.375907 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:48.375880    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bf696ed-fdbc-4776-8527-7fc89ca2b35b-maas-api-xks-ca" pod "2bf696ed-fdbc-4776-8527-7fc89ca2b35b" (UID: "2bf696ed-fdbc-4776-8527-7fc89ca2b35b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:42:48.375907 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:48.375892    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bf696ed-fdbc-4776-8527-7fc89ca2b35b-kube-api-access-5wgv7" pod "2bf696ed-fdbc-4776-8527-7fc89ca2b35b" (UID: "2bf696ed-fdbc-4776-8527-7fc89ca2b35b"). InnerVolumeSpecName "kube-api-access-5wgv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:42:48.474838 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:48.474805    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wgv7\" (UniqueName: \"kubernetes.io/projected/2bf696ed-fdbc-4776-8527-7fc89ca2b35b-kube-api-access-5wgv7\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:42:48.474838 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:48.474836    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bf696ed-fdbc-4776-8527-7fc89ca2b35b-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:42:48.474838 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:48.474846    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bf696ed-fdbc-4776-8527-7fc89ca2b35b-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:42:49.079623 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:49.079593    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe4e0bf-wxkhd"
Oct 30 21:42:49.079809 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:49.079595    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe4e0bf-wxkhd" event={"ID":"2bf696ed-fdbc-4776-8527-7fc89ca2b35b","Type":"ContainerDied","Data":"08b5ab2ad7f72d9155a948bb116ec0305a3fdb98e7f3e632709580fc54a532cb"}
Oct 30 21:42:49.079809 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:42:49.079718    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08b5ab2ad7f72d9155a948bb116ec0305a3fdb98e7f3e632709580fc54a532cb"
Oct 30 21:42:54.457894 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:42:54.457858    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:43:05.906350 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:05.906314    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9"]
Oct 30 21:43:05.906764 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:05.906733    5713 cpu_manager.go:414] "containerMap: removing container" podUID="2bf696ed-fdbc-4776-8527-7fc89ca2b35b" containerName="revoke-keys"
Oct 30 21:43:05.906764 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:05.906743    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bf696ed-fdbc-4776-8527-7fc89ca2b35b" containerName="revoke-keys"
Oct 30 21:43:05.906897 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:05.906822    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bf696ed-fdbc-4776-8527-7fc89ca2b35b" containerName="revoke-keys"
Oct 30 21:43:05.910199 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:05.910181    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9"
Oct 30 21:43:05.946554 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:05.946518    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d406ea5-aa2d-4a1d-a010-65959654939a-istiod-ca-cert\") pod \"e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9\" (UID: \"1d406ea5-aa2d-4a1d-a010-65959654939a\") " pod="openshift-ingress/e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9"
Oct 30 21:43:05.946726 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:05.946614    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d406ea5-aa2d-4a1d-a010-65959654939a-istio-podinfo\") pod \"e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9\" (UID: \"1d406ea5-aa2d-4a1d-a010-65959654939a\") " pod="openshift-ingress/e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9"
Oct 30 21:43:05.946840 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:05.946775    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d406ea5-aa2d-4a1d-a010-65959654939a-credential-socket\") pod \"e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9\" (UID: \"1d406ea5-aa2d-4a1d-a010-65959654939a\") " pod="openshift-ingress/e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9"
Oct 30 21:43:05.946915 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:05.946875    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d406ea5-aa2d-4a1d-a010-65959654939a-workload-socket\") pod \"e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9\" (UID: \"1d406ea5-aa2d-4a1d-a010-65959654939a\") " pod="openshift-ingress/e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9"
Oct 30 21:43:05.946983 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:05.946959    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d406ea5-aa2d-4a1d-a010-65959654939a-workload-certs\") pod \"e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9\" (UID: \"1d406ea5-aa2d-4a1d-a010-65959654939a\") " pod="openshift-ingress/e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9"
Oct 30 21:43:05.947043 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:05.946985    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d406ea5-aa2d-4a1d-a010-65959654939a-istio-envoy\") pod \"e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9\" (UID: \"1d406ea5-aa2d-4a1d-a010-65959654939a\") " pod="openshift-ingress/e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9"
Oct 30 21:43:05.947043 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:05.947011    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d406ea5-aa2d-4a1d-a010-65959654939a-istio-token\") pod \"e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9\" (UID: \"1d406ea5-aa2d-4a1d-a010-65959654939a\") " pod="openshift-ingress/e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9"
Oct 30 21:43:05.947164 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:05.947066    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d406ea5-aa2d-4a1d-a010-65959654939a-istio-data\") pod \"e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9\" (UID: \"1d406ea5-aa2d-4a1d-a010-65959654939a\") " pod="openshift-ingress/e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9"
Oct 30 21:43:05.947164 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:05.947096    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx7cf\" (UniqueName: \"kubernetes.io/projected/1d406ea5-aa2d-4a1d-a010-65959654939a-kube-api-access-tx7cf\") pod \"e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9\" (UID: \"1d406ea5-aa2d-4a1d-a010-65959654939a\") " pod="openshift-ingress/e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9"
Oct 30 21:43:05.962547 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:05.962521    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-1a1f4ae2-openshift-default-dockercfg-xznhj\""
Oct 30 21:43:06.016024 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:06.015988    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9"]
Oct 30 21:43:06.047778 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:06.047739    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d406ea5-aa2d-4a1d-a010-65959654939a-workload-socket\") pod \"e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9\" (UID: \"1d406ea5-aa2d-4a1d-a010-65959654939a\") " pod="openshift-ingress/e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9"
Oct 30 21:43:06.048022 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:06.047854    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d406ea5-aa2d-4a1d-a010-65959654939a-workload-certs\") pod \"e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9\" (UID: \"1d406ea5-aa2d-4a1d-a010-65959654939a\") " pod="openshift-ingress/e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9"
Oct 30 21:43:06.048022 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:06.047873    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d406ea5-aa2d-4a1d-a010-65959654939a-istio-envoy\") pod \"e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9\" (UID: \"1d406ea5-aa2d-4a1d-a010-65959654939a\") " pod="openshift-ingress/e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9"
Oct 30 21:43:06.048022 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:06.047894    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d406ea5-aa2d-4a1d-a010-65959654939a-istio-token\") pod \"e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9\" (UID: \"1d406ea5-aa2d-4a1d-a010-65959654939a\") " pod="openshift-ingress/e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9"
Oct 30 21:43:06.048022 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:06.047944    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d406ea5-aa2d-4a1d-a010-65959654939a-istio-data\") pod \"e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9\" (UID: \"1d406ea5-aa2d-4a1d-a010-65959654939a\") " pod="openshift-ingress/e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9"
Oct 30 21:43:06.048022 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:06.047972    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx7cf\" (UniqueName: \"kubernetes.io/projected/1d406ea5-aa2d-4a1d-a010-65959654939a-kube-api-access-tx7cf\") pod \"e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9\" (UID: \"1d406ea5-aa2d-4a1d-a010-65959654939a\") " pod="openshift-ingress/e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9"
Oct 30 21:43:06.049283 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:06.048065    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d406ea5-aa2d-4a1d-a010-65959654939a-istiod-ca-cert\") pod \"e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9\" (UID: \"1d406ea5-aa2d-4a1d-a010-65959654939a\") " pod="openshift-ingress/e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9"
Oct 30 21:43:06.049283 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:06.048104    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d406ea5-aa2d-4a1d-a010-65959654939a-istio-podinfo\") pod \"e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9\" (UID: \"1d406ea5-aa2d-4a1d-a010-65959654939a\") " pod="openshift-ingress/e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9"
Oct 30 21:43:06.049283 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:06.048262    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d406ea5-aa2d-4a1d-a010-65959654939a-credential-socket\") pod \"e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9\" (UID: \"1d406ea5-aa2d-4a1d-a010-65959654939a\") " pod="openshift-ingress/e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9"
Oct 30 21:43:06.049283 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:06.048265    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d406ea5-aa2d-4a1d-a010-65959654939a-workload-socket\") pod \"e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9\" (UID: \"1d406ea5-aa2d-4a1d-a010-65959654939a\") " pod="openshift-ingress/e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9"
Oct 30 21:43:06.049283 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:06.048875    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d406ea5-aa2d-4a1d-a010-65959654939a-credential-socket\") pod \"e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9\" (UID: \"1d406ea5-aa2d-4a1d-a010-65959654939a\") " pod="openshift-ingress/e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9"
Oct 30 21:43:06.049862 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:06.049844    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d406ea5-aa2d-4a1d-a010-65959654939a-istiod-ca-cert\") pod \"e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9\" (UID: \"1d406ea5-aa2d-4a1d-a010-65959654939a\") " pod="openshift-ingress/e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9"
Oct 30 21:43:06.050103 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:06.050065    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d406ea5-aa2d-4a1d-a010-65959654939a-istio-data\") pod \"e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9\" (UID: \"1d406ea5-aa2d-4a1d-a010-65959654939a\") " pod="openshift-ingress/e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9"
Oct 30 21:43:06.051298 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:06.051268    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d406ea5-aa2d-4a1d-a010-65959654939a-workload-certs\") pod \"e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9\" (UID: \"1d406ea5-aa2d-4a1d-a010-65959654939a\") " pod="openshift-ingress/e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9"
Oct 30 21:43:06.053908 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:06.053884    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d406ea5-aa2d-4a1d-a010-65959654939a-istio-envoy\") pod \"e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9\" (UID: \"1d406ea5-aa2d-4a1d-a010-65959654939a\") " pod="openshift-ingress/e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9"
Oct 30 21:43:06.056819 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:06.054915    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d406ea5-aa2d-4a1d-a010-65959654939a-istio-podinfo\") pod \"e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9\" (UID: \"1d406ea5-aa2d-4a1d-a010-65959654939a\") " pod="openshift-ingress/e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9"
Oct 30 21:43:06.106470 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:06.106435    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx7cf\" (UniqueName: \"kubernetes.io/projected/1d406ea5-aa2d-4a1d-a010-65959654939a-kube-api-access-tx7cf\") pod \"e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9\" (UID: \"1d406ea5-aa2d-4a1d-a010-65959654939a\") " pod="openshift-ingress/e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9"
Oct 30 21:43:06.106939 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:06.106921    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d406ea5-aa2d-4a1d-a010-65959654939a-istio-token\") pod \"e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9\" (UID: \"1d406ea5-aa2d-4a1d-a010-65959654939a\") " pod="openshift-ingress/e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9"
Oct 30 21:43:06.223297 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:06.223268    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9"
Oct 30 21:43:06.652011 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:06.651982    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9"]
Oct 30 21:43:06.654256 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:43:06.654223    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d406ea5_aa2d_4a1d_a010_65959654939a.slice/crio-e162c68b6c1ab50a7b2b0cbca33418889268352c7c0e56214fc3f4f8b68e217f WatchSource:0}: Error finding container e162c68b6c1ab50a7b2b0cbca33418889268352c7c0e56214fc3f4f8b68e217f: Status 404 returned error can't find the container with id e162c68b6c1ab50a7b2b0cbca33418889268352c7c0e56214fc3f4f8b68e217f
Oct 30 21:43:06.656437 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:06.656404    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:43:06.656513 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:06.656473    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:43:06.656513 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:06.656501    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:43:07.182452 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:07.182415    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9" event={"ID":"1d406ea5-aa2d-4a1d-a010-65959654939a","Type":"ContainerStarted","Data":"a90b4314fd34a3638df0ed35ef57d111c0899228bdca43b87be32c99aeaff6b1"}
Oct 30 21:43:07.182885 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:07.182467    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9" event={"ID":"1d406ea5-aa2d-4a1d-a010-65959654939a","Type":"ContainerStarted","Data":"e162c68b6c1ab50a7b2b0cbca33418889268352c7c0e56214fc3f4f8b68e217f"}
Oct 30 21:43:07.224038 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:07.224003    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9"
Oct 30 21:43:07.229287 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:07.229260    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9"
Oct 30 21:43:07.302111 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:07.302059    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9" podStartSLOduration=2.302045629 podStartE2EDuration="2.302045629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:07.300514253 +0000 UTC m=+3870.365290654" watchObservedRunningTime="2026-10-30 21:43:07.302045629 +0000 UTC m=+3870.366822027"
Oct 30 21:43:08.188233 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:08.188202    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9"
Oct 30 21:43:08.189264 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:08.189245    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9"
Oct 30 21:43:13.118128 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:13.118085    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1a1f4ae2-58bf5c4759-s559l"]
Oct 30 21:43:13.121803 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:13.121758    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1a1f4ae2-58bf5c4759-s559l"
Oct 30 21:43:13.179709 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:13.179678    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1a1f4ae2-58bf5c4759-s559l"]
Oct 30 21:43:13.229829 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:13.221119    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c83aa65-8754-445c-b6f1-260dce60ae6c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1a1f4ae2-58bf5c4759-s559l\" (UID: \"8c83aa65-8754-445c-b6f1-260dce60ae6c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1a1f4ae2-58bf5c4759-s559l"
Oct 30 21:43:13.229829 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:13.221260    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dll4t\" (UniqueName: \"kubernetes.io/projected/8c83aa65-8754-445c-b6f1-260dce60ae6c-kube-api-access-dll4t\") pod \"payload-pre-processing-e2e-ait-1a1f4ae2-58bf5c4759-s559l\" (UID: \"8c83aa65-8754-445c-b6f1-260dce60ae6c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1a1f4ae2-58bf5c4759-s559l"
Oct 30 21:43:13.229829 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:13.226633    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-1a1f4ae2\""
Oct 30 21:43:13.229829 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:13.226654    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-1a1f4ae2-dockercfg-6kmbc\""
Oct 30 21:43:13.306082 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:13.306048    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1a1f4ae2-5c86778f44-mhzh2"]
Oct 30 21:43:13.309598 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:13.309580    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1a1f4ae2-5c86778f44-mhzh2"
Oct 30 21:43:13.321742 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:13.321716    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c83aa65-8754-445c-b6f1-260dce60ae6c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1a1f4ae2-58bf5c4759-s559l\" (UID: \"8c83aa65-8754-445c-b6f1-260dce60ae6c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1a1f4ae2-58bf5c4759-s559l"
Oct 30 21:43:13.321898 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:13.321858    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dll4t\" (UniqueName: \"kubernetes.io/projected/8c83aa65-8754-445c-b6f1-260dce60ae6c-kube-api-access-dll4t\") pod \"payload-pre-processing-e2e-ait-1a1f4ae2-58bf5c4759-s559l\" (UID: \"8c83aa65-8754-445c-b6f1-260dce60ae6c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1a1f4ae2-58bf5c4759-s559l"
Oct 30 21:43:13.322356 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:13.322338    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c83aa65-8754-445c-b6f1-260dce60ae6c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1a1f4ae2-58bf5c4759-s559l\" (UID: \"8c83aa65-8754-445c-b6f1-260dce60ae6c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1a1f4ae2-58bf5c4759-s559l"
Oct 30 21:43:13.378255 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:13.378164    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1a1f4ae2-5c86778f44-mhzh2"]
Oct 30 21:43:13.382660 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:13.382623    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dll4t\" (UniqueName: \"kubernetes.io/projected/8c83aa65-8754-445c-b6f1-260dce60ae6c-kube-api-access-dll4t\") pod \"payload-pre-processing-e2e-ait-1a1f4ae2-58bf5c4759-s559l\" (UID: \"8c83aa65-8754-445c-b6f1-260dce60ae6c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1a1f4ae2-58bf5c4759-s559l"
Oct 30 21:43:13.423132 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:13.423101    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65806364-2b75-48a3-8d85-9ef7bc8f9074-plugins-config-volume\") pod \"payload-processing-e2e-ait-1a1f4ae2-5c86778f44-mhzh2\" (UID: \"65806364-2b75-48a3-8d85-9ef7bc8f9074\") " pod="openshift-ingress/payload-processing-e2e-ait-1a1f4ae2-5c86778f44-mhzh2"
Oct 30 21:43:13.423324 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:13.423168    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kl54\" (UniqueName: \"kubernetes.io/projected/65806364-2b75-48a3-8d85-9ef7bc8f9074-kube-api-access-4kl54\") pod \"payload-processing-e2e-ait-1a1f4ae2-5c86778f44-mhzh2\" (UID: \"65806364-2b75-48a3-8d85-9ef7bc8f9074\") " pod="openshift-ingress/payload-processing-e2e-ait-1a1f4ae2-5c86778f44-mhzh2"
Oct 30 21:43:13.434946 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:13.434915    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1a1f4ae2-58bf5c4759-s559l"
Oct 30 21:43:13.524109 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:13.524064    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kl54\" (UniqueName: \"kubernetes.io/projected/65806364-2b75-48a3-8d85-9ef7bc8f9074-kube-api-access-4kl54\") pod \"payload-processing-e2e-ait-1a1f4ae2-5c86778f44-mhzh2\" (UID: \"65806364-2b75-48a3-8d85-9ef7bc8f9074\") " pod="openshift-ingress/payload-processing-e2e-ait-1a1f4ae2-5c86778f44-mhzh2"
Oct 30 21:43:13.524267 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:13.524198    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65806364-2b75-48a3-8d85-9ef7bc8f9074-plugins-config-volume\") pod \"payload-processing-e2e-ait-1a1f4ae2-5c86778f44-mhzh2\" (UID: \"65806364-2b75-48a3-8d85-9ef7bc8f9074\") " pod="openshift-ingress/payload-processing-e2e-ait-1a1f4ae2-5c86778f44-mhzh2"
Oct 30 21:43:13.524966 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:13.524938    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65806364-2b75-48a3-8d85-9ef7bc8f9074-plugins-config-volume\") pod \"payload-processing-e2e-ait-1a1f4ae2-5c86778f44-mhzh2\" (UID: \"65806364-2b75-48a3-8d85-9ef7bc8f9074\") " pod="openshift-ingress/payload-processing-e2e-ait-1a1f4ae2-5c86778f44-mhzh2"
Oct 30 21:43:13.582149 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:13.582116    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kl54\" (UniqueName: \"kubernetes.io/projected/65806364-2b75-48a3-8d85-9ef7bc8f9074-kube-api-access-4kl54\") pod \"payload-processing-e2e-ait-1a1f4ae2-5c86778f44-mhzh2\" (UID: \"65806364-2b75-48a3-8d85-9ef7bc8f9074\") " pod="openshift-ingress/payload-processing-e2e-ait-1a1f4ae2-5c86778f44-mhzh2"
Oct 30 21:43:13.622157 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:13.622128    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1a1f4ae2-5c86778f44-mhzh2"
Oct 30 21:43:13.810288 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:13.810259    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1a1f4ae2-58bf5c4759-s559l"]
Oct 30 21:43:13.811948 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:43:13.811896    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c83aa65_8754_445c_b6f1_260dce60ae6c.slice/crio-fe8148ea42bddca9260aa7c868f9b47718fbb5fca93269cf6239a4fce09e6438 WatchSource:0}: Error finding container fe8148ea42bddca9260aa7c868f9b47718fbb5fca93269cf6239a4fce09e6438: Status 404 returned error can't find the container with id fe8148ea42bddca9260aa7c868f9b47718fbb5fca93269cf6239a4fce09e6438
Oct 30 21:43:13.996239 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:13.996212    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1a1f4ae2-5c86778f44-mhzh2"]
Oct 30 21:43:13.998846 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:43:13.998812    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65806364_2b75_48a3_8d85_9ef7bc8f9074.slice/crio-b8accc72caca68688491362efeffad6751be716b101389993f60305a4e8addb5 WatchSource:0}: Error finding container b8accc72caca68688491362efeffad6751be716b101389993f60305a4e8addb5: Status 404 returned error can't find the container with id b8accc72caca68688491362efeffad6751be716b101389993f60305a4e8addb5
Oct 30 21:43:14.234467 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:14.234429    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1a1f4ae2-5c86778f44-mhzh2" event={"ID":"65806364-2b75-48a3-8d85-9ef7bc8f9074","Type":"ContainerStarted","Data":"199f8040857fa08e4706957a5b2245582857a1771d70c22fb657213bd0a435a1"}
Oct 30 21:43:14.234953 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:14.234497    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-1a1f4ae2-5c86778f44-mhzh2"
Oct 30 21:43:14.234953 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:14.234524    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1a1f4ae2-5c86778f44-mhzh2" event={"ID":"65806364-2b75-48a3-8d85-9ef7bc8f9074","Type":"ContainerStarted","Data":"b8accc72caca68688491362efeffad6751be716b101389993f60305a4e8addb5"}
Oct 30 21:43:14.236098 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:14.236064    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1a1f4ae2-58bf5c4759-s559l" event={"ID":"8c83aa65-8754-445c-b6f1-260dce60ae6c","Type":"ContainerStarted","Data":"e7574c80b9dfd49afd7fa7e429392f25be24154d33f9d903fe63773e7e9991d6"}
Oct 30 21:43:14.236214 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:14.236118    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1a1f4ae2-58bf5c4759-s559l" event={"ID":"8c83aa65-8754-445c-b6f1-260dce60ae6c","Type":"ContainerStarted","Data":"fe8148ea42bddca9260aa7c868f9b47718fbb5fca93269cf6239a4fce09e6438"}
Oct 30 21:43:14.236214 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:14.236157    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1a1f4ae2-58bf5c4759-s559l"
Oct 30 21:43:14.350560 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:14.350451    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-1a1f4ae2-5c86778f44-mhzh2" podStartSLOduration=1.350433284 podStartE2EDuration="1.350433284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:14.349328714 +0000 UTC m=+3877.414105112" watchObservedRunningTime="2026-10-30 21:43:14.350433284 +0000 UTC m=+3877.415209725"
Oct 30 21:43:14.471518 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:14.471450    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-1a1f4ae2-58bf5c4759-s559l" podStartSLOduration=1.471430943 podStartE2EDuration="1.471430943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:14.467719144 +0000 UTC m=+3877.532495565" watchObservedRunningTime="2026-10-30 21:43:14.471430943 +0000 UTC m=+3877.536207342"
Oct 30 21:43:18.046569 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:18.046482    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4z97z"]
Oct 30 21:43:18.096651 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:18.096621    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4z97z"]
Oct 30 21:43:19.062995 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:19.062963    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="580dd8f4-d053-4878-b779-675505987f89" path="/var/lib/kubelet/pods/580dd8f4-d053-4878-b779-675505987f89/volumes"
Oct 30 21:43:24.482889 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:43:24.482852    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:43:25.259970 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:25.259940    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-1a1f4ae2-5c86778f44-mhzh2"
Oct 30 21:43:25.260269 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:25.260017    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1a1f4ae2-58bf5c4759-s559l"
Oct 30 21:43:32.340177 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:32.340141    5713 generic.go:356] "Generic (PLEG): container finished" podID="1d406ea5-aa2d-4a1d-a010-65959654939a" containerID="a90b4314fd34a3638df0ed35ef57d111c0899228bdca43b87be32c99aeaff6b1" exitCode=0
Oct 30 21:43:32.340636 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:32.340218    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9" event={"ID":"1d406ea5-aa2d-4a1d-a010-65959654939a","Type":"ContainerDied","Data":"a90b4314fd34a3638df0ed35ef57d111c0899228bdca43b87be32c99aeaff6b1"}
Oct 30 21:43:32.340636 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:32.340623    5713 scope.go:122] "RemoveContainer" containerID="a90b4314fd34a3638df0ed35ef57d111c0899228bdca43b87be32c99aeaff6b1"
Oct 30 21:43:32.341861 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:32.341835    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:43:32.341927 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:32.341904    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:43:32.341962 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:32.341936    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:43:33.363319 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:33.363287    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9" event={"ID":"1d406ea5-aa2d-4a1d-a010-65959654939a","Type":"ContainerStarted","Data":"96bd64ac5517c6b658d50a840209c5116a3d369ec38e580f682f427be7de5249"}
Oct 30 21:43:34.224277 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:34.224234    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9"
Oct 30 21:43:34.225219 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:34.225192    5713 patch_prober.go:29] interesting pod/e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" start-of-body=
Oct 30 21:43:34.225331 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:34.225238    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9" podUID="1d406ea5-aa2d-4a1d-a010-65959654939a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused"
Oct 30 21:43:35.223650 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:35.223613    5713 patch_prober.go:29] interesting pod/e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" start-of-body=
Oct 30 21:43:35.224112 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:35.223670    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9" podUID="1d406ea5-aa2d-4a1d-a010-65959654939a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused"
Oct 30 21:43:36.224423 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:36.224392    5713 patch_prober.go:29] interesting pod/e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" start-of-body=
Oct 30 21:43:36.224878 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:36.224447    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9" podUID="1d406ea5-aa2d-4a1d-a010-65959654939a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused"
Oct 30 21:43:37.224500 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:37.224466    5713 patch_prober.go:29] interesting pod/e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" start-of-body=
Oct 30 21:43:37.224919 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:37.224515    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9" podUID="1d406ea5-aa2d-4a1d-a010-65959654939a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused"
Oct 30 21:43:37.891383 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:37.891353    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d4597073-8ff0-4a4e-933c-c64e934fe977/prometheus/0.log"
Oct 30 21:43:37.900320 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:37.900291    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d4597073-8ff0-4a4e-933c-c64e934fe977/prometheus/0.log"
Oct 30 21:43:37.906284 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:37.906258    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r7lpf_764461d5-e0b2-4856-a97e-0c5d3d34a497/ovn-acl-logging/0.log"
Oct 30 21:43:37.918539 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:37.918514    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r7lpf_764461d5-e0b2-4856-a97e-0c5d3d34a497/ovn-acl-logging/0.log"
Oct 30 21:43:38.188764 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:38.188730    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9"
Oct 30 21:43:38.224006 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:38.223971    5713 patch_prober.go:29] interesting pod/e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" start-of-body=
Oct 30 21:43:38.224185 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:38.224018    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9" podUID="1d406ea5-aa2d-4a1d-a010-65959654939a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused"
Oct 30 21:43:39.228276 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:39.228242    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9"
Oct 30 21:43:39.229261 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:39.229239    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9"
Oct 30 21:43:46.061229 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:46.061197    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1a1f4ae2-4l6sx"]
Oct 30 21:43:46.064745 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:46.064729    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1a1f4ae2-4l6sx"
Oct 30 21:43:46.121524 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:46.121491    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1a1f4ae2-4l6sx"]
Oct 30 21:43:46.168809 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:46.168739    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:43:46.168976 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:46.168821    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rwj5h\""
Oct 30 21:43:46.168976 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:46.168833    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:43:46.168976 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:46.168771    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:43:46.189411 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:46.189387    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/503501f8-172f-48a1-8552-d9bf370f8ba0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1a1f4ae2-4l6sx\" (UID: \"503501f8-172f-48a1-8552-d9bf370f8ba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1a1f4ae2-4l6sx"
Oct 30 21:43:46.189486 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:46.189418    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/503501f8-172f-48a1-8552-d9bf370f8ba0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1a1f4ae2-4l6sx\" (UID: \"503501f8-172f-48a1-8552-d9bf370f8ba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1a1f4ae2-4l6sx"
Oct 30 21:43:46.189524 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:46.189504    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8bs9\" (UniqueName: \"kubernetes.io/projected/503501f8-172f-48a1-8552-d9bf370f8ba0-kube-api-access-l8bs9\") pod \"maas-api-revoke-keys-e2e-ait-1a1f4ae2-4l6sx\" (UID: \"503501f8-172f-48a1-8552-d9bf370f8ba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1a1f4ae2-4l6sx"
Oct 30 21:43:46.291162 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:46.291120    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8bs9\" (UniqueName: \"kubernetes.io/projected/503501f8-172f-48a1-8552-d9bf370f8ba0-kube-api-access-l8bs9\") pod \"maas-api-revoke-keys-e2e-ait-1a1f4ae2-4l6sx\" (UID: \"503501f8-172f-48a1-8552-d9bf370f8ba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1a1f4ae2-4l6sx"
Oct 30 21:43:46.291344 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:46.291323    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/503501f8-172f-48a1-8552-d9bf370f8ba0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1a1f4ae2-4l6sx\" (UID: \"503501f8-172f-48a1-8552-d9bf370f8ba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1a1f4ae2-4l6sx"
Oct 30 21:43:46.291393 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:46.291352    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/503501f8-172f-48a1-8552-d9bf370f8ba0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1a1f4ae2-4l6sx\" (UID: \"503501f8-172f-48a1-8552-d9bf370f8ba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1a1f4ae2-4l6sx"
Oct 30 21:43:46.292016 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:46.291991    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/503501f8-172f-48a1-8552-d9bf370f8ba0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1a1f4ae2-4l6sx\" (UID: \"503501f8-172f-48a1-8552-d9bf370f8ba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1a1f4ae2-4l6sx"
Oct 30 21:43:46.293893 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:46.293870    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/503501f8-172f-48a1-8552-d9bf370f8ba0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1a1f4ae2-4l6sx\" (UID: \"503501f8-172f-48a1-8552-d9bf370f8ba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1a1f4ae2-4l6sx"
Oct 30 21:43:46.350812 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:46.350734    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8bs9\" (UniqueName: \"kubernetes.io/projected/503501f8-172f-48a1-8552-d9bf370f8ba0-kube-api-access-l8bs9\") pod \"maas-api-revoke-keys-e2e-ait-1a1f4ae2-4l6sx\" (UID: \"503501f8-172f-48a1-8552-d9bf370f8ba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1a1f4ae2-4l6sx"
Oct 30 21:43:46.378675 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:46.378646    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1a1f4ae2-4l6sx"
Oct 30 21:43:46.750701 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:46.750673    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1a1f4ae2-4l6sx"]
Oct 30 21:43:46.752902 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:43:46.752867    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod503501f8_172f_48a1_8552_d9bf370f8ba0.slice/crio-0474867c72c2f1e99eaedc6364775be52dee3628190910cfb82812c5665715f5 WatchSource:0}: Error finding container 0474867c72c2f1e99eaedc6364775be52dee3628190910cfb82812c5665715f5: Status 404 returned error can't find the container with id 0474867c72c2f1e99eaedc6364775be52dee3628190910cfb82812c5665715f5
Oct 30 21:43:47.448298 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:47.448257    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1a1f4ae2-4l6sx" event={"ID":"503501f8-172f-48a1-8552-d9bf370f8ba0","Type":"ContainerStarted","Data":"fe076e8b466870c949e6bac9d70c8dc3342a4cb8ac181176a484ba0f788e3a8d"}
Oct 30 21:43:47.448698 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:47.448320    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1a1f4ae2-4l6sx" event={"ID":"503501f8-172f-48a1-8552-d9bf370f8ba0","Type":"ContainerStarted","Data":"0474867c72c2f1e99eaedc6364775be52dee3628190910cfb82812c5665715f5"}
Oct 30 21:43:48.455675 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:48.455634    5713 generic.go:356] "Generic (PLEG): container finished" podID="503501f8-172f-48a1-8552-d9bf370f8ba0" containerID="fe076e8b466870c949e6bac9d70c8dc3342a4cb8ac181176a484ba0f788e3a8d" exitCode=0
Oct 30 21:43:48.455675 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:48.455660    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1a1f4ae2-4l6sx" event={"ID":"503501f8-172f-48a1-8552-d9bf370f8ba0","Type":"ContainerDied","Data":"fe076e8b466870c949e6bac9d70c8dc3342a4cb8ac181176a484ba0f788e3a8d"}
Oct 30 21:43:49.596954 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:49.596931    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1a1f4ae2-4l6sx"
Oct 30 21:43:49.725583 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:49.725497    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/503501f8-172f-48a1-8552-d9bf370f8ba0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/503501f8-172f-48a1-8552-d9bf370f8ba0-maas-api-xks-ca\") pod \"503501f8-172f-48a1-8552-d9bf370f8ba0\" (UID: \"503501f8-172f-48a1-8552-d9bf370f8ba0\") "
Oct 30 21:43:49.725583 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:49.725536    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/503501f8-172f-48a1-8552-d9bf370f8ba0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/503501f8-172f-48a1-8552-d9bf370f8ba0-maas-api-openshift-ca\") pod \"503501f8-172f-48a1-8552-d9bf370f8ba0\" (UID: \"503501f8-172f-48a1-8552-d9bf370f8ba0\") "
Oct 30 21:43:49.725873 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:49.725645    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/503501f8-172f-48a1-8552-d9bf370f8ba0-kube-api-access-l8bs9\" (UniqueName: \"kubernetes.io/projected/503501f8-172f-48a1-8552-d9bf370f8ba0-kube-api-access-l8bs9\") pod \"503501f8-172f-48a1-8552-d9bf370f8ba0\" (UID: \"503501f8-172f-48a1-8552-d9bf370f8ba0\") "
Oct 30 21:43:49.725961 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:49.725937    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/503501f8-172f-48a1-8552-d9bf370f8ba0-maas-api-openshift-ca" pod "503501f8-172f-48a1-8552-d9bf370f8ba0" (UID: "503501f8-172f-48a1-8552-d9bf370f8ba0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:43:49.727871 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:49.727843    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/503501f8-172f-48a1-8552-d9bf370f8ba0-maas-api-xks-ca" pod "503501f8-172f-48a1-8552-d9bf370f8ba0" (UID: "503501f8-172f-48a1-8552-d9bf370f8ba0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:43:49.727989 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:49.727907    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/503501f8-172f-48a1-8552-d9bf370f8ba0-kube-api-access-l8bs9" pod "503501f8-172f-48a1-8552-d9bf370f8ba0" (UID: "503501f8-172f-48a1-8552-d9bf370f8ba0"). InnerVolumeSpecName "kube-api-access-l8bs9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:43:49.826813 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:49.826758    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8bs9\" (UniqueName: \"kubernetes.io/projected/503501f8-172f-48a1-8552-d9bf370f8ba0-kube-api-access-l8bs9\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:43:49.826813 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:49.826812    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/503501f8-172f-48a1-8552-d9bf370f8ba0-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:43:49.827020 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:49.826825    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/503501f8-172f-48a1-8552-d9bf370f8ba0-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:43:50.467549 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:50.467516    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1a1f4ae2-4l6sx"
Oct 30 21:43:50.467745 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:50.467517    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1a1f4ae2-4l6sx" event={"ID":"503501f8-172f-48a1-8552-d9bf370f8ba0","Type":"ContainerDied","Data":"0474867c72c2f1e99eaedc6364775be52dee3628190910cfb82812c5665715f5"}
Oct 30 21:43:50.467745 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:50.467654    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0474867c72c2f1e99eaedc6364775be52dee3628190910cfb82812c5665715f5"
Oct 30 21:43:54.457485 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:43:54.457444    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:43:57.597456 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:57.597412    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1a1f4ae2-5c86778f44-mhzh2"]
Oct 30 21:43:57.710395 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:43:57.710364    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1a1f4ae2-58bf5c4759-s559l"]
Oct 30 21:44:02.598057 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:02.597963    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-1a1f4ae2-5c86778f44-mhzh2" podUID="65806364-2b75-48a3-8d85-9ef7bc8f9074" containerName="payload-processing" containerID="cri-o://199f8040857fa08e4706957a5b2245582857a1771d70c22fb657213bd0a435a1" gracePeriod=25
Oct 30 21:44:02.711466 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:02.711420    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-1a1f4ae2-58bf5c4759-s559l" podUID="8c83aa65-8754-445c-b6f1-260dce60ae6c" containerName="payload-pre-processing" containerID="cri-o://e7574c80b9dfd49afd7fa7e429392f25be24154d33f9d903fe63773e7e9991d6" gracePeriod=25
Oct 30 21:44:02.853303 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:02.853221    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1a1f4ae2-5c86778f44-mhzh2"
Oct 30 21:44:02.967451 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:02.967424    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65806364-2b75-48a3-8d85-9ef7bc8f9074-kube-api-access-4kl54\" (UniqueName: \"kubernetes.io/projected/65806364-2b75-48a3-8d85-9ef7bc8f9074-kube-api-access-4kl54\") pod \"65806364-2b75-48a3-8d85-9ef7bc8f9074\" (UID: \"65806364-2b75-48a3-8d85-9ef7bc8f9074\") "
Oct 30 21:44:02.967619 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:02.967541    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65806364-2b75-48a3-8d85-9ef7bc8f9074-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65806364-2b75-48a3-8d85-9ef7bc8f9074-plugins-config-volume\") pod \"65806364-2b75-48a3-8d85-9ef7bc8f9074\" (UID: \"65806364-2b75-48a3-8d85-9ef7bc8f9074\") "
Oct 30 21:44:02.968277 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:02.968244    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65806364-2b75-48a3-8d85-9ef7bc8f9074-plugins-config-volume" pod "65806364-2b75-48a3-8d85-9ef7bc8f9074" (UID: "65806364-2b75-48a3-8d85-9ef7bc8f9074"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:44:02.970724 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:02.970693    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65806364-2b75-48a3-8d85-9ef7bc8f9074-kube-api-access-4kl54" pod "65806364-2b75-48a3-8d85-9ef7bc8f9074" (UID: "65806364-2b75-48a3-8d85-9ef7bc8f9074"). InnerVolumeSpecName "kube-api-access-4kl54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:44:02.985586 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:02.985557    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1a1f4ae2-58bf5c4759-s559l"
Oct 30 21:44:03.069006 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:03.068978    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c83aa65-8754-445c-b6f1-260dce60ae6c-kube-api-access-dll4t\" (UniqueName: \"kubernetes.io/projected/8c83aa65-8754-445c-b6f1-260dce60ae6c-kube-api-access-dll4t\") pod \"8c83aa65-8754-445c-b6f1-260dce60ae6c\" (UID: \"8c83aa65-8754-445c-b6f1-260dce60ae6c\") "
Oct 30 21:44:03.069197 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:03.069105    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c83aa65-8754-445c-b6f1-260dce60ae6c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c83aa65-8754-445c-b6f1-260dce60ae6c-plugins-config-volume\") pod \"8c83aa65-8754-445c-b6f1-260dce60ae6c\" (UID: \"8c83aa65-8754-445c-b6f1-260dce60ae6c\") "
Oct 30 21:44:03.069554 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:03.069460    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c83aa65-8754-445c-b6f1-260dce60ae6c-plugins-config-volume" pod "8c83aa65-8754-445c-b6f1-260dce60ae6c" (UID: "8c83aa65-8754-445c-b6f1-260dce60ae6c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:44:03.069554 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:03.069510    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kl54\" (UniqueName: \"kubernetes.io/projected/65806364-2b75-48a3-8d85-9ef7bc8f9074-kube-api-access-4kl54\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:44:03.069554 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:03.069531    5713 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65806364-2b75-48a3-8d85-9ef7bc8f9074-plugins-config-volume\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:44:03.071324 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:03.071299    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c83aa65-8754-445c-b6f1-260dce60ae6c-kube-api-access-dll4t" pod "8c83aa65-8754-445c-b6f1-260dce60ae6c" (UID: "8c83aa65-8754-445c-b6f1-260dce60ae6c"). InnerVolumeSpecName "kube-api-access-dll4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:44:03.170406 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:03.170313    5713 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c83aa65-8754-445c-b6f1-260dce60ae6c-plugins-config-volume\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:44:03.170406 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:03.170346    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dll4t\" (UniqueName: \"kubernetes.io/projected/8c83aa65-8754-445c-b6f1-260dce60ae6c-kube-api-access-dll4t\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:44:03.540729 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:03.540693    5713 generic.go:356] "Generic (PLEG): container finished" podID="8c83aa65-8754-445c-b6f1-260dce60ae6c" containerID="e7574c80b9dfd49afd7fa7e429392f25be24154d33f9d903fe63773e7e9991d6" exitCode=0
Oct 30 21:44:03.540950 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:03.540741    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1a1f4ae2-58bf5c4759-s559l"
Oct 30 21:44:03.540950 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:03.540766    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1a1f4ae2-58bf5c4759-s559l" event={"ID":"8c83aa65-8754-445c-b6f1-260dce60ae6c","Type":"ContainerDied","Data":"e7574c80b9dfd49afd7fa7e429392f25be24154d33f9d903fe63773e7e9991d6"}
Oct 30 21:44:03.540950 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:03.540843    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1a1f4ae2-58bf5c4759-s559l" event={"ID":"8c83aa65-8754-445c-b6f1-260dce60ae6c","Type":"ContainerDied","Data":"fe8148ea42bddca9260aa7c868f9b47718fbb5fca93269cf6239a4fce09e6438"}
Oct 30 21:44:03.540950 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:03.540875    5713 scope.go:122] "RemoveContainer" containerID="e7574c80b9dfd49afd7fa7e429392f25be24154d33f9d903fe63773e7e9991d6"
Oct 30 21:44:03.542387 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:03.542362    5713 generic.go:356] "Generic (PLEG): container finished" podID="65806364-2b75-48a3-8d85-9ef7bc8f9074" containerID="199f8040857fa08e4706957a5b2245582857a1771d70c22fb657213bd0a435a1" exitCode=0
Oct 30 21:44:03.542523 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:03.542410    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1a1f4ae2-5c86778f44-mhzh2"
Oct 30 21:44:03.542523 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:03.542411    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1a1f4ae2-5c86778f44-mhzh2" event={"ID":"65806364-2b75-48a3-8d85-9ef7bc8f9074","Type":"ContainerDied","Data":"199f8040857fa08e4706957a5b2245582857a1771d70c22fb657213bd0a435a1"}
Oct 30 21:44:03.542523 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:03.542464    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1a1f4ae2-5c86778f44-mhzh2" event={"ID":"65806364-2b75-48a3-8d85-9ef7bc8f9074","Type":"ContainerDied","Data":"b8accc72caca68688491362efeffad6751be716b101389993f60305a4e8addb5"}
Oct 30 21:44:03.551171 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:03.551152    5713 scope.go:122] "RemoveContainer" containerID="e7574c80b9dfd49afd7fa7e429392f25be24154d33f9d903fe63773e7e9991d6"
Oct 30 21:44:03.551456 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:44:03.551441    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7574c80b9dfd49afd7fa7e429392f25be24154d33f9d903fe63773e7e9991d6\": container with ID starting with e7574c80b9dfd49afd7fa7e429392f25be24154d33f9d903fe63773e7e9991d6 not found: ID does not exist" containerID="e7574c80b9dfd49afd7fa7e429392f25be24154d33f9d903fe63773e7e9991d6"
Oct 30 21:44:03.551502 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:03.551466    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7574c80b9dfd49afd7fa7e429392f25be24154d33f9d903fe63773e7e9991d6"} err="failed to get container status \"e7574c80b9dfd49afd7fa7e429392f25be24154d33f9d903fe63773e7e9991d6\": rpc error: code = NotFound desc = could not find container \"e7574c80b9dfd49afd7fa7e429392f25be24154d33f9d903fe63773e7e9991d6\": container with ID starting with e7574c80b9dfd49afd7fa7e429392f25be24154d33f9d903fe63773e7e9991d6 not found: ID does not exist"
Oct 30 21:44:03.551502 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:03.551482    5713 scope.go:122] "RemoveContainer" containerID="199f8040857fa08e4706957a5b2245582857a1771d70c22fb657213bd0a435a1"
Oct 30 21:44:03.561102 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:03.561082    5713 scope.go:122] "RemoveContainer" containerID="199f8040857fa08e4706957a5b2245582857a1771d70c22fb657213bd0a435a1"
Oct 30 21:44:03.561396 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:44:03.561378    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"199f8040857fa08e4706957a5b2245582857a1771d70c22fb657213bd0a435a1\": container with ID starting with 199f8040857fa08e4706957a5b2245582857a1771d70c22fb657213bd0a435a1 not found: ID does not exist" containerID="199f8040857fa08e4706957a5b2245582857a1771d70c22fb657213bd0a435a1"
Oct 30 21:44:03.561453 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:03.561406    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"199f8040857fa08e4706957a5b2245582857a1771d70c22fb657213bd0a435a1"} err="failed to get container status \"199f8040857fa08e4706957a5b2245582857a1771d70c22fb657213bd0a435a1\": rpc error: code = NotFound desc = could not find container \"199f8040857fa08e4706957a5b2245582857a1771d70c22fb657213bd0a435a1\": container with ID starting with 199f8040857fa08e4706957a5b2245582857a1771d70c22fb657213bd0a435a1 not found: ID does not exist"
Oct 30 21:44:03.704112 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:03.704070    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1a1f4ae2-5c86778f44-mhzh2"]
Oct 30 21:44:03.710155 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:03.710126    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1a1f4ae2-5c86778f44-mhzh2"]
Oct 30 21:44:03.819188 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:03.819100    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1a1f4ae2-58bf5c4759-s559l"]
Oct 30 21:44:03.825087 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:03.825059    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1a1f4ae2-58bf5c4759-s559l"]
Oct 30 21:44:05.012891 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:05.012858    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1a1f4ae2-4l6sx"]
Oct 30 21:44:05.063093 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:05.063059    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65806364-2b75-48a3-8d85-9ef7bc8f9074" path="/var/lib/kubelet/pods/65806364-2b75-48a3-8d85-9ef7bc8f9074/volumes"
Oct 30 21:44:05.063451 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:05.063438    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c83aa65-8754-445c-b6f1-260dce60ae6c" path="/var/lib/kubelet/pods/8c83aa65-8754-445c-b6f1-260dce60ae6c/volumes"
Oct 30 21:44:05.066106 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:05.066081    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1a1f4ae2-4l6sx"]
Oct 30 21:44:07.061906 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:07.061877    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="503501f8-172f-48a1-8552-d9bf370f8ba0" path="/var/lib/kubelet/pods/503501f8-172f-48a1-8552-d9bf370f8ba0/volumes"
Oct 30 21:44:07.218596 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:07.218565    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9"]
Oct 30 21:44:07.218782 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:07.218760    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9" podUID="1d406ea5-aa2d-4a1d-a010-65959654939a" containerName="istio-proxy" containerID="cri-o://96bd64ac5517c6b658d50a840209c5116a3d369ec38e580f682f427be7de5249" gracePeriod=30
Oct 30 21:44:09.074137 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:09.074101    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r"]
Oct 30 21:44:09.074718 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:09.074697    5713 cpu_manager.go:414] "containerMap: removing container" podUID="8c83aa65-8754-445c-b6f1-260dce60ae6c" containerName="payload-pre-processing"
Oct 30 21:44:09.074834 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:09.074720    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c83aa65-8754-445c-b6f1-260dce60ae6c" containerName="payload-pre-processing"
Oct 30 21:44:09.074834 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:09.074741    5713 cpu_manager.go:414] "containerMap: removing container" podUID="65806364-2b75-48a3-8d85-9ef7bc8f9074" containerName="payload-processing"
Oct 30 21:44:09.074834 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:09.074748    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65806364-2b75-48a3-8d85-9ef7bc8f9074" containerName="payload-processing"
Oct 30 21:44:09.074834 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:09.074831    5713 cpu_manager.go:414] "containerMap: removing container" podUID="503501f8-172f-48a1-8552-d9bf370f8ba0" containerName="revoke-keys"
Oct 30 21:44:09.075055 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:09.074840    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="503501f8-172f-48a1-8552-d9bf370f8ba0" containerName="revoke-keys"
Oct 30 21:44:09.075055 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:09.074943    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="503501f8-172f-48a1-8552-d9bf370f8ba0" containerName="revoke-keys"
Oct 30 21:44:09.075055 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:09.074959    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c83aa65-8754-445c-b6f1-260dce60ae6c" containerName="payload-pre-processing"
Oct 30 21:44:09.075055 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:09.074974    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="65806364-2b75-48a3-8d85-9ef7bc8f9074" containerName="payload-processing"
Oct 30 21:44:09.079934 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:09.079913    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r"
Oct 30 21:44:09.178881 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:09.178844    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r"]
Oct 30 21:44:09.224781 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:09.224710    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-0e8a1a-openshift-default-dockercfg-8wdz6\""
Oct 30 21:44:09.229404 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:09.229370    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd00fc40-6246-45ea-abf3-4c9e8ab142f3-istio-data\") pod \"e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r\" (UID: \"dd00fc40-6246-45ea-abf3-4c9e8ab142f3\") " pod="openshift-ingress/e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r"
Oct 30 21:44:09.229779 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:09.229752    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd00fc40-6246-45ea-abf3-4c9e8ab142f3-workload-certs\") pod \"e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r\" (UID: \"dd00fc40-6246-45ea-abf3-4c9e8ab142f3\") " pod="openshift-ingress/e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r"
Oct 30 21:44:09.230107 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:09.230087    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd00fc40-6246-45ea-abf3-4c9e8ab142f3-workload-socket\") pod \"e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r\" (UID: \"dd00fc40-6246-45ea-abf3-4c9e8ab142f3\") " pod="openshift-ingress/e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r"
Oct 30 21:44:09.230523 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:09.230498    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd00fc40-6246-45ea-abf3-4c9e8ab142f3-istio-envoy\") pod \"e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r\" (UID: \"dd00fc40-6246-45ea-abf3-4c9e8ab142f3\") " pod="openshift-ingress/e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r"
Oct 30 21:44:09.230641 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:09.230607    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnhvs\" (UniqueName: \"kubernetes.io/projected/dd00fc40-6246-45ea-abf3-4c9e8ab142f3-kube-api-access-dnhvs\") pod \"e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r\" (UID: \"dd00fc40-6246-45ea-abf3-4c9e8ab142f3\") " pod="openshift-ingress/e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r"
Oct 30 21:44:09.230735 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:09.230695    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd00fc40-6246-45ea-abf3-4c9e8ab142f3-istiod-ca-cert\") pod \"e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r\" (UID: \"dd00fc40-6246-45ea-abf3-4c9e8ab142f3\") " pod="openshift-ingress/e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r"
Oct 30 21:44:09.230854 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:09.230746    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd00fc40-6246-45ea-abf3-4c9e8ab142f3-istio-podinfo\") pod \"e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r\" (UID: \"dd00fc40-6246-45ea-abf3-4c9e8ab142f3\") " pod="openshift-ingress/e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r"
Oct 30 21:44:09.230854 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:09.230773    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd00fc40-6246-45ea-abf3-4c9e8ab142f3-istio-token\") pod \"e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r\" (UID: \"dd00fc40-6246-45ea-abf3-4c9e8ab142f3\") " pod="openshift-ingress/e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r"
Oct 30 21:44:09.230979 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:09.230893    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd00fc40-6246-45ea-abf3-4c9e8ab142f3-credential-socket\") pod \"e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r\" (UID: \"dd00fc40-6246-45ea-abf3-4c9e8ab142f3\") " pod="openshift-ingress/e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r"
Oct 30 21:44:09.231369 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:09.231347    5713 patch_prober.go:29] interesting pod/e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:44:09.231538 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:09.231510    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9" podUID="1d406ea5-aa2d-4a1d-a010-65959654939a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:44:09.331462 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:09.331373    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnhvs\" (UniqueName: \"kubernetes.io/projected/dd00fc40-6246-45ea-abf3-4c9e8ab142f3-kube-api-access-dnhvs\") pod \"e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r\" (UID: \"dd00fc40-6246-45ea-abf3-4c9e8ab142f3\") " pod="openshift-ingress/e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r"
Oct 30 21:44:09.331462 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:09.331413    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd00fc40-6246-45ea-abf3-4c9e8ab142f3-istiod-ca-cert\") pod \"e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r\" (UID: \"dd00fc40-6246-45ea-abf3-4c9e8ab142f3\") " pod="openshift-ingress/e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r"
Oct 30 21:44:09.331462 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:09.331437    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd00fc40-6246-45ea-abf3-4c9e8ab142f3-istio-podinfo\") pod \"e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r\" (UID: \"dd00fc40-6246-45ea-abf3-4c9e8ab142f3\") " pod="openshift-ingress/e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r"
Oct 30 21:44:09.331462 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:09.331454    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd00fc40-6246-45ea-abf3-4c9e8ab142f3-istio-token\") pod \"e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r\" (UID: \"dd00fc40-6246-45ea-abf3-4c9e8ab142f3\") " pod="openshift-ingress/e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r"
Oct 30 21:44:09.332560 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:09.331490    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd00fc40-6246-45ea-abf3-4c9e8ab142f3-credential-socket\") pod \"e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r\" (UID: \"dd00fc40-6246-45ea-abf3-4c9e8ab142f3\") " pod="openshift-ingress/e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r"
Oct 30 21:44:09.332560 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:09.331564    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd00fc40-6246-45ea-abf3-4c9e8ab142f3-istio-data\") pod \"e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r\" (UID: \"dd00fc40-6246-45ea-abf3-4c9e8ab142f3\") " pod="openshift-ingress/e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r"
Oct 30 21:44:09.332560 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:09.331624    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd00fc40-6246-45ea-abf3-4c9e8ab142f3-workload-certs\") pod \"e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r\" (UID: \"dd00fc40-6246-45ea-abf3-4c9e8ab142f3\") " pod="openshift-ingress/e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r"
Oct 30 21:44:09.332560 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:09.331656    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd00fc40-6246-45ea-abf3-4c9e8ab142f3-workload-socket\") pod \"e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r\" (UID: \"dd00fc40-6246-45ea-abf3-4c9e8ab142f3\") " pod="openshift-ingress/e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r"
Oct 30 21:44:09.332560 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:09.331696    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd00fc40-6246-45ea-abf3-4c9e8ab142f3-istio-envoy\") pod \"e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r\" (UID: \"dd00fc40-6246-45ea-abf3-4c9e8ab142f3\") " pod="openshift-ingress/e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r"
Oct 30 21:44:09.332560 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:09.332118    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd00fc40-6246-45ea-abf3-4c9e8ab142f3-credential-socket\") pod \"e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r\" (UID: \"dd00fc40-6246-45ea-abf3-4c9e8ab142f3\") " pod="openshift-ingress/e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r"
Oct 30 21:44:09.332560 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:09.332258    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd00fc40-6246-45ea-abf3-4c9e8ab142f3-workload-certs\") pod \"e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r\" (UID: \"dd00fc40-6246-45ea-abf3-4c9e8ab142f3\") " pod="openshift-ingress/e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r"
Oct 30 21:44:09.332560 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:09.332536    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd00fc40-6246-45ea-abf3-4c9e8ab142f3-workload-socket\") pod \"e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r\" (UID: \"dd00fc40-6246-45ea-abf3-4c9e8ab142f3\") " pod="openshift-ingress/e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r"
Oct 30 21:44:09.333268 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:09.332540    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd00fc40-6246-45ea-abf3-4c9e8ab142f3-istio-data\") pod \"e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r\" (UID: \"dd00fc40-6246-45ea-abf3-4c9e8ab142f3\") " pod="openshift-ingress/e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r"
Oct 30 21:44:09.333268 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:09.332978    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd00fc40-6246-45ea-abf3-4c9e8ab142f3-istiod-ca-cert\") pod \"e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r\" (UID: \"dd00fc40-6246-45ea-abf3-4c9e8ab142f3\") " pod="openshift-ingress/e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r"
Oct 30 21:44:09.335077 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:09.335048    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd00fc40-6246-45ea-abf3-4c9e8ab142f3-istio-envoy\") pod \"e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r\" (UID: \"dd00fc40-6246-45ea-abf3-4c9e8ab142f3\") " pod="openshift-ingress/e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r"
Oct 30 21:44:09.335205 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:09.335167    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd00fc40-6246-45ea-abf3-4c9e8ab142f3-istio-podinfo\") pod \"e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r\" (UID: \"dd00fc40-6246-45ea-abf3-4c9e8ab142f3\") " pod="openshift-ingress/e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r"
Oct 30 21:44:09.390612 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:09.390580    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd00fc40-6246-45ea-abf3-4c9e8ab142f3-istio-token\") pod \"e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r\" (UID: \"dd00fc40-6246-45ea-abf3-4c9e8ab142f3\") " pod="openshift-ingress/e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r"
Oct 30 21:44:09.390747 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:09.390694    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnhvs\" (UniqueName: \"kubernetes.io/projected/dd00fc40-6246-45ea-abf3-4c9e8ab142f3-kube-api-access-dnhvs\") pod \"e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r\" (UID: \"dd00fc40-6246-45ea-abf3-4c9e8ab142f3\") " pod="openshift-ingress/e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r"
Oct 30 21:44:09.393517 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:09.393499    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r"
Oct 30 21:44:09.821595 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:09.821513    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r"]
Oct 30 21:44:09.824170 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:44:09.824127    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd00fc40_6246_45ea_abf3_4c9e8ab142f3.slice/crio-9c3754f9a3328b7af9ed5b42278c3238e2a61c4946dcac385f2f0b366dccf1c4 WatchSource:0}: Error finding container 9c3754f9a3328b7af9ed5b42278c3238e2a61c4946dcac385f2f0b366dccf1c4: Status 404 returned error can't find the container with id 9c3754f9a3328b7af9ed5b42278c3238e2a61c4946dcac385f2f0b366dccf1c4
Oct 30 21:44:09.826327 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:09.826299    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:44:09.826420 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:09.826363    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:44:09.826420 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:09.826391    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:44:10.585905 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:10.585863    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r" event={"ID":"dd00fc40-6246-45ea-abf3-4c9e8ab142f3","Type":"ContainerStarted","Data":"b5ac9b91d24402c89b780e29d2e6cda9d796151ef079ce7e93fa7042e5f84ccf"}
Oct 30 21:44:10.586312 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:10.585923    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r" event={"ID":"dd00fc40-6246-45ea-abf3-4c9e8ab142f3","Type":"ContainerStarted","Data":"9c3754f9a3328b7af9ed5b42278c3238e2a61c4946dcac385f2f0b366dccf1c4"}
Oct 30 21:44:10.802034 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:10.801978    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r" podStartSLOduration=1.8019628760000002 podStartE2EDuration="1.801962876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:10.750473405 +0000 UTC m=+3933.815249797" watchObservedRunningTime="2026-10-30 21:44:10.801962876 +0000 UTC m=+3933.866739273"
Oct 30 21:44:11.394685 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:11.394647    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r"
Oct 30 21:44:11.399972 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:11.399940    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r"
Oct 30 21:44:11.591121 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:11.591087    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r"
Oct 30 21:44:11.592021 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:11.592004    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r"
Oct 30 21:44:12.479206 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:12.479177    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9"
Oct 30 21:44:12.560728 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:12.560640    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d406ea5-aa2d-4a1d-a010-65959654939a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d406ea5-aa2d-4a1d-a010-65959654939a-credential-socket\") pod \"1d406ea5-aa2d-4a1d-a010-65959654939a\" (UID: \"1d406ea5-aa2d-4a1d-a010-65959654939a\") "
Oct 30 21:44:12.560728 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:12.560699    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1d406ea5-aa2d-4a1d-a010-65959654939a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d406ea5-aa2d-4a1d-a010-65959654939a-istio-podinfo\") pod \"1d406ea5-aa2d-4a1d-a010-65959654939a\" (UID: \"1d406ea5-aa2d-4a1d-a010-65959654939a\") "
Oct 30 21:44:12.560964 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:12.560732    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d406ea5-aa2d-4a1d-a010-65959654939a-istio-token\" (UniqueName: \"kubernetes.io/projected/1d406ea5-aa2d-4a1d-a010-65959654939a-istio-token\") pod \"1d406ea5-aa2d-4a1d-a010-65959654939a\" (UID: \"1d406ea5-aa2d-4a1d-a010-65959654939a\") "
Oct 30 21:44:12.560964 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:12.560750    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d406ea5-aa2d-4a1d-a010-65959654939a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d406ea5-aa2d-4a1d-a010-65959654939a-workload-socket\") pod \"1d406ea5-aa2d-4a1d-a010-65959654939a\" (UID: \"1d406ea5-aa2d-4a1d-a010-65959654939a\") "
Oct 30 21:44:12.560964 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:12.560836    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d406ea5-aa2d-4a1d-a010-65959654939a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d406ea5-aa2d-4a1d-a010-65959654939a-istio-data\") pod \"1d406ea5-aa2d-4a1d-a010-65959654939a\" (UID: \"1d406ea5-aa2d-4a1d-a010-65959654939a\") "
Oct 30 21:44:12.560964 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:12.560878    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d406ea5-aa2d-4a1d-a010-65959654939a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d406ea5-aa2d-4a1d-a010-65959654939a-istio-envoy\") pod \"1d406ea5-aa2d-4a1d-a010-65959654939a\" (UID: \"1d406ea5-aa2d-4a1d-a010-65959654939a\") "
Oct 30 21:44:12.560964 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:12.560946    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d406ea5-aa2d-4a1d-a010-65959654939a-kube-api-access-tx7cf\" (UniqueName: \"kubernetes.io/projected/1d406ea5-aa2d-4a1d-a010-65959654939a-kube-api-access-tx7cf\") pod \"1d406ea5-aa2d-4a1d-a010-65959654939a\" (UID: \"1d406ea5-aa2d-4a1d-a010-65959654939a\") "
Oct 30 21:44:12.561332 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:12.560982    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d406ea5-aa2d-4a1d-a010-65959654939a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d406ea5-aa2d-4a1d-a010-65959654939a-istiod-ca-cert\") pod \"1d406ea5-aa2d-4a1d-a010-65959654939a\" (UID: \"1d406ea5-aa2d-4a1d-a010-65959654939a\") "
Oct 30 21:44:12.561332 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:12.560976    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d406ea5-aa2d-4a1d-a010-65959654939a-credential-socket" pod "1d406ea5-aa2d-4a1d-a010-65959654939a" (UID: "1d406ea5-aa2d-4a1d-a010-65959654939a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:44:12.561332 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:12.561017    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d406ea5-aa2d-4a1d-a010-65959654939a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d406ea5-aa2d-4a1d-a010-65959654939a-workload-certs\") pod \"1d406ea5-aa2d-4a1d-a010-65959654939a\" (UID: \"1d406ea5-aa2d-4a1d-a010-65959654939a\") "
Oct 30 21:44:12.561332 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:12.561124    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d406ea5-aa2d-4a1d-a010-65959654939a-workload-socket" pod "1d406ea5-aa2d-4a1d-a010-65959654939a" (UID: "1d406ea5-aa2d-4a1d-a010-65959654939a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:44:12.561332 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:12.561139    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d406ea5-aa2d-4a1d-a010-65959654939a-istio-data" pod "1d406ea5-aa2d-4a1d-a010-65959654939a" (UID: "1d406ea5-aa2d-4a1d-a010-65959654939a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:44:12.561770 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:12.561421    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d406ea5-aa2d-4a1d-a010-65959654939a-workload-certs" pod "1d406ea5-aa2d-4a1d-a010-65959654939a" (UID: "1d406ea5-aa2d-4a1d-a010-65959654939a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:44:12.561770 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:12.561438    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d406ea5-aa2d-4a1d-a010-65959654939a-istiod-ca-cert" pod "1d406ea5-aa2d-4a1d-a010-65959654939a" (UID: "1d406ea5-aa2d-4a1d-a010-65959654939a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:44:12.561770 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:12.561443    5713 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d406ea5-aa2d-4a1d-a010-65959654939a-istio-data\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:44:12.561770 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:12.561470    5713 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d406ea5-aa2d-4a1d-a010-65959654939a-credential-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:44:12.561770 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:12.561485    5713 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d406ea5-aa2d-4a1d-a010-65959654939a-workload-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:44:12.563261 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:12.563229    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d406ea5-aa2d-4a1d-a010-65959654939a-kube-api-access-tx7cf" pod "1d406ea5-aa2d-4a1d-a010-65959654939a" (UID: "1d406ea5-aa2d-4a1d-a010-65959654939a"). InnerVolumeSpecName "kube-api-access-tx7cf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:44:12.563394 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:12.563352    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d406ea5-aa2d-4a1d-a010-65959654939a-istio-token" pod "1d406ea5-aa2d-4a1d-a010-65959654939a" (UID: "1d406ea5-aa2d-4a1d-a010-65959654939a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:44:12.563644 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:12.563624    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d406ea5-aa2d-4a1d-a010-65959654939a-istio-envoy" pod "1d406ea5-aa2d-4a1d-a010-65959654939a" (UID: "1d406ea5-aa2d-4a1d-a010-65959654939a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:44:12.563721 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:12.563637    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d406ea5-aa2d-4a1d-a010-65959654939a-istio-podinfo" pod "1d406ea5-aa2d-4a1d-a010-65959654939a" (UID: "1d406ea5-aa2d-4a1d-a010-65959654939a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:44:12.598775 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:12.598736    5713 generic.go:356] "Generic (PLEG): container finished" podID="1d406ea5-aa2d-4a1d-a010-65959654939a" containerID="96bd64ac5517c6b658d50a840209c5116a3d369ec38e580f682f427be7de5249" exitCode=0
Oct 30 21:44:12.599840 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:12.598824    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9"
Oct 30 21:44:12.599840 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:12.598820    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9" event={"ID":"1d406ea5-aa2d-4a1d-a010-65959654939a","Type":"ContainerDied","Data":"96bd64ac5517c6b658d50a840209c5116a3d369ec38e580f682f427be7de5249"}
Oct 30 21:44:12.599840 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:12.598883    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9" event={"ID":"1d406ea5-aa2d-4a1d-a010-65959654939a","Type":"ContainerDied","Data":"e162c68b6c1ab50a7b2b0cbca33418889268352c7c0e56214fc3f4f8b68e217f"}
Oct 30 21:44:12.599840 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:12.598914    5713 scope.go:122] "RemoveContainer" containerID="96bd64ac5517c6b658d50a840209c5116a3d369ec38e580f682f427be7de5249"
Oct 30 21:44:12.617783 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:12.617752    5713 scope.go:122] "RemoveContainer" containerID="a90b4314fd34a3638df0ed35ef57d111c0899228bdca43b87be32c99aeaff6b1"
Oct 30 21:44:12.627257 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:12.627234    5713 scope.go:122] "RemoveContainer" containerID="96bd64ac5517c6b658d50a840209c5116a3d369ec38e580f682f427be7de5249"
Oct 30 21:44:12.627553 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:44:12.627535    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96bd64ac5517c6b658d50a840209c5116a3d369ec38e580f682f427be7de5249\": container with ID starting with 96bd64ac5517c6b658d50a840209c5116a3d369ec38e580f682f427be7de5249 not found: ID does not exist" containerID="96bd64ac5517c6b658d50a840209c5116a3d369ec38e580f682f427be7de5249"
Oct 30 21:44:12.627613 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:12.627564    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96bd64ac5517c6b658d50a840209c5116a3d369ec38e580f682f427be7de5249"} err="failed to get container status \"96bd64ac5517c6b658d50a840209c5116a3d369ec38e580f682f427be7de5249\": rpc error: code = NotFound desc = could not find container \"96bd64ac5517c6b658d50a840209c5116a3d369ec38e580f682f427be7de5249\": container with ID starting with 96bd64ac5517c6b658d50a840209c5116a3d369ec38e580f682f427be7de5249 not found: ID does not exist"
Oct 30 21:44:12.627613 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:12.627582    5713 scope.go:122] "RemoveContainer" containerID="a90b4314fd34a3638df0ed35ef57d111c0899228bdca43b87be32c99aeaff6b1"
Oct 30 21:44:12.627836 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:44:12.627822    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a90b4314fd34a3638df0ed35ef57d111c0899228bdca43b87be32c99aeaff6b1\": container with ID starting with a90b4314fd34a3638df0ed35ef57d111c0899228bdca43b87be32c99aeaff6b1 not found: ID does not exist" containerID="a90b4314fd34a3638df0ed35ef57d111c0899228bdca43b87be32c99aeaff6b1"
Oct 30 21:44:12.627908 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:12.627840    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a90b4314fd34a3638df0ed35ef57d111c0899228bdca43b87be32c99aeaff6b1"} err="failed to get container status \"a90b4314fd34a3638df0ed35ef57d111c0899228bdca43b87be32c99aeaff6b1\": rpc error: code = NotFound desc = could not find container \"a90b4314fd34a3638df0ed35ef57d111c0899228bdca43b87be32c99aeaff6b1\": container with ID starting with a90b4314fd34a3638df0ed35ef57d111c0899228bdca43b87be32c99aeaff6b1 not found: ID does not exist"
Oct 30 21:44:12.663068 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:12.663030    5713 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d406ea5-aa2d-4a1d-a010-65959654939a-istio-envoy\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:44:12.663068 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:12.663066    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tx7cf\" (UniqueName: \"kubernetes.io/projected/1d406ea5-aa2d-4a1d-a010-65959654939a-kube-api-access-tx7cf\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:44:12.663358 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:12.663083    5713 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d406ea5-aa2d-4a1d-a010-65959654939a-istiod-ca-cert\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:44:12.663358 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:12.663101    5713 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d406ea5-aa2d-4a1d-a010-65959654939a-workload-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:44:12.663358 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:12.663116    5713 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d406ea5-aa2d-4a1d-a010-65959654939a-istio-podinfo\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:44:12.663358 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:12.663129    5713 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d406ea5-aa2d-4a1d-a010-65959654939a-istio-token\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:44:12.764094 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:12.764051    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9"]
Oct 30 21:44:12.767889 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:12.767863    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-1a1f4ae2-openshift-default-b8fdf746c-bmfg9"]
Oct 30 21:44:13.062036 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:13.062006    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d406ea5-aa2d-4a1d-a010-65959654939a" path="/var/lib/kubelet/pods/1d406ea5-aa2d-4a1d-a010-65959654939a/volumes"
Oct 30 21:44:17.361910 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:17.361859    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-0e8a1a-848bfd56cb-w9ph6"]
Oct 30 21:44:17.362556 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:17.362532    5713 cpu_manager.go:414] "containerMap: removing container" podUID="1d406ea5-aa2d-4a1d-a010-65959654939a" containerName="istio-proxy"
Oct 30 21:44:17.362626 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:17.362560    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d406ea5-aa2d-4a1d-a010-65959654939a" containerName="istio-proxy"
Oct 30 21:44:17.362626 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:17.362593    5713 cpu_manager.go:414] "containerMap: removing container" podUID="1d406ea5-aa2d-4a1d-a010-65959654939a" containerName="istio-proxy"
Oct 30 21:44:17.362626 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:17.362601    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d406ea5-aa2d-4a1d-a010-65959654939a" containerName="istio-proxy"
Oct 30 21:44:17.362819 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:17.362738    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d406ea5-aa2d-4a1d-a010-65959654939a" containerName="istio-proxy"
Oct 30 21:44:17.362819 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:17.362753    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d406ea5-aa2d-4a1d-a010-65959654939a" containerName="istio-proxy"
Oct 30 21:44:17.367561 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:17.367535    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-0e8a1a-848bfd56cb-w9ph6"
Oct 30 21:44:17.406286 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:17.406251    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt6vw\" (UniqueName: \"kubernetes.io/projected/0f01f7e7-94ab-4197-9441-c96b1a6577af-kube-api-access-vt6vw\") pod \"payload-processing-e2e-authcfg-http-0e8a1a-848bfd56cb-w9ph6\" (UID: \"0f01f7e7-94ab-4197-9441-c96b1a6577af\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-0e8a1a-848bfd56cb-w9ph6"
Oct 30 21:44:17.406495 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:17.406420    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f01f7e7-94ab-4197-9441-c96b1a6577af-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-0e8a1a-848bfd56cb-w9ph6\" (UID: \"0f01f7e7-94ab-4197-9441-c96b1a6577af\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-0e8a1a-848bfd56cb-w9ph6"
Oct 30 21:44:17.407155 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:17.407130    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-0e8a1a-848bfd56cb-w9ph6"]
Oct 30 21:44:17.453602 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:17.453573    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-0e8a1a\""
Oct 30 21:44:17.453835 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:17.453671    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-0e8a1a-dockercfg-nk8st\""
Oct 30 21:44:17.507373 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:17.507340    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt6vw\" (UniqueName: \"kubernetes.io/projected/0f01f7e7-94ab-4197-9441-c96b1a6577af-kube-api-access-vt6vw\") pod \"payload-processing-e2e-authcfg-http-0e8a1a-848bfd56cb-w9ph6\" (UID: \"0f01f7e7-94ab-4197-9441-c96b1a6577af\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-0e8a1a-848bfd56cb-w9ph6"
Oct 30 21:44:17.507572 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:17.507414    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f01f7e7-94ab-4197-9441-c96b1a6577af-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-0e8a1a-848bfd56cb-w9ph6\" (UID: \"0f01f7e7-94ab-4197-9441-c96b1a6577af\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-0e8a1a-848bfd56cb-w9ph6"
Oct 30 21:44:17.508133 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:17.508109    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f01f7e7-94ab-4197-9441-c96b1a6577af-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-0e8a1a-848bfd56cb-w9ph6\" (UID: \"0f01f7e7-94ab-4197-9441-c96b1a6577af\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-0e8a1a-848bfd56cb-w9ph6"
Oct 30 21:44:17.565828 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:17.565769    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt6vw\" (UniqueName: \"kubernetes.io/projected/0f01f7e7-94ab-4197-9441-c96b1a6577af-kube-api-access-vt6vw\") pod \"payload-processing-e2e-authcfg-http-0e8a1a-848bfd56cb-w9ph6\" (UID: \"0f01f7e7-94ab-4197-9441-c96b1a6577af\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-0e8a1a-848bfd56cb-w9ph6"
Oct 30 21:44:17.682311 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:17.682282    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-0e8a1a-848bfd56cb-w9ph6"
Oct 30 21:44:18.053476 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:18.053453    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-0e8a1a-848bfd56cb-w9ph6"]
Oct 30 21:44:18.055321 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:44:18.055290    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f01f7e7_94ab_4197_9441_c96b1a6577af.slice/crio-62d5f1f8b1e02429a6f9cda0985240d3d82a21a99cc5f04db7b639f8dee9ed2a WatchSource:0}: Error finding container 62d5f1f8b1e02429a6f9cda0985240d3d82a21a99cc5f04db7b639f8dee9ed2a: Status 404 returned error can't find the container with id 62d5f1f8b1e02429a6f9cda0985240d3d82a21a99cc5f04db7b639f8dee9ed2a
Oct 30 21:44:18.635903 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:18.635865    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-0e8a1a-848bfd56cb-w9ph6" event={"ID":"0f01f7e7-94ab-4197-9441-c96b1a6577af","Type":"ContainerStarted","Data":"d51c8259bf9a2c2762004e1789b76fe476cdf0d832ba9a3cde15e2e8b2b3fdec"}
Oct 30 21:44:18.636436 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:18.636068    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-0e8a1a-848bfd56cb-w9ph6"
Oct 30 21:44:18.636436 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:18.636111    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-0e8a1a-848bfd56cb-w9ph6" event={"ID":"0f01f7e7-94ab-4197-9441-c96b1a6577af","Type":"ContainerStarted","Data":"62d5f1f8b1e02429a6f9cda0985240d3d82a21a99cc5f04db7b639f8dee9ed2a"}
Oct 30 21:44:24.458160 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:44:24.458120    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:44:29.659711 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:29.659676    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-0e8a1a-848bfd56cb-w9ph6"
Oct 30 21:44:29.774677 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:29.774615    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-0e8a1a-848bfd56cb-w9ph6" podStartSLOduration=12.774599913 podStartE2EDuration="12.774599913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:18.755314832 +0000 UTC m=+3941.820091232" watchObservedRunningTime="2026-10-30 21:44:29.774599913 +0000 UTC m=+3952.839376308"
Oct 30 21:44:36.741633 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:36.741598    5713 generic.go:356] "Generic (PLEG): container finished" podID="dd00fc40-6246-45ea-abf3-4c9e8ab142f3" containerID="b5ac9b91d24402c89b780e29d2e6cda9d796151ef079ce7e93fa7042e5f84ccf" exitCode=0
Oct 30 21:44:36.742092 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:36.741649    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r" event={"ID":"dd00fc40-6246-45ea-abf3-4c9e8ab142f3","Type":"ContainerDied","Data":"b5ac9b91d24402c89b780e29d2e6cda9d796151ef079ce7e93fa7042e5f84ccf"}
Oct 30 21:44:36.742092 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:36.742032    5713 scope.go:122] "RemoveContainer" containerID="b5ac9b91d24402c89b780e29d2e6cda9d796151ef079ce7e93fa7042e5f84ccf"
Oct 30 21:44:36.743235 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:36.743208    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:44:36.743302 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:36.743287    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:44:36.743340 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:36.743322    5713 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:44:37.764448 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:37.764406    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r" event={"ID":"dd00fc40-6246-45ea-abf3-4c9e8ab142f3","Type":"ContainerStarted","Data":"116e3e9d598e22c488f49e2f6c4674b622011760a3f74a873531db91861b1791"}
Oct 30 21:44:38.394648 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:38.394615    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r"
Oct 30 21:44:38.395951 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:38.395925    5713 patch_prober.go:29] interesting pod/e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" start-of-body=
Oct 30 21:44:38.396068 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:38.395984    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r" podUID="dd00fc40-6246-45ea-abf3-4c9e8ab142f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused"
Oct 30 21:44:39.394674 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:39.394632    5713 patch_prober.go:29] interesting pod/e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" start-of-body=
Oct 30 21:44:39.395230 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:39.394688    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r" podUID="dd00fc40-6246-45ea-abf3-4c9e8ab142f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused"
Oct 30 21:44:40.394206 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:40.394169    5713 patch_prober.go:29] interesting pod/e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" start-of-body=
Oct 30 21:44:40.394432 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:40.394232    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r" podUID="dd00fc40-6246-45ea-abf3-4c9e8ab142f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused"
Oct 30 21:44:40.622158 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:40.622125    5713 scope.go:122] "RemoveContainer" containerID="595ab2f0b78d6d34ba0a63607f8cd23bb284bab412a74651af88284dc71dcd24"
Oct 30 21:44:41.393985 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:41.393947    5713 patch_prober.go:29] interesting pod/e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" start-of-body=
Oct 30 21:44:41.394174 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:41.394026    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r" podUID="dd00fc40-6246-45ea-abf3-4c9e8ab142f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused"
Oct 30 21:44:41.591415 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:41.591380    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r"
Oct 30 21:44:42.394512 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:42.394462    5713 patch_prober.go:29] interesting pod/e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" start-of-body=
Oct 30 21:44:42.395102 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:42.394524    5713 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r" podUID="dd00fc40-6246-45ea-abf3-4c9e8ab142f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused"
Oct 30 21:44:43.398698 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:43.398671    5713 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r"
Oct 30 21:44:43.399686 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:43.399665    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r"
Oct 30 21:44:50.050688 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:50.050465    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16fe7dab-ftjwv"]
Oct 30 21:44:50.098352 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:50.098316    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16fe7dab-ftjwv"]
Oct 30 21:44:51.061606 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:44:51.061574    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f40a5e4f-c02f-4d81-9041-11d3947d26cf" path="/var/lib/kubelet/pods/f40a5e4f-c02f-4d81-9041-11d3947d26cf/volumes"
Oct 30 21:44:54.458126 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:44:54.458091    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:45:00.163158 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:00.163129    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-0e8a1a-29889945-npcwl"]
Oct 30 21:45:00.166769 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:00.166752    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889945-kf4vt"]
Oct 30 21:45:00.166891 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:00.166872    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-0e8a1a-29889945-npcwl"
Oct 30 21:45:00.170076 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:00.170059    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-kf4vt"
Oct 30 21:45:00.270184 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:00.270141    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:45:00.270184 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:00.270170    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:45:00.270437 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:00.270321    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rwj5h\""
Oct 30 21:45:00.272324 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:00.272299    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-0e8a1a-29889945-npcwl"]
Oct 30 21:45:00.274175 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:00.274151    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889945-kf4vt"]
Oct 30 21:45:00.288094 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:00.288065    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06e319fe-a324-4617-993e-018ac19deabb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-authcfg-http-0e8a1a-29889945-npcwl\" (UID: \"06e319fe-a324-4617-993e-018ac19deabb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-0e8a1a-29889945-npcwl"
Oct 30 21:45:00.288188 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:00.288110    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-775rl\" (UniqueName: \"kubernetes.io/projected/04136bce-8b4c-43d6-80d1-beab8a8a041f-kube-api-access-775rl\") pod \"maas-api-key-cleanup-29889945-kf4vt\" (UID: \"04136bce-8b4c-43d6-80d1-beab8a8a041f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-kf4vt"
Oct 30 21:45:00.288188 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:00.288128    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04136bce-8b4c-43d6-80d1-beab8a8a041f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889945-kf4vt\" (UID: \"04136bce-8b4c-43d6-80d1-beab8a8a041f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-kf4vt"
Oct 30 21:45:00.288282 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:00.288265    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs94n\" (UniqueName: \"kubernetes.io/projected/06e319fe-a324-4617-993e-018ac19deabb-kube-api-access-fs94n\") pod \"maas-api-key-cleanup-e2e-authcfg-http-0e8a1a-29889945-npcwl\" (UID: \"06e319fe-a324-4617-993e-018ac19deabb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-0e8a1a-29889945-npcwl"
Oct 30 21:45:00.389619 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:00.389581    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06e319fe-a324-4617-993e-018ac19deabb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-authcfg-http-0e8a1a-29889945-npcwl\" (UID: \"06e319fe-a324-4617-993e-018ac19deabb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-0e8a1a-29889945-npcwl"
Oct 30 21:45:00.389619 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:00.389633    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-775rl\" (UniqueName: \"kubernetes.io/projected/04136bce-8b4c-43d6-80d1-beab8a8a041f-kube-api-access-775rl\") pod \"maas-api-key-cleanup-29889945-kf4vt\" (UID: \"04136bce-8b4c-43d6-80d1-beab8a8a041f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-kf4vt"
Oct 30 21:45:00.389915 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:00.389651    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04136bce-8b4c-43d6-80d1-beab8a8a041f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889945-kf4vt\" (UID: \"04136bce-8b4c-43d6-80d1-beab8a8a041f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-kf4vt"
Oct 30 21:45:00.389915 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:00.389710    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs94n\" (UniqueName: \"kubernetes.io/projected/06e319fe-a324-4617-993e-018ac19deabb-kube-api-access-fs94n\") pod \"maas-api-key-cleanup-e2e-authcfg-http-0e8a1a-29889945-npcwl\" (UID: \"06e319fe-a324-4617-993e-018ac19deabb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-0e8a1a-29889945-npcwl"
Oct 30 21:45:00.390448 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:00.390413    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04136bce-8b4c-43d6-80d1-beab8a8a041f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889945-kf4vt\" (UID: \"04136bce-8b4c-43d6-80d1-beab8a8a041f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-kf4vt"
Oct 30 21:45:00.390616 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:00.390469    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06e319fe-a324-4617-993e-018ac19deabb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-authcfg-http-0e8a1a-29889945-npcwl\" (UID: \"06e319fe-a324-4617-993e-018ac19deabb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-0e8a1a-29889945-npcwl"
Oct 30 21:45:00.448436 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:00.448401    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-775rl\" (UniqueName: \"kubernetes.io/projected/04136bce-8b4c-43d6-80d1-beab8a8a041f-kube-api-access-775rl\") pod \"maas-api-key-cleanup-29889945-kf4vt\" (UID: \"04136bce-8b4c-43d6-80d1-beab8a8a041f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-kf4vt"
Oct 30 21:45:00.450639 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:00.450614    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs94n\" (UniqueName: \"kubernetes.io/projected/06e319fe-a324-4617-993e-018ac19deabb-kube-api-access-fs94n\") pod \"maas-api-key-cleanup-e2e-authcfg-http-0e8a1a-29889945-npcwl\" (UID: \"06e319fe-a324-4617-993e-018ac19deabb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-0e8a1a-29889945-npcwl"
Oct 30 21:45:00.481319 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:00.481295    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-0e8a1a-29889945-npcwl"
Oct 30 21:45:00.486455 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:00.486438    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-kf4vt"
Oct 30 21:45:00.856304 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:00.856275    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-0e8a1a-29889945-npcwl"]
Oct 30 21:45:00.858010 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:45:00.857983    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06e319fe_a324_4617_993e_018ac19deabb.slice/crio-8bb9c62c1047ede9ce4023e813838a859aedb292856d59b39e27961d3b4e4f28 WatchSource:0}: Error finding container 8bb9c62c1047ede9ce4023e813838a859aedb292856d59b39e27961d3b4e4f28: Status 404 returned error can't find the container with id 8bb9c62c1047ede9ce4023e813838a859aedb292856d59b39e27961d3b4e4f28
Oct 30 21:45:00.893697 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:00.893673    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-0e8a1a-29889945-npcwl" event={"ID":"06e319fe-a324-4617-993e-018ac19deabb","Type":"ContainerStarted","Data":"8bb9c62c1047ede9ce4023e813838a859aedb292856d59b39e27961d3b4e4f28"}
Oct 30 21:45:00.907507 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:00.907441    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889945-kf4vt"]
Oct 30 21:45:00.909893 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:45:00.909868    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04136bce_8b4c_43d6_80d1_beab8a8a041f.slice/crio-a9f011ed6e9825ae8b589f91160aa88d7d9c1e4fd9bfbdf63a8d520d6c9b846a WatchSource:0}: Error finding container a9f011ed6e9825ae8b589f91160aa88d7d9c1e4fd9bfbdf63a8d520d6c9b846a: Status 404 returned error can't find the container with id a9f011ed6e9825ae8b589f91160aa88d7d9c1e4fd9bfbdf63a8d520d6c9b846a
Oct 30 21:45:01.901324 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:01.901234    5713 generic.go:356] "Generic (PLEG): container finished" podID="04136bce-8b4c-43d6-80d1-beab8a8a041f" containerID="0ecff14e69d800b616d92253c9988466844068de9c0530b1207919767eb6c1a9" exitCode=60
Oct 30 21:45:01.901324 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:01.901285    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-kf4vt" event={"ID":"04136bce-8b4c-43d6-80d1-beab8a8a041f","Type":"ContainerDied","Data":"0ecff14e69d800b616d92253c9988466844068de9c0530b1207919767eb6c1a9"}
Oct 30 21:45:01.901872 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:01.901373    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-kf4vt" event={"ID":"04136bce-8b4c-43d6-80d1-beab8a8a041f","Type":"ContainerStarted","Data":"a9f011ed6e9825ae8b589f91160aa88d7d9c1e4fd9bfbdf63a8d520d6c9b846a"}
Oct 30 21:45:01.901872 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:01.901635    5713 scope.go:122] "RemoveContainer" containerID="0ecff14e69d800b616d92253c9988466844068de9c0530b1207919767eb6c1a9"
Oct 30 21:45:01.903244 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:01.903142    5713 generic.go:356] "Generic (PLEG): container finished" podID="06e319fe-a324-4617-993e-018ac19deabb" containerID="5e38b1d0b38e4b9b3849688b10cb386c4306b33c03ad6b5b7ca31db6bb2a5ea6" exitCode=60
Oct 30 21:45:01.903244 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:01.903209    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-0e8a1a-29889945-npcwl" event={"ID":"06e319fe-a324-4617-993e-018ac19deabb","Type":"ContainerDied","Data":"5e38b1d0b38e4b9b3849688b10cb386c4306b33c03ad6b5b7ca31db6bb2a5ea6"}
Oct 30 21:45:01.903470 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:01.903457    5713 scope.go:122] "RemoveContainer" containerID="5e38b1d0b38e4b9b3849688b10cb386c4306b33c03ad6b5b7ca31db6bb2a5ea6"
Oct 30 21:45:02.911153 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:02.911118    5713 generic.go:356] "Generic (PLEG): container finished" podID="06e319fe-a324-4617-993e-018ac19deabb" containerID="2bbbb1981daf0c954949316133da1601a5f02b4f8c72489eafa306baeb044e9e" exitCode=60
Oct 30 21:45:02.911595 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:02.911208    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-0e8a1a-29889945-npcwl" event={"ID":"06e319fe-a324-4617-993e-018ac19deabb","Type":"ContainerDied","Data":"2bbbb1981daf0c954949316133da1601a5f02b4f8c72489eafa306baeb044e9e"}
Oct 30 21:45:02.911595 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:02.911273    5713 scope.go:122] "RemoveContainer" containerID="5e38b1d0b38e4b9b3849688b10cb386c4306b33c03ad6b5b7ca31db6bb2a5ea6"
Oct 30 21:45:02.911595 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:02.911532    5713 scope.go:122] "RemoveContainer" containerID="2bbbb1981daf0c954949316133da1601a5f02b4f8c72489eafa306baeb044e9e"
Oct 30 21:45:02.911924 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:45:02.911894    5713 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-authcfg-http-0e8a1a-29889945-npcwl_odh-ai-gateway-infra(06e319fe-a324-4617-993e-018ac19deabb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-0e8a1a-29889945-npcwl" podUID="06e319fe-a324-4617-993e-018ac19deabb"
Oct 30 21:45:02.913539 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:02.913519    5713 generic.go:356] "Generic (PLEG): container finished" podID="04136bce-8b4c-43d6-80d1-beab8a8a041f" containerID="7e787d547813e8f45d75e39d4cec480c2e834979b8448760f2a1e0ef9ea2d72f" exitCode=60
Oct 30 21:45:02.913643 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:02.913565    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-kf4vt" event={"ID":"04136bce-8b4c-43d6-80d1-beab8a8a041f","Type":"ContainerDied","Data":"7e787d547813e8f45d75e39d4cec480c2e834979b8448760f2a1e0ef9ea2d72f"}
Oct 30 21:45:02.913845 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:02.913830    5713 scope.go:122] "RemoveContainer" containerID="7e787d547813e8f45d75e39d4cec480c2e834979b8448760f2a1e0ef9ea2d72f"
Oct 30 21:45:02.914084 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:45:02.914066    5713 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889945-kf4vt_odh-ai-gateway-infra(04136bce-8b4c-43d6-80d1-beab8a8a041f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-kf4vt" podUID="04136bce-8b4c-43d6-80d1-beab8a8a041f"
Oct 30 21:45:02.923076 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:02.923055    5713 scope.go:122] "RemoveContainer" containerID="0ecff14e69d800b616d92253c9988466844068de9c0530b1207919767eb6c1a9"
Oct 30 21:45:03.924391 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:03.924362    5713 scope.go:122] "RemoveContainer" containerID="7e787d547813e8f45d75e39d4cec480c2e834979b8448760f2a1e0ef9ea2d72f"
Oct 30 21:45:03.925723 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:45:03.925688    5713 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889945-kf4vt_odh-ai-gateway-infra(04136bce-8b4c-43d6-80d1-beab8a8a041f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-kf4vt" podUID="04136bce-8b4c-43d6-80d1-beab8a8a041f"
Oct 30 21:45:03.926845 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:03.926753    5713 scope.go:122] "RemoveContainer" containerID="2bbbb1981daf0c954949316133da1601a5f02b4f8c72489eafa306baeb044e9e"
Oct 30 21:45:03.927105 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:45:03.927080    5713 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-authcfg-http-0e8a1a-29889945-npcwl_odh-ai-gateway-infra(06e319fe-a324-4617-993e-018ac19deabb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-0e8a1a-29889945-npcwl" podUID="06e319fe-a324-4617-993e-018ac19deabb"
Oct 30 21:45:14.056309 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:14.056272    5713 scope.go:122] "RemoveContainer" containerID="2bbbb1981daf0c954949316133da1601a5f02b4f8c72489eafa306baeb044e9e"
Oct 30 21:45:14.991876 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:14.991840    5713 generic.go:356] "Generic (PLEG): container finished" podID="06e319fe-a324-4617-993e-018ac19deabb" containerID="bf5d2d4d7c2172c0557e3578931027e5c2a8e41e903803bfa01bbb4f96e16660" exitCode=60
Oct 30 21:45:14.991876 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:14.991877    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-0e8a1a-29889945-npcwl" event={"ID":"06e319fe-a324-4617-993e-018ac19deabb","Type":"ContainerDied","Data":"bf5d2d4d7c2172c0557e3578931027e5c2a8e41e903803bfa01bbb4f96e16660"}
Oct 30 21:45:14.992140 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:14.991924    5713 scope.go:122] "RemoveContainer" containerID="2bbbb1981daf0c954949316133da1601a5f02b4f8c72489eafa306baeb044e9e"
Oct 30 21:45:14.992310 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:14.992290    5713 scope.go:122] "RemoveContainer" containerID="bf5d2d4d7c2172c0557e3578931027e5c2a8e41e903803bfa01bbb4f96e16660"
Oct 30 21:45:14.992566 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:45:14.992545    5713 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-authcfg-http-0e8a1a-29889945-npcwl_odh-ai-gateway-infra(06e319fe-a324-4617-993e-018ac19deabb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-0e8a1a-29889945-npcwl" podUID="06e319fe-a324-4617-993e-018ac19deabb"
Oct 30 21:45:16.056332 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:16.056296    5713 scope.go:122] "RemoveContainer" containerID="7e787d547813e8f45d75e39d4cec480c2e834979b8448760f2a1e0ef9ea2d72f"
Oct 30 21:45:16.117348 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:16.117320    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-0e8a1a-29889945-npcwl"]
Oct 30 21:45:16.260340 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:16.260154    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-0e8a1a-29889945-npcwl"
Oct 30 21:45:16.348446 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:16.348419    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06e319fe-a324-4617-993e-018ac19deabb-kube-api-access-fs94n\" (UniqueName: \"kubernetes.io/projected/06e319fe-a324-4617-993e-018ac19deabb-kube-api-access-fs94n\") pod \"06e319fe-a324-4617-993e-018ac19deabb\" (UID: \"06e319fe-a324-4617-993e-018ac19deabb\") "
Oct 30 21:45:16.348577 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:16.348454    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06e319fe-a324-4617-993e-018ac19deabb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06e319fe-a324-4617-993e-018ac19deabb-maas-api-service-ca\") pod \"06e319fe-a324-4617-993e-018ac19deabb\" (UID: \"06e319fe-a324-4617-993e-018ac19deabb\") "
Oct 30 21:45:16.348861 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:16.348840    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06e319fe-a324-4617-993e-018ac19deabb-maas-api-service-ca" pod "06e319fe-a324-4617-993e-018ac19deabb" (UID: "06e319fe-a324-4617-993e-018ac19deabb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:45:16.350648 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:16.350623    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06e319fe-a324-4617-993e-018ac19deabb-kube-api-access-fs94n" pod "06e319fe-a324-4617-993e-018ac19deabb" (UID: "06e319fe-a324-4617-993e-018ac19deabb"). InnerVolumeSpecName "kube-api-access-fs94n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:16.450247 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:16.450214    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fs94n\" (UniqueName: \"kubernetes.io/projected/06e319fe-a324-4617-993e-018ac19deabb-kube-api-access-fs94n\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:45:16.450247 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:16.450241    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06e319fe-a324-4617-993e-018ac19deabb-maas-api-service-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:45:17.006569 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:17.006538    5713 generic.go:356] "Generic (PLEG): container finished" podID="04136bce-8b4c-43d6-80d1-beab8a8a041f" containerID="a7f5589ac2aac7575df87931e7d41bbacbe18e1d6c2daf12a1ab4e5375149c99" exitCode=60
Oct 30 21:45:17.006759 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:17.006601    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-kf4vt" event={"ID":"04136bce-8b4c-43d6-80d1-beab8a8a041f","Type":"ContainerDied","Data":"a7f5589ac2aac7575df87931e7d41bbacbe18e1d6c2daf12a1ab4e5375149c99"}
Oct 30 21:45:17.006759 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:17.006654    5713 scope.go:122] "RemoveContainer" containerID="7e787d547813e8f45d75e39d4cec480c2e834979b8448760f2a1e0ef9ea2d72f"
Oct 30 21:45:17.007097 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:17.007081    5713 scope.go:122] "RemoveContainer" containerID="a7f5589ac2aac7575df87931e7d41bbacbe18e1d6c2daf12a1ab4e5375149c99"
Oct 30 21:45:17.007365 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:45:17.007343    5713 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889945-kf4vt_odh-ai-gateway-infra(04136bce-8b4c-43d6-80d1-beab8a8a041f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-kf4vt" podUID="04136bce-8b4c-43d6-80d1-beab8a8a041f"
Oct 30 21:45:17.008403 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:17.008385    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-0e8a1a-29889945-npcwl"
Oct 30 21:45:17.008519 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:17.008385    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-0e8a1a-29889945-npcwl" event={"ID":"06e319fe-a324-4617-993e-018ac19deabb","Type":"ContainerDied","Data":"8bb9c62c1047ede9ce4023e813838a859aedb292856d59b39e27961d3b4e4f28"}
Oct 30 21:45:17.021178 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:17.021161    5713 scope.go:122] "RemoveContainer" containerID="bf5d2d4d7c2172c0557e3578931027e5c2a8e41e903803bfa01bbb4f96e16660"
Oct 30 21:45:17.287511 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:17.287474    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-0e8a1a-29889945-npcwl"]
Oct 30 21:45:17.293483 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:17.293459    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-0e8a1a-29889945-npcwl"]
Oct 30 21:45:18.131401 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:18.131368    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889945-kf4vt"]
Oct 30 21:45:18.269735 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:18.269710    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-kf4vt"
Oct 30 21:45:18.368947 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:18.368903    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04136bce-8b4c-43d6-80d1-beab8a8a041f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04136bce-8b4c-43d6-80d1-beab8a8a041f-maas-api-service-ca\") pod \"04136bce-8b4c-43d6-80d1-beab8a8a041f\" (UID: \"04136bce-8b4c-43d6-80d1-beab8a8a041f\") "
Oct 30 21:45:18.368947 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:18.368953    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04136bce-8b4c-43d6-80d1-beab8a8a041f-kube-api-access-775rl\" (UniqueName: \"kubernetes.io/projected/04136bce-8b4c-43d6-80d1-beab8a8a041f-kube-api-access-775rl\") pod \"04136bce-8b4c-43d6-80d1-beab8a8a041f\" (UID: \"04136bce-8b4c-43d6-80d1-beab8a8a041f\") "
Oct 30 21:45:18.369390 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:18.369280    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04136bce-8b4c-43d6-80d1-beab8a8a041f-maas-api-service-ca" pod "04136bce-8b4c-43d6-80d1-beab8a8a041f" (UID: "04136bce-8b4c-43d6-80d1-beab8a8a041f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:45:18.371382 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:18.371357    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04136bce-8b4c-43d6-80d1-beab8a8a041f-kube-api-access-775rl" pod "04136bce-8b4c-43d6-80d1-beab8a8a041f" (UID: "04136bce-8b4c-43d6-80d1-beab8a8a041f"). InnerVolumeSpecName "kube-api-access-775rl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:18.469952 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:18.469919    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04136bce-8b4c-43d6-80d1-beab8a8a041f-maas-api-service-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:45:18.469952 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:18.469950    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-775rl\" (UniqueName: \"kubernetes.io/projected/04136bce-8b4c-43d6-80d1-beab8a8a041f-kube-api-access-775rl\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:45:19.025485 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:19.025449    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-kf4vt" event={"ID":"04136bce-8b4c-43d6-80d1-beab8a8a041f","Type":"ContainerDied","Data":"a9f011ed6e9825ae8b589f91160aa88d7d9c1e4fd9bfbdf63a8d520d6c9b846a"}
Oct 30 21:45:19.025485 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:19.025487    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-kf4vt"
Oct 30 21:45:19.025738 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:19.025513    5713 scope.go:122] "RemoveContainer" containerID="a7f5589ac2aac7575df87931e7d41bbacbe18e1d6c2daf12a1ab4e5375149c99"
Oct 30 21:45:19.062396 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:19.062366    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06e319fe-a324-4617-993e-018ac19deabb" path="/var/lib/kubelet/pods/06e319fe-a324-4617-993e-018ac19deabb/volumes"
Oct 30 21:45:19.189684 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:19.189651    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889945-kf4vt"]
Oct 30 21:45:19.239191 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:19.239156    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889945-kf4vt"]
Oct 30 21:45:21.062010 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:21.061975    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04136bce-8b4c-43d6-80d1-beab8a8a041f" path="/var/lib/kubelet/pods/04136bce-8b4c-43d6-80d1-beab8a8a041f/volumes"
Oct 30 21:45:24.457834 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:45:24.457776    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:45:54.459191 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:45:54.459147    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:45:59.140562 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:59.140528    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0e8a1a-hx8wm"]
Oct 30 21:45:59.141135 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:59.141116    5713 cpu_manager.go:414] "containerMap: removing container" podUID="04136bce-8b4c-43d6-80d1-beab8a8a041f" containerName="cleanup"
Oct 30 21:45:59.141234 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:59.141138    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04136bce-8b4c-43d6-80d1-beab8a8a041f" containerName="cleanup"
Oct 30 21:45:59.141234 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:59.141168    5713 cpu_manager.go:414] "containerMap: removing container" podUID="06e319fe-a324-4617-993e-018ac19deabb" containerName="cleanup"
Oct 30 21:45:59.141234 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:59.141175    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06e319fe-a324-4617-993e-018ac19deabb" containerName="cleanup"
Oct 30 21:45:59.141234 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:59.141202    5713 cpu_manager.go:414] "containerMap: removing container" podUID="04136bce-8b4c-43d6-80d1-beab8a8a041f" containerName="cleanup"
Oct 30 21:45:59.141234 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:59.141209    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04136bce-8b4c-43d6-80d1-beab8a8a041f" containerName="cleanup"
Oct 30 21:45:59.141234 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:59.141230    5713 cpu_manager.go:414] "containerMap: removing container" podUID="06e319fe-a324-4617-993e-018ac19deabb" containerName="cleanup"
Oct 30 21:45:59.141573 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:59.141238    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06e319fe-a324-4617-993e-018ac19deabb" containerName="cleanup"
Oct 30 21:45:59.141573 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:59.141257    5713 cpu_manager.go:414] "containerMap: removing container" podUID="06e319fe-a324-4617-993e-018ac19deabb" containerName="cleanup"
Oct 30 21:45:59.141573 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:59.141264    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06e319fe-a324-4617-993e-018ac19deabb" containerName="cleanup"
Oct 30 21:45:59.141573 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:59.141361    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="04136bce-8b4c-43d6-80d1-beab8a8a041f" containerName="cleanup"
Oct 30 21:45:59.141573 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:59.141377    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="04136bce-8b4c-43d6-80d1-beab8a8a041f" containerName="cleanup"
Oct 30 21:45:59.141573 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:59.141387    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="06e319fe-a324-4617-993e-018ac19deabb" containerName="cleanup"
Oct 30 21:45:59.141573 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:59.141397    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="06e319fe-a324-4617-993e-018ac19deabb" containerName="cleanup"
Oct 30 21:45:59.141573 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:59.141411    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="06e319fe-a324-4617-993e-018ac19deabb" containerName="cleanup"
Oct 30 21:45:59.144653 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:59.144633    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0e8a1a-hx8wm"
Oct 30 21:45:59.184999 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:59.184970    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/513eb7c6-ccd2-4d25-814f-cea6aa21b921-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-0e8a1a-hx8wm\" (UID: \"513eb7c6-ccd2-4d25-814f-cea6aa21b921\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0e8a1a-hx8wm"
Oct 30 21:45:59.185125 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:59.185109    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/513eb7c6-ccd2-4d25-814f-cea6aa21b921-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-0e8a1a-hx8wm\" (UID: \"513eb7c6-ccd2-4d25-814f-cea6aa21b921\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0e8a1a-hx8wm"
Oct 30 21:45:59.185177 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:59.185164    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8hxh\" (UniqueName: \"kubernetes.io/projected/513eb7c6-ccd2-4d25-814f-cea6aa21b921-kube-api-access-k8hxh\") pod \"maas-api-revoke-keys-e2e-authcfg-http-0e8a1a-hx8wm\" (UID: \"513eb7c6-ccd2-4d25-814f-cea6aa21b921\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0e8a1a-hx8wm"
Oct 30 21:45:59.198403 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:59.198380    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:45:59.248367 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:59.248341    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:45:59.248367 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:59.248362    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rwj5h\""
Oct 30 21:45:59.248367 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:59.248341    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:45:59.251367 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:59.251343    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0e8a1a-hx8wm"]
Oct 30 21:45:59.286097 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:59.286071    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/513eb7c6-ccd2-4d25-814f-cea6aa21b921-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-0e8a1a-hx8wm\" (UID: \"513eb7c6-ccd2-4d25-814f-cea6aa21b921\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0e8a1a-hx8wm"
Oct 30 21:45:59.286223 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:59.286124    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8hxh\" (UniqueName: \"kubernetes.io/projected/513eb7c6-ccd2-4d25-814f-cea6aa21b921-kube-api-access-k8hxh\") pod \"maas-api-revoke-keys-e2e-authcfg-http-0e8a1a-hx8wm\" (UID: \"513eb7c6-ccd2-4d25-814f-cea6aa21b921\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0e8a1a-hx8wm"
Oct 30 21:45:59.286223 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:59.286174    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/513eb7c6-ccd2-4d25-814f-cea6aa21b921-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-0e8a1a-hx8wm\" (UID: \"513eb7c6-ccd2-4d25-814f-cea6aa21b921\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0e8a1a-hx8wm"
Oct 30 21:45:59.286914 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:59.286895    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/513eb7c6-ccd2-4d25-814f-cea6aa21b921-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-0e8a1a-hx8wm\" (UID: \"513eb7c6-ccd2-4d25-814f-cea6aa21b921\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0e8a1a-hx8wm"
Oct 30 21:45:59.288683 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:59.288663    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/513eb7c6-ccd2-4d25-814f-cea6aa21b921-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-0e8a1a-hx8wm\" (UID: \"513eb7c6-ccd2-4d25-814f-cea6aa21b921\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0e8a1a-hx8wm"
Oct 30 21:45:59.343003 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:59.342971    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8hxh\" (UniqueName: \"kubernetes.io/projected/513eb7c6-ccd2-4d25-814f-cea6aa21b921-kube-api-access-k8hxh\") pod \"maas-api-revoke-keys-e2e-authcfg-http-0e8a1a-hx8wm\" (UID: \"513eb7c6-ccd2-4d25-814f-cea6aa21b921\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0e8a1a-hx8wm"
Oct 30 21:45:59.457668 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:59.457646    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0e8a1a-hx8wm"
Oct 30 21:45:59.832420 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:45:59.832390    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0e8a1a-hx8wm"]
Oct 30 21:45:59.834556 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:45:59.834517    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod513eb7c6_ccd2_4d25_814f_cea6aa21b921.slice/crio-602137e915a9edc6f120ec454532498df8ebe3062a644614c91388fd3cdf4013 WatchSource:0}: Error finding container 602137e915a9edc6f120ec454532498df8ebe3062a644614c91388fd3cdf4013: Status 404 returned error can't find the container with id 602137e915a9edc6f120ec454532498df8ebe3062a644614c91388fd3cdf4013
Oct 30 21:46:00.258772 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:00.258737    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0e8a1a-hx8wm" event={"ID":"513eb7c6-ccd2-4d25-814f-cea6aa21b921","Type":"ContainerStarted","Data":"cdb95b7bda571961f7cbefafc545e98c28c46648156140c24bcbd8c5b65d3465"}
Oct 30 21:46:00.258772 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:00.258798    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0e8a1a-hx8wm" event={"ID":"513eb7c6-ccd2-4d25-814f-cea6aa21b921","Type":"ContainerStarted","Data":"602137e915a9edc6f120ec454532498df8ebe3062a644614c91388fd3cdf4013"}
Oct 30 21:46:00.376940 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:00.376880    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0e8a1a-hx8wm" podStartSLOduration=1.37686276 podStartE2EDuration="1.37686276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:00.376426284 +0000 UTC m=+4043.441202688" watchObservedRunningTime="2026-10-30 21:46:00.37686276 +0000 UTC m=+4043.441639158"
Oct 30 21:46:01.265563 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:01.265530    5713 generic.go:356] "Generic (PLEG): container finished" podID="513eb7c6-ccd2-4d25-814f-cea6aa21b921" containerID="cdb95b7bda571961f7cbefafc545e98c28c46648156140c24bcbd8c5b65d3465" exitCode=0
Oct 30 21:46:01.266016 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:01.265617    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0e8a1a-hx8wm" event={"ID":"513eb7c6-ccd2-4d25-814f-cea6aa21b921","Type":"ContainerDied","Data":"cdb95b7bda571961f7cbefafc545e98c28c46648156140c24bcbd8c5b65d3465"}
Oct 30 21:46:02.409929 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:02.409901    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0e8a1a-hx8wm"
Oct 30 21:46:02.519347 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:02.519312    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/513eb7c6-ccd2-4d25-814f-cea6aa21b921-kube-api-access-k8hxh\" (UniqueName: \"kubernetes.io/projected/513eb7c6-ccd2-4d25-814f-cea6aa21b921-kube-api-access-k8hxh\") pod \"513eb7c6-ccd2-4d25-814f-cea6aa21b921\" (UID: \"513eb7c6-ccd2-4d25-814f-cea6aa21b921\") "
Oct 30 21:46:02.519551 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:02.519454    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/513eb7c6-ccd2-4d25-814f-cea6aa21b921-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/513eb7c6-ccd2-4d25-814f-cea6aa21b921-maas-api-xks-ca\") pod \"513eb7c6-ccd2-4d25-814f-cea6aa21b921\" (UID: \"513eb7c6-ccd2-4d25-814f-cea6aa21b921\") "
Oct 30 21:46:02.519551 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:02.519488    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/513eb7c6-ccd2-4d25-814f-cea6aa21b921-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/513eb7c6-ccd2-4d25-814f-cea6aa21b921-maas-api-openshift-ca\") pod \"513eb7c6-ccd2-4d25-814f-cea6aa21b921\" (UID: \"513eb7c6-ccd2-4d25-814f-cea6aa21b921\") "
Oct 30 21:46:02.519950 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:02.519925    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/513eb7c6-ccd2-4d25-814f-cea6aa21b921-maas-api-openshift-ca" pod "513eb7c6-ccd2-4d25-814f-cea6aa21b921" (UID: "513eb7c6-ccd2-4d25-814f-cea6aa21b921"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:46:02.521809 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:02.521712    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/513eb7c6-ccd2-4d25-814f-cea6aa21b921-maas-api-xks-ca" pod "513eb7c6-ccd2-4d25-814f-cea6aa21b921" (UID: "513eb7c6-ccd2-4d25-814f-cea6aa21b921"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:46:02.521809 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:02.521761    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/513eb7c6-ccd2-4d25-814f-cea6aa21b921-kube-api-access-k8hxh" pod "513eb7c6-ccd2-4d25-814f-cea6aa21b921" (UID: "513eb7c6-ccd2-4d25-814f-cea6aa21b921"). InnerVolumeSpecName "kube-api-access-k8hxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:46:02.620371 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:02.620336    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8hxh\" (UniqueName: \"kubernetes.io/projected/513eb7c6-ccd2-4d25-814f-cea6aa21b921-kube-api-access-k8hxh\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:46:02.620371 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:02.620370    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/513eb7c6-ccd2-4d25-814f-cea6aa21b921-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:46:02.620371 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:02.620384    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/513eb7c6-ccd2-4d25-814f-cea6aa21b921-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:46:03.277238 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:03.277204    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0e8a1a-hx8wm" event={"ID":"513eb7c6-ccd2-4d25-814f-cea6aa21b921","Type":"ContainerDied","Data":"602137e915a9edc6f120ec454532498df8ebe3062a644614c91388fd3cdf4013"}
Oct 30 21:46:03.277238 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:03.277223    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0e8a1a-hx8wm"
Oct 30 21:46:03.277459 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:03.277254    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="602137e915a9edc6f120ec454532498df8ebe3062a644614c91388fd3cdf4013"
Oct 30 21:46:10.684846 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:10.684807    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-0e8a1a-848bfd56cb-w9ph6"]
Oct 30 21:46:15.686055 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:15.685987    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-0e8a1a-848bfd56cb-w9ph6" podUID="0f01f7e7-94ab-4197-9441-c96b1a6577af" containerName="payload-processing" containerID="cri-o://d51c8259bf9a2c2762004e1789b76fe476cdf0d832ba9a3cde15e2e8b2b3fdec" gracePeriod=25
Oct 30 21:46:15.946229 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:15.946170    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-0e8a1a-848bfd56cb-w9ph6"
Oct 30 21:46:16.058912 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:16.058870    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f01f7e7-94ab-4197-9441-c96b1a6577af-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f01f7e7-94ab-4197-9441-c96b1a6577af-plugins-config-volume\") pod \"0f01f7e7-94ab-4197-9441-c96b1a6577af\" (UID: \"0f01f7e7-94ab-4197-9441-c96b1a6577af\") "
Oct 30 21:46:16.059128 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:16.058945    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f01f7e7-94ab-4197-9441-c96b1a6577af-kube-api-access-vt6vw\" (UniqueName: \"kubernetes.io/projected/0f01f7e7-94ab-4197-9441-c96b1a6577af-kube-api-access-vt6vw\") pod \"0f01f7e7-94ab-4197-9441-c96b1a6577af\" (UID: \"0f01f7e7-94ab-4197-9441-c96b1a6577af\") "
Oct 30 21:46:16.059308 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:16.059278    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f01f7e7-94ab-4197-9441-c96b1a6577af-plugins-config-volume" pod "0f01f7e7-94ab-4197-9441-c96b1a6577af" (UID: "0f01f7e7-94ab-4197-9441-c96b1a6577af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:46:16.059634 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:16.059614    5713 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f01f7e7-94ab-4197-9441-c96b1a6577af-plugins-config-volume\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:46:16.062005 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:16.061976    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f01f7e7-94ab-4197-9441-c96b1a6577af-kube-api-access-vt6vw" pod "0f01f7e7-94ab-4197-9441-c96b1a6577af" (UID: "0f01f7e7-94ab-4197-9441-c96b1a6577af"). InnerVolumeSpecName "kube-api-access-vt6vw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:46:16.160407 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:16.160372    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vt6vw\" (UniqueName: \"kubernetes.io/projected/0f01f7e7-94ab-4197-9441-c96b1a6577af-kube-api-access-vt6vw\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:46:16.353017 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:16.352932    5713 generic.go:356] "Generic (PLEG): container finished" podID="0f01f7e7-94ab-4197-9441-c96b1a6577af" containerID="d51c8259bf9a2c2762004e1789b76fe476cdf0d832ba9a3cde15e2e8b2b3fdec" exitCode=0
Oct 30 21:46:16.353017 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:16.352978    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-0e8a1a-848bfd56cb-w9ph6"
Oct 30 21:46:16.353242 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:16.353021    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-0e8a1a-848bfd56cb-w9ph6" event={"ID":"0f01f7e7-94ab-4197-9441-c96b1a6577af","Type":"ContainerDied","Data":"d51c8259bf9a2c2762004e1789b76fe476cdf0d832ba9a3cde15e2e8b2b3fdec"}
Oct 30 21:46:16.353242 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:16.353070    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-0e8a1a-848bfd56cb-w9ph6" event={"ID":"0f01f7e7-94ab-4197-9441-c96b1a6577af","Type":"ContainerDied","Data":"62d5f1f8b1e02429a6f9cda0985240d3d82a21a99cc5f04db7b639f8dee9ed2a"}
Oct 30 21:46:16.353242 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:16.353091    5713 scope.go:122] "RemoveContainer" containerID="d51c8259bf9a2c2762004e1789b76fe476cdf0d832ba9a3cde15e2e8b2b3fdec"
Oct 30 21:46:16.363320 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:16.363300    5713 scope.go:122] "RemoveContainer" containerID="d51c8259bf9a2c2762004e1789b76fe476cdf0d832ba9a3cde15e2e8b2b3fdec"
Oct 30 21:46:16.363647 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:46:16.363626    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d51c8259bf9a2c2762004e1789b76fe476cdf0d832ba9a3cde15e2e8b2b3fdec\": container with ID starting with d51c8259bf9a2c2762004e1789b76fe476cdf0d832ba9a3cde15e2e8b2b3fdec not found: ID does not exist" containerID="d51c8259bf9a2c2762004e1789b76fe476cdf0d832ba9a3cde15e2e8b2b3fdec"
Oct 30 21:46:16.363689 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:16.363658    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d51c8259bf9a2c2762004e1789b76fe476cdf0d832ba9a3cde15e2e8b2b3fdec"} err="failed to get container status \"d51c8259bf9a2c2762004e1789b76fe476cdf0d832ba9a3cde15e2e8b2b3fdec\": rpc error: code = NotFound desc = could not find container \"d51c8259bf9a2c2762004e1789b76fe476cdf0d832ba9a3cde15e2e8b2b3fdec\": container with ID starting with d51c8259bf9a2c2762004e1789b76fe476cdf0d832ba9a3cde15e2e8b2b3fdec not found: ID does not exist"
Oct 30 21:46:16.515832 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:16.515743    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-0e8a1a-848bfd56cb-w9ph6"]
Oct 30 21:46:16.521439 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:16.521412    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-0e8a1a-848bfd56cb-w9ph6"]
Oct 30 21:46:17.069799 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:17.069765    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f01f7e7-94ab-4197-9441-c96b1a6577af" path="/var/lib/kubelet/pods/0f01f7e7-94ab-4197-9441-c96b1a6577af/volumes"
Oct 30 21:46:23.051432 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:23.051394    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0e8a1a-hx8wm"]
Oct 30 21:46:23.100782 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:23.100746    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0e8a1a-hx8wm"]
Oct 30 21:46:24.459245 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:46:24.459210    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:46:25.062329 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:25.062289    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="513eb7c6-ccd2-4d25-814f-cea6aa21b921" path="/var/lib/kubelet/pods/513eb7c6-ccd2-4d25-814f-cea6aa21b921/volumes"
Oct 30 21:46:29.570579 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:29.570541    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r"]
Oct 30 21:46:29.571187 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:29.570869    5713 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r" podUID="dd00fc40-6246-45ea-abf3-4c9e8ab142f3" containerName="istio-proxy" containerID="cri-o://116e3e9d598e22c488f49e2f6c4674b622011760a3f74a873531db91861b1791" gracePeriod=30
Oct 30 21:46:34.850764 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:34.850703    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r"
Oct 30 21:46:34.951257 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:34.951215    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd00fc40-6246-45ea-abf3-4c9e8ab142f3-istio-token\" (UniqueName: \"kubernetes.io/projected/dd00fc40-6246-45ea-abf3-4c9e8ab142f3-istio-token\") pod \"dd00fc40-6246-45ea-abf3-4c9e8ab142f3\" (UID: \"dd00fc40-6246-45ea-abf3-4c9e8ab142f3\") "
Oct 30 21:46:34.951445 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:34.951284    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd00fc40-6246-45ea-abf3-4c9e8ab142f3-kube-api-access-dnhvs\" (UniqueName: \"kubernetes.io/projected/dd00fc40-6246-45ea-abf3-4c9e8ab142f3-kube-api-access-dnhvs\") pod \"dd00fc40-6246-45ea-abf3-4c9e8ab142f3\" (UID: \"dd00fc40-6246-45ea-abf3-4c9e8ab142f3\") "
Oct 30 21:46:34.951445 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:34.951326    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd00fc40-6246-45ea-abf3-4c9e8ab142f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd00fc40-6246-45ea-abf3-4c9e8ab142f3-workload-certs\") pod \"dd00fc40-6246-45ea-abf3-4c9e8ab142f3\" (UID: \"dd00fc40-6246-45ea-abf3-4c9e8ab142f3\") "
Oct 30 21:46:34.951445 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:34.951362    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd00fc40-6246-45ea-abf3-4c9e8ab142f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd00fc40-6246-45ea-abf3-4c9e8ab142f3-istiod-ca-cert\") pod \"dd00fc40-6246-45ea-abf3-4c9e8ab142f3\" (UID: \"dd00fc40-6246-45ea-abf3-4c9e8ab142f3\") "
Oct 30 21:46:34.951445 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:34.951422    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd00fc40-6246-45ea-abf3-4c9e8ab142f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd00fc40-6246-45ea-abf3-4c9e8ab142f3-istio-podinfo\") pod \"dd00fc40-6246-45ea-abf3-4c9e8ab142f3\" (UID: \"dd00fc40-6246-45ea-abf3-4c9e8ab142f3\") "
Oct 30 21:46:34.951687 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:34.951460    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd00fc40-6246-45ea-abf3-4c9e8ab142f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd00fc40-6246-45ea-abf3-4c9e8ab142f3-istio-data\") pod \"dd00fc40-6246-45ea-abf3-4c9e8ab142f3\" (UID: \"dd00fc40-6246-45ea-abf3-4c9e8ab142f3\") "
Oct 30 21:46:34.951687 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:34.951509    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd00fc40-6246-45ea-abf3-4c9e8ab142f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd00fc40-6246-45ea-abf3-4c9e8ab142f3-workload-socket\") pod \"dd00fc40-6246-45ea-abf3-4c9e8ab142f3\" (UID: \"dd00fc40-6246-45ea-abf3-4c9e8ab142f3\") "
Oct 30 21:46:34.951687 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:34.951571    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd00fc40-6246-45ea-abf3-4c9e8ab142f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd00fc40-6246-45ea-abf3-4c9e8ab142f3-istio-envoy\") pod \"dd00fc40-6246-45ea-abf3-4c9e8ab142f3\" (UID: \"dd00fc40-6246-45ea-abf3-4c9e8ab142f3\") "
Oct 30 21:46:34.951687 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:34.951636    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd00fc40-6246-45ea-abf3-4c9e8ab142f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd00fc40-6246-45ea-abf3-4c9e8ab142f3-credential-socket\") pod \"dd00fc40-6246-45ea-abf3-4c9e8ab142f3\" (UID: \"dd00fc40-6246-45ea-abf3-4c9e8ab142f3\") "
Oct 30 21:46:34.951687 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:34.951667    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd00fc40-6246-45ea-abf3-4c9e8ab142f3-workload-certs" pod "dd00fc40-6246-45ea-abf3-4c9e8ab142f3" (UID: "dd00fc40-6246-45ea-abf3-4c9e8ab142f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:34.952042 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:34.952014    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd00fc40-6246-45ea-abf3-4c9e8ab142f3-istiod-ca-cert" pod "dd00fc40-6246-45ea-abf3-4c9e8ab142f3" (UID: "dd00fc40-6246-45ea-abf3-4c9e8ab142f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:46:34.952137 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:34.952058    5713 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd00fc40-6246-45ea-abf3-4c9e8ab142f3-workload-certs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:46:34.952314 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:34.952267    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd00fc40-6246-45ea-abf3-4c9e8ab142f3-workload-socket" pod "dd00fc40-6246-45ea-abf3-4c9e8ab142f3" (UID: "dd00fc40-6246-45ea-abf3-4c9e8ab142f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:34.952314 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:34.952296    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd00fc40-6246-45ea-abf3-4c9e8ab142f3-credential-socket" pod "dd00fc40-6246-45ea-abf3-4c9e8ab142f3" (UID: "dd00fc40-6246-45ea-abf3-4c9e8ab142f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:34.952480 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:34.952459    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd00fc40-6246-45ea-abf3-4c9e8ab142f3-istio-data" pod "dd00fc40-6246-45ea-abf3-4c9e8ab142f3" (UID: "dd00fc40-6246-45ea-abf3-4c9e8ab142f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:34.954427 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:34.954338    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd00fc40-6246-45ea-abf3-4c9e8ab142f3-kube-api-access-dnhvs" pod "dd00fc40-6246-45ea-abf3-4c9e8ab142f3" (UID: "dd00fc40-6246-45ea-abf3-4c9e8ab142f3"). InnerVolumeSpecName "kube-api-access-dnhvs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:46:34.955006 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:34.954978    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd00fc40-6246-45ea-abf3-4c9e8ab142f3-istio-token" pod "dd00fc40-6246-45ea-abf3-4c9e8ab142f3" (UID: "dd00fc40-6246-45ea-abf3-4c9e8ab142f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:46:34.955234 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:34.955209    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd00fc40-6246-45ea-abf3-4c9e8ab142f3-istio-podinfo" pod "dd00fc40-6246-45ea-abf3-4c9e8ab142f3" (UID: "dd00fc40-6246-45ea-abf3-4c9e8ab142f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:46:34.955525 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:34.955498    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd00fc40-6246-45ea-abf3-4c9e8ab142f3-istio-envoy" pod "dd00fc40-6246-45ea-abf3-4c9e8ab142f3" (UID: "dd00fc40-6246-45ea-abf3-4c9e8ab142f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:46:35.106615 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:35.106516    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnhvs\" (UniqueName: \"kubernetes.io/projected/dd00fc40-6246-45ea-abf3-4c9e8ab142f3-kube-api-access-dnhvs\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:46:35.116502 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:35.116476    5713 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd00fc40-6246-45ea-abf3-4c9e8ab142f3-istiod-ca-cert\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:46:35.116730 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:35.116714    5713 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd00fc40-6246-45ea-abf3-4c9e8ab142f3-istio-podinfo\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:46:35.116850 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:35.116829    5713 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd00fc40-6246-45ea-abf3-4c9e8ab142f3-istio-data\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:46:35.116933 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:35.116855    5713 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd00fc40-6246-45ea-abf3-4c9e8ab142f3-workload-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:46:35.116933 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:35.116870    5713 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd00fc40-6246-45ea-abf3-4c9e8ab142f3-istio-envoy\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:46:35.116933 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:35.116886    5713 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd00fc40-6246-45ea-abf3-4c9e8ab142f3-credential-socket\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:46:35.116933 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:35.116900    5713 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd00fc40-6246-45ea-abf3-4c9e8ab142f3-istio-token\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:46:35.491809 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:35.490559    5713 generic.go:356] "Generic (PLEG): container finished" podID="dd00fc40-6246-45ea-abf3-4c9e8ab142f3" containerID="116e3e9d598e22c488f49e2f6c4674b622011760a3f74a873531db91861b1791" exitCode=0
Oct 30 21:46:35.491809 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:35.490644    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r" event={"ID":"dd00fc40-6246-45ea-abf3-4c9e8ab142f3","Type":"ContainerDied","Data":"116e3e9d598e22c488f49e2f6c4674b622011760a3f74a873531db91861b1791"}
Oct 30 21:46:35.491809 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:35.490720    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r" event={"ID":"dd00fc40-6246-45ea-abf3-4c9e8ab142f3","Type":"ContainerDied","Data":"9c3754f9a3328b7af9ed5b42278c3238e2a61c4946dcac385f2f0b366dccf1c4"}
Oct 30 21:46:35.491809 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:35.490754    5713 scope.go:122] "RemoveContainer" containerID="116e3e9d598e22c488f49e2f6c4674b622011760a3f74a873531db91861b1791"
Oct 30 21:46:35.491809 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:35.491020    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r"
Oct 30 21:46:35.519275 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:35.519254    5713 scope.go:122] "RemoveContainer" containerID="b5ac9b91d24402c89b780e29d2e6cda9d796151ef079ce7e93fa7042e5f84ccf"
Oct 30 21:46:35.527662 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:35.527643    5713 scope.go:122] "RemoveContainer" containerID="116e3e9d598e22c488f49e2f6c4674b622011760a3f74a873531db91861b1791"
Oct 30 21:46:35.527951 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:46:35.527930    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"116e3e9d598e22c488f49e2f6c4674b622011760a3f74a873531db91861b1791\": container with ID starting with 116e3e9d598e22c488f49e2f6c4674b622011760a3f74a873531db91861b1791 not found: ID does not exist" containerID="116e3e9d598e22c488f49e2f6c4674b622011760a3f74a873531db91861b1791"
Oct 30 21:46:35.528005 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:35.527961    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"116e3e9d598e22c488f49e2f6c4674b622011760a3f74a873531db91861b1791"} err="failed to get container status \"116e3e9d598e22c488f49e2f6c4674b622011760a3f74a873531db91861b1791\": rpc error: code = NotFound desc = could not find container \"116e3e9d598e22c488f49e2f6c4674b622011760a3f74a873531db91861b1791\": container with ID starting with 116e3e9d598e22c488f49e2f6c4674b622011760a3f74a873531db91861b1791 not found: ID does not exist"
Oct 30 21:46:35.528005 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:35.527977    5713 scope.go:122] "RemoveContainer" containerID="b5ac9b91d24402c89b780e29d2e6cda9d796151ef079ce7e93fa7042e5f84ccf"
Oct 30 21:46:35.528236 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:46:35.528220    5713 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5ac9b91d24402c89b780e29d2e6cda9d796151ef079ce7e93fa7042e5f84ccf\": container with ID starting with b5ac9b91d24402c89b780e29d2e6cda9d796151ef079ce7e93fa7042e5f84ccf not found: ID does not exist" containerID="b5ac9b91d24402c89b780e29d2e6cda9d796151ef079ce7e93fa7042e5f84ccf"
Oct 30 21:46:35.528283 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:35.528243    5713 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5ac9b91d24402c89b780e29d2e6cda9d796151ef079ce7e93fa7042e5f84ccf"} err="failed to get container status \"b5ac9b91d24402c89b780e29d2e6cda9d796151ef079ce7e93fa7042e5f84ccf\": rpc error: code = NotFound desc = could not find container \"b5ac9b91d24402c89b780e29d2e6cda9d796151ef079ce7e93fa7042e5f84ccf\": container with ID starting with b5ac9b91d24402c89b780e29d2e6cda9d796151ef079ce7e93fa7042e5f84ccf not found: ID does not exist"
Oct 30 21:46:35.669814 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:35.669745    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r"]
Oct 30 21:46:35.701244 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:35.701056    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-0e8a1a-openshift-default-869d76b69-l458r"]
Oct 30 21:46:37.062433 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:37.062394    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd00fc40-6246-45ea-abf3-4c9e8ab142f3" path="/var/lib/kubelet/pods/dd00fc40-6246-45ea-abf3-4c9e8ab142f3/volumes"
Oct 30 21:46:40.697338 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:40.697307    5713 scope.go:122] "RemoveContainer" containerID="92425b4f6d7eb2ee00fb195316b990f56038940c34e6fa64a567a3c18f8f5c77"
Oct 30 21:46:40.706735 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:46:40.706712    5713 scope.go:122] "RemoveContainer" containerID="f5a657577e4aa9e4d5d77a38eb9bf2167043d414f8964201eef61cd4b2c5fd25"
Oct 30 21:46:54.521779 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:46:54.521741    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:47:00.150835 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:00.150777    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ftrj6"]
Oct 30 21:47:00.151367 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:00.151200    5713 cpu_manager.go:414] "containerMap: removing container" podUID="513eb7c6-ccd2-4d25-814f-cea6aa21b921" containerName="revoke-keys"
Oct 30 21:47:00.151367 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:00.151211    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="513eb7c6-ccd2-4d25-814f-cea6aa21b921" containerName="revoke-keys"
Oct 30 21:47:00.151367 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:00.151226    5713 cpu_manager.go:414] "containerMap: removing container" podUID="0f01f7e7-94ab-4197-9441-c96b1a6577af" containerName="payload-processing"
Oct 30 21:47:00.151367 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:00.151230    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f01f7e7-94ab-4197-9441-c96b1a6577af" containerName="payload-processing"
Oct 30 21:47:00.151367 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:00.151243    5713 cpu_manager.go:414] "containerMap: removing container" podUID="dd00fc40-6246-45ea-abf3-4c9e8ab142f3" containerName="istio-proxy"
Oct 30 21:47:00.151367 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:00.151248    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd00fc40-6246-45ea-abf3-4c9e8ab142f3" containerName="istio-proxy"
Oct 30 21:47:00.151367 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:00.151263    5713 cpu_manager.go:414] "containerMap: removing container" podUID="04136bce-8b4c-43d6-80d1-beab8a8a041f" containerName="cleanup"
Oct 30 21:47:00.151367 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:00.151268    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04136bce-8b4c-43d6-80d1-beab8a8a041f" containerName="cleanup"
Oct 30 21:47:00.151367 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:00.151276    5713 cpu_manager.go:414] "containerMap: removing container" podUID="dd00fc40-6246-45ea-abf3-4c9e8ab142f3" containerName="istio-proxy"
Oct 30 21:47:00.151367 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:00.151281    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd00fc40-6246-45ea-abf3-4c9e8ab142f3" containerName="istio-proxy"
Oct 30 21:47:00.151367 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:00.151337    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f01f7e7-94ab-4197-9441-c96b1a6577af" containerName="payload-processing"
Oct 30 21:47:00.151367 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:00.151347    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd00fc40-6246-45ea-abf3-4c9e8ab142f3" containerName="istio-proxy"
Oct 30 21:47:00.151367 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:00.151355    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="04136bce-8b4c-43d6-80d1-beab8a8a041f" containerName="cleanup"
Oct 30 21:47:00.151367 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:00.151364    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="513eb7c6-ccd2-4d25-814f-cea6aa21b921" containerName="revoke-keys"
Oct 30 21:47:00.154517 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:00.154501    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ftrj6"
Oct 30 21:47:00.207744 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:00.207701    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:47:00.257469 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:00.257427    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:47:00.257469 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:00.257434    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:47:00.257469 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:00.257460    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rwj5h\""
Oct 30 21:47:00.259487 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:00.259466    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ftrj6"]
Oct 30 21:47:00.276157 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:00.276130    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1c2f92a-44cb-460d-9520-367f5447415e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-ftrj6\" (UID: \"a1c2f92a-44cb-460d-9520-367f5447415e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ftrj6"
Oct 30 21:47:00.276306 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:00.276167    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48fsv\" (UniqueName: \"kubernetes.io/projected/a1c2f92a-44cb-460d-9520-367f5447415e-kube-api-access-48fsv\") pod \"maas-api-revoke-sub-f03629db-ftrj6\" (UID: \"a1c2f92a-44cb-460d-9520-367f5447415e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ftrj6"
Oct 30 21:47:00.276358 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:00.276304    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1c2f92a-44cb-460d-9520-367f5447415e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-ftrj6\" (UID: \"a1c2f92a-44cb-460d-9520-367f5447415e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ftrj6"
Oct 30 21:47:00.377850 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:00.377815    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1c2f92a-44cb-460d-9520-367f5447415e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-ftrj6\" (UID: \"a1c2f92a-44cb-460d-9520-367f5447415e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ftrj6"
Oct 30 21:47:00.378027 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:00.377886    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1c2f92a-44cb-460d-9520-367f5447415e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-ftrj6\" (UID: \"a1c2f92a-44cb-460d-9520-367f5447415e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ftrj6"
Oct 30 21:47:00.378027 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:00.377914    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48fsv\" (UniqueName: \"kubernetes.io/projected/a1c2f92a-44cb-460d-9520-367f5447415e-kube-api-access-48fsv\") pod \"maas-api-revoke-sub-f03629db-ftrj6\" (UID: \"a1c2f92a-44cb-460d-9520-367f5447415e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ftrj6"
Oct 30 21:47:00.378484 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:00.378452    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1c2f92a-44cb-460d-9520-367f5447415e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-ftrj6\" (UID: \"a1c2f92a-44cb-460d-9520-367f5447415e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ftrj6"
Oct 30 21:47:00.380350 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:00.380326    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1c2f92a-44cb-460d-9520-367f5447415e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-ftrj6\" (UID: \"a1c2f92a-44cb-460d-9520-367f5447415e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ftrj6"
Oct 30 21:47:00.435169 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:00.435147    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48fsv\" (UniqueName: \"kubernetes.io/projected/a1c2f92a-44cb-460d-9520-367f5447415e-kube-api-access-48fsv\") pod \"maas-api-revoke-sub-f03629db-ftrj6\" (UID: \"a1c2f92a-44cb-460d-9520-367f5447415e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ftrj6"
Oct 30 21:47:00.467843 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:00.467804    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ftrj6"
Oct 30 21:47:00.841604 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:00.841575    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ftrj6"]
Oct 30 21:47:00.844465 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:47:00.844432    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1c2f92a_44cb_460d_9520_367f5447415e.slice/crio-06680d55081342a4a7ed46db0c95f8f8d4ec469427733432afac3fee2f8b0a1b WatchSource:0}: Error finding container 06680d55081342a4a7ed46db0c95f8f8d4ec469427733432afac3fee2f8b0a1b: Status 404 returned error can't find the container with id 06680d55081342a4a7ed46db0c95f8f8d4ec469427733432afac3fee2f8b0a1b
Oct 30 21:47:01.655195 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:01.655156    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ftrj6" event={"ID":"a1c2f92a-44cb-460d-9520-367f5447415e","Type":"ContainerStarted","Data":"37d3b282d9b4219003f22c9764afbf245ca06f3fea95c6bf00cf0b1ba04e6ab6"}
Oct 30 21:47:01.655666 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:01.655249    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ftrj6" event={"ID":"a1c2f92a-44cb-460d-9520-367f5447415e","Type":"ContainerStarted","Data":"06680d55081342a4a7ed46db0c95f8f8d4ec469427733432afac3fee2f8b0a1b"}
Oct 30 21:47:01.768813 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:01.768732    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ftrj6" podStartSLOduration=1.768718114 podStartE2EDuration="1.768718114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:01.76777731 +0000 UTC m=+4104.832553720" watchObservedRunningTime="2026-10-30 21:47:01.768718114 +0000 UTC m=+4104.833494512"
Oct 30 21:47:02.661474 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:02.661439    5713 generic.go:356] "Generic (PLEG): container finished" podID="a1c2f92a-44cb-460d-9520-367f5447415e" containerID="37d3b282d9b4219003f22c9764afbf245ca06f3fea95c6bf00cf0b1ba04e6ab6" exitCode=0
Oct 30 21:47:02.661927 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:02.661518    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ftrj6" event={"ID":"a1c2f92a-44cb-460d-9520-367f5447415e","Type":"ContainerDied","Data":"37d3b282d9b4219003f22c9764afbf245ca06f3fea95c6bf00cf0b1ba04e6ab6"}
Oct 30 21:47:03.788145 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:03.788121    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ftrj6"
Oct 30 21:47:03.911120 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:03.911089    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1c2f92a-44cb-460d-9520-367f5447415e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1c2f92a-44cb-460d-9520-367f5447415e-maas-api-openshift-ca\") pod \"a1c2f92a-44cb-460d-9520-367f5447415e\" (UID: \"a1c2f92a-44cb-460d-9520-367f5447415e\") "
Oct 30 21:47:03.911305 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:03.911133    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1c2f92a-44cb-460d-9520-367f5447415e-kube-api-access-48fsv\" (UniqueName: \"kubernetes.io/projected/a1c2f92a-44cb-460d-9520-367f5447415e-kube-api-access-48fsv\") pod \"a1c2f92a-44cb-460d-9520-367f5447415e\" (UID: \"a1c2f92a-44cb-460d-9520-367f5447415e\") "
Oct 30 21:47:03.911305 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:03.911227    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1c2f92a-44cb-460d-9520-367f5447415e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1c2f92a-44cb-460d-9520-367f5447415e-maas-api-xks-ca\") pod \"a1c2f92a-44cb-460d-9520-367f5447415e\" (UID: \"a1c2f92a-44cb-460d-9520-367f5447415e\") "
Oct 30 21:47:03.911482 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:03.911452    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1c2f92a-44cb-460d-9520-367f5447415e-maas-api-openshift-ca" pod "a1c2f92a-44cb-460d-9520-367f5447415e" (UID: "a1c2f92a-44cb-460d-9520-367f5447415e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:47:03.913397 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:03.913365    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1c2f92a-44cb-460d-9520-367f5447415e-kube-api-access-48fsv" pod "a1c2f92a-44cb-460d-9520-367f5447415e" (UID: "a1c2f92a-44cb-460d-9520-367f5447415e"). InnerVolumeSpecName "kube-api-access-48fsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:47:03.913397 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:03.913381    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1c2f92a-44cb-460d-9520-367f5447415e-maas-api-xks-ca" pod "a1c2f92a-44cb-460d-9520-367f5447415e" (UID: "a1c2f92a-44cb-460d-9520-367f5447415e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:47:04.012267 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:04.012235    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1c2f92a-44cb-460d-9520-367f5447415e-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:47:04.012267 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:04.012265    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1c2f92a-44cb-460d-9520-367f5447415e-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:47:04.012267 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:04.012276    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48fsv\" (UniqueName: \"kubernetes.io/projected/a1c2f92a-44cb-460d-9520-367f5447415e-kube-api-access-48fsv\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:47:04.673721 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:04.673686    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ftrj6"
Oct 30 21:47:04.673939 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:04.673715    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ftrj6" event={"ID":"a1c2f92a-44cb-460d-9520-367f5447415e","Type":"ContainerDied","Data":"06680d55081342a4a7ed46db0c95f8f8d4ec469427733432afac3fee2f8b0a1b"}
Oct 30 21:47:04.673939 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:04.673764    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06680d55081342a4a7ed46db0c95f8f8d4ec469427733432afac3fee2f8b0a1b"
Oct 30 21:47:15.044757 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:15.044716    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xlb8h"]
Oct 30 21:47:15.094484 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:15.094449    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xlb8h"]
Oct 30 21:47:17.062003 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:17.061970    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dbc5bdf-a04c-4786-8292-53ccd57fec92" path="/var/lib/kubelet/pods/4dbc5bdf-a04c-4786-8292-53ccd57fec92/volumes"
Oct 30 21:47:20.150814 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:20.146898    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-8nzgh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:47:20.150814 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:20.146970    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-8nzgh" podUID="d91e8f40-c2d6-455c-8d73-9f83b615f3dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:47:20.251957 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:20.251909    5713 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-fdp2g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:47:20.252924 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:20.252891    5713 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-fdp2g" podUID="b905ceb0-d3ed-426a-90fe-3717158cee72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:47:21.046314 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:21.046277    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-lkhvn"]
Oct 30 21:47:21.046741 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:21.046728    5713 cpu_manager.go:414] "containerMap: removing container" podUID="a1c2f92a-44cb-460d-9520-367f5447415e" containerName="revoke-keys"
Oct 30 21:47:21.046816 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:21.046743    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1c2f92a-44cb-460d-9520-367f5447415e" containerName="revoke-keys"
Oct 30 21:47:21.046858 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:21.046825    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1c2f92a-44cb-460d-9520-367f5447415e" containerName="revoke-keys"
Oct 30 21:47:21.046858 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:21.046835    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd00fc40-6246-45ea-abf3-4c9e8ab142f3" containerName="istio-proxy"
Oct 30 21:47:21.050037 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:21.050021    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-lkhvn"
Oct 30 21:47:21.103855 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:21.103817    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:47:21.115392 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:21.115358    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/094a1958-a3ea-4c9e-a1bd-8761cfe8d783-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-lkhvn\" (UID: \"094a1958-a3ea-4c9e-a1bd-8761cfe8d783\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-lkhvn"
Oct 30 21:47:21.115582 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:21.115517    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/094a1958-a3ea-4c9e-a1bd-8761cfe8d783-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-lkhvn\" (UID: \"094a1958-a3ea-4c9e-a1bd-8761cfe8d783\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-lkhvn"
Oct 30 21:47:21.115657 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:21.115586    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5x5f\" (UniqueName: \"kubernetes.io/projected/094a1958-a3ea-4c9e-a1bd-8761cfe8d783-kube-api-access-s5x5f\") pod \"maas-api-revoke-sub-0165d7d2-lkhvn\" (UID: \"094a1958-a3ea-4c9e-a1bd-8761cfe8d783\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-lkhvn"
Oct 30 21:47:21.153613 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:21.153568    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:47:21.153613 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:21.153583    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:47:21.153613 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:21.153611    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rwj5h\""
Oct 30 21:47:21.155614 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:21.155591    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-lkhvn"]
Oct 30 21:47:21.216846 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:21.216810    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/094a1958-a3ea-4c9e-a1bd-8761cfe8d783-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-lkhvn\" (UID: \"094a1958-a3ea-4c9e-a1bd-8761cfe8d783\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-lkhvn"
Oct 30 21:47:21.217056 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:21.216871    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5x5f\" (UniqueName: \"kubernetes.io/projected/094a1958-a3ea-4c9e-a1bd-8761cfe8d783-kube-api-access-s5x5f\") pod \"maas-api-revoke-sub-0165d7d2-lkhvn\" (UID: \"094a1958-a3ea-4c9e-a1bd-8761cfe8d783\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-lkhvn"
Oct 30 21:47:21.217056 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:21.216945    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/094a1958-a3ea-4c9e-a1bd-8761cfe8d783-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-lkhvn\" (UID: \"094a1958-a3ea-4c9e-a1bd-8761cfe8d783\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-lkhvn"
Oct 30 21:47:21.217620 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:21.217598    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/094a1958-a3ea-4c9e-a1bd-8761cfe8d783-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-lkhvn\" (UID: \"094a1958-a3ea-4c9e-a1bd-8761cfe8d783\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-lkhvn"
Oct 30 21:47:21.219373 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:21.219352    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/094a1958-a3ea-4c9e-a1bd-8761cfe8d783-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-lkhvn\" (UID: \"094a1958-a3ea-4c9e-a1bd-8761cfe8d783\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-lkhvn"
Oct 30 21:47:21.273595 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:21.273566    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5x5f\" (UniqueName: \"kubernetes.io/projected/094a1958-a3ea-4c9e-a1bd-8761cfe8d783-kube-api-access-s5x5f\") pod \"maas-api-revoke-sub-0165d7d2-lkhvn\" (UID: \"094a1958-a3ea-4c9e-a1bd-8761cfe8d783\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-lkhvn"
Oct 30 21:47:21.363578 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:21.363487    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-lkhvn"
Oct 30 21:47:21.736383 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:21.736357    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-lkhvn"]
Oct 30 21:47:21.738406 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:47:21.738379    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod094a1958_a3ea_4c9e_a1bd_8761cfe8d783.slice/crio-01cf5bdff38089b3067b0fc4bbeba4bc25afeafb956d23bebbfbf5502161166c WatchSource:0}: Error finding container 01cf5bdff38089b3067b0fc4bbeba4bc25afeafb956d23bebbfbf5502161166c: Status 404 returned error can't find the container with id 01cf5bdff38089b3067b0fc4bbeba4bc25afeafb956d23bebbfbf5502161166c
Oct 30 21:47:21.804679 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:21.804637    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-lkhvn" event={"ID":"094a1958-a3ea-4c9e-a1bd-8761cfe8d783","Type":"ContainerStarted","Data":"01cf5bdff38089b3067b0fc4bbeba4bc25afeafb956d23bebbfbf5502161166c"}
Oct 30 21:47:22.811959 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:22.811925    5713 generic.go:356] "Generic (PLEG): container finished" podID="094a1958-a3ea-4c9e-a1bd-8761cfe8d783" containerID="0ce9777075c61444eb41c628c6098f8b420456c296608fe2455d6f6b5560445f" exitCode=0
Oct 30 21:47:22.812367 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:22.812006    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-lkhvn" event={"ID":"094a1958-a3ea-4c9e-a1bd-8761cfe8d783","Type":"ContainerDied","Data":"0ce9777075c61444eb41c628c6098f8b420456c296608fe2455d6f6b5560445f"}
Oct 30 21:47:23.959742 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:23.959712    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-lkhvn"
Oct 30 21:47:24.044990 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:24.044955    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/094a1958-a3ea-4c9e-a1bd-8761cfe8d783-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/094a1958-a3ea-4c9e-a1bd-8761cfe8d783-maas-api-xks-ca\") pod \"094a1958-a3ea-4c9e-a1bd-8761cfe8d783\" (UID: \"094a1958-a3ea-4c9e-a1bd-8761cfe8d783\") "
Oct 30 21:47:24.045184 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:24.044998    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/094a1958-a3ea-4c9e-a1bd-8761cfe8d783-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/094a1958-a3ea-4c9e-a1bd-8761cfe8d783-maas-api-openshift-ca\") pod \"094a1958-a3ea-4c9e-a1bd-8761cfe8d783\" (UID: \"094a1958-a3ea-4c9e-a1bd-8761cfe8d783\") "
Oct 30 21:47:24.045184 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:24.045089    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/094a1958-a3ea-4c9e-a1bd-8761cfe8d783-kube-api-access-s5x5f\" (UniqueName: \"kubernetes.io/projected/094a1958-a3ea-4c9e-a1bd-8761cfe8d783-kube-api-access-s5x5f\") pod \"094a1958-a3ea-4c9e-a1bd-8761cfe8d783\" (UID: \"094a1958-a3ea-4c9e-a1bd-8761cfe8d783\") "
Oct 30 21:47:24.045388 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:24.045364    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/094a1958-a3ea-4c9e-a1bd-8761cfe8d783-maas-api-openshift-ca" pod "094a1958-a3ea-4c9e-a1bd-8761cfe8d783" (UID: "094a1958-a3ea-4c9e-a1bd-8761cfe8d783"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:47:24.045497 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:24.045480    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/094a1958-a3ea-4c9e-a1bd-8761cfe8d783-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:47:24.047268 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:24.047246    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/094a1958-a3ea-4c9e-a1bd-8761cfe8d783-maas-api-xks-ca" pod "094a1958-a3ea-4c9e-a1bd-8761cfe8d783" (UID: "094a1958-a3ea-4c9e-a1bd-8761cfe8d783"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:47:24.047383 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:24.047343    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/094a1958-a3ea-4c9e-a1bd-8761cfe8d783-kube-api-access-s5x5f" pod "094a1958-a3ea-4c9e-a1bd-8761cfe8d783" (UID: "094a1958-a3ea-4c9e-a1bd-8761cfe8d783"). InnerVolumeSpecName "kube-api-access-s5x5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:47:24.146630 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:24.146530    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/094a1958-a3ea-4c9e-a1bd-8761cfe8d783-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:47:24.146630 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:24.146572    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5x5f\" (UniqueName: \"kubernetes.io/projected/094a1958-a3ea-4c9e-a1bd-8761cfe8d783-kube-api-access-s5x5f\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:47:24.458413 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:47:24.458376    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:47:24.831986 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:24.831898    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-lkhvn" event={"ID":"094a1958-a3ea-4c9e-a1bd-8761cfe8d783","Type":"ContainerDied","Data":"01cf5bdff38089b3067b0fc4bbeba4bc25afeafb956d23bebbfbf5502161166c"}
Oct 30 21:47:24.831986 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:24.831922    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-lkhvn"
Oct 30 21:47:24.831986 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:24.831944    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01cf5bdff38089b3067b0fc4bbeba4bc25afeafb956d23bebbfbf5502161166c"
Oct 30 21:47:32.783927 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:32.783890    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nbcq9"]
Oct 30 21:47:32.784552 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:32.784470    5713 cpu_manager.go:414] "containerMap: removing container" podUID="094a1958-a3ea-4c9e-a1bd-8761cfe8d783" containerName="revoke-keys"
Oct 30 21:47:32.784552 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:32.784496    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="094a1958-a3ea-4c9e-a1bd-8761cfe8d783" containerName="revoke-keys"
Oct 30 21:47:32.784705 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:32.784605    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="094a1958-a3ea-4c9e-a1bd-8761cfe8d783" containerName="revoke-keys"
Oct 30 21:47:32.787903 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:32.787879    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nbcq9"
Oct 30 21:47:32.835705 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:32.835664    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92ac4fd3-ce27-42bb-a582-0438cda86c28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-nbcq9\" (UID: \"92ac4fd3-ce27-42bb-a582-0438cda86c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nbcq9"
Oct 30 21:47:32.835705 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:32.835705    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfmtd\" (UniqueName: \"kubernetes.io/projected/92ac4fd3-ce27-42bb-a582-0438cda86c28-kube-api-access-zfmtd\") pod \"maas-api-revoke-sub-6b19338f-nbcq9\" (UID: \"92ac4fd3-ce27-42bb-a582-0438cda86c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nbcq9"
Oct 30 21:47:32.835948 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:32.835809    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92ac4fd3-ce27-42bb-a582-0438cda86c28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-nbcq9\" (UID: \"92ac4fd3-ce27-42bb-a582-0438cda86c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nbcq9"
Oct 30 21:47:32.844118 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:32.844078    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nbcq9"]
Oct 30 21:47:32.890998 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:32.890952    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:47:32.891213 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:32.891052    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rwj5h\""
Oct 30 21:47:32.891213 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:32.891067    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:47:32.891213 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:32.891081    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:47:32.936715 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:32.936682    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92ac4fd3-ce27-42bb-a582-0438cda86c28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-nbcq9\" (UID: \"92ac4fd3-ce27-42bb-a582-0438cda86c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nbcq9"
Oct 30 21:47:32.936878 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:32.936718    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfmtd\" (UniqueName: \"kubernetes.io/projected/92ac4fd3-ce27-42bb-a582-0438cda86c28-kube-api-access-zfmtd\") pod \"maas-api-revoke-sub-6b19338f-nbcq9\" (UID: \"92ac4fd3-ce27-42bb-a582-0438cda86c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nbcq9"
Oct 30 21:47:32.936878 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:32.936844    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92ac4fd3-ce27-42bb-a582-0438cda86c28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-nbcq9\" (UID: \"92ac4fd3-ce27-42bb-a582-0438cda86c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nbcq9"
Oct 30 21:47:32.937350 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:32.937332    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92ac4fd3-ce27-42bb-a582-0438cda86c28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-nbcq9\" (UID: \"92ac4fd3-ce27-42bb-a582-0438cda86c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nbcq9"
Oct 30 21:47:32.939329 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:32.939314    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92ac4fd3-ce27-42bb-a582-0438cda86c28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-nbcq9\" (UID: \"92ac4fd3-ce27-42bb-a582-0438cda86c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nbcq9"
Oct 30 21:47:32.995190 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:32.995154    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfmtd\" (UniqueName: \"kubernetes.io/projected/92ac4fd3-ce27-42bb-a582-0438cda86c28-kube-api-access-zfmtd\") pod \"maas-api-revoke-sub-6b19338f-nbcq9\" (UID: \"92ac4fd3-ce27-42bb-a582-0438cda86c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nbcq9"
Oct 30 21:47:33.101812 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:33.101710    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nbcq9"
Oct 30 21:47:33.476205 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:33.476145    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nbcq9"]
Oct 30 21:47:33.478850 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:47:33.478815    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92ac4fd3_ce27_42bb_a582_0438cda86c28.slice/crio-aa191d241e19887933ef62fe90b941fe46cc3e79ee76824453c3aabdb4d49310 WatchSource:0}: Error finding container aa191d241e19887933ef62fe90b941fe46cc3e79ee76824453c3aabdb4d49310: Status 404 returned error can't find the container with id aa191d241e19887933ef62fe90b941fe46cc3e79ee76824453c3aabdb4d49310
Oct 30 21:47:33.888274 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:33.888232    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nbcq9" event={"ID":"92ac4fd3-ce27-42bb-a582-0438cda86c28","Type":"ContainerStarted","Data":"ff1bce43555352372cb64ba0eb38458c013c6118de1e7bfd09195dae5b14cbf9"}
Oct 30 21:47:33.888833 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:33.888305    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nbcq9" event={"ID":"92ac4fd3-ce27-42bb-a582-0438cda86c28","Type":"ContainerStarted","Data":"aa191d241e19887933ef62fe90b941fe46cc3e79ee76824453c3aabdb4d49310"}
Oct 30 21:47:34.004885 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:34.004754    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nbcq9" podStartSLOduration=2.004738519 podStartE2EDuration="2.004738519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:34.002781268 +0000 UTC m=+4137.067557666" watchObservedRunningTime="2026-10-30 21:47:34.004738519 +0000 UTC m=+4137.069514917"
Oct 30 21:47:34.894680 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:34.894646    5713 generic.go:356] "Generic (PLEG): container finished" podID="92ac4fd3-ce27-42bb-a582-0438cda86c28" containerID="ff1bce43555352372cb64ba0eb38458c013c6118de1e7bfd09195dae5b14cbf9" exitCode=0
Oct 30 21:47:34.895108 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:34.894726    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nbcq9" event={"ID":"92ac4fd3-ce27-42bb-a582-0438cda86c28","Type":"ContainerDied","Data":"ff1bce43555352372cb64ba0eb38458c013c6118de1e7bfd09195dae5b14cbf9"}
Oct 30 21:47:36.037234 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:36.037202    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nbcq9"
Oct 30 21:47:36.171537 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:36.171459    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92ac4fd3-ce27-42bb-a582-0438cda86c28-kube-api-access-zfmtd\" (UniqueName: \"kubernetes.io/projected/92ac4fd3-ce27-42bb-a582-0438cda86c28-kube-api-access-zfmtd\") pod \"92ac4fd3-ce27-42bb-a582-0438cda86c28\" (UID: \"92ac4fd3-ce27-42bb-a582-0438cda86c28\") "
Oct 30 21:47:36.171696 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:36.171615    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92ac4fd3-ce27-42bb-a582-0438cda86c28-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92ac4fd3-ce27-42bb-a582-0438cda86c28-maas-api-openshift-ca\") pod \"92ac4fd3-ce27-42bb-a582-0438cda86c28\" (UID: \"92ac4fd3-ce27-42bb-a582-0438cda86c28\") "
Oct 30 21:47:36.171696 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:36.171637    5713 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92ac4fd3-ce27-42bb-a582-0438cda86c28-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92ac4fd3-ce27-42bb-a582-0438cda86c28-maas-api-xks-ca\") pod \"92ac4fd3-ce27-42bb-a582-0438cda86c28\" (UID: \"92ac4fd3-ce27-42bb-a582-0438cda86c28\") "
Oct 30 21:47:36.172054 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:36.172028    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92ac4fd3-ce27-42bb-a582-0438cda86c28-maas-api-openshift-ca" pod "92ac4fd3-ce27-42bb-a582-0438cda86c28" (UID: "92ac4fd3-ce27-42bb-a582-0438cda86c28"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:47:36.173922 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:36.173891    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92ac4fd3-ce27-42bb-a582-0438cda86c28-kube-api-access-zfmtd" pod "92ac4fd3-ce27-42bb-a582-0438cda86c28" (UID: "92ac4fd3-ce27-42bb-a582-0438cda86c28"). InnerVolumeSpecName "kube-api-access-zfmtd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:47:36.174161 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:36.174139    5713 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92ac4fd3-ce27-42bb-a582-0438cda86c28-maas-api-xks-ca" pod "92ac4fd3-ce27-42bb-a582-0438cda86c28" (UID: "92ac4fd3-ce27-42bb-a582-0438cda86c28"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:47:36.272881 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:36.272843    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92ac4fd3-ce27-42bb-a582-0438cda86c28-maas-api-openshift-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:47:36.272881 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:36.272874    5713 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92ac4fd3-ce27-42bb-a582-0438cda86c28-maas-api-xks-ca\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:47:36.272881 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:36.272883    5713 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfmtd\" (UniqueName: \"kubernetes.io/projected/92ac4fd3-ce27-42bb-a582-0438cda86c28-kube-api-access-zfmtd\") on node \"ip-10-0-128-165.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:47:36.906869 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:36.906837    5713 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nbcq9"
Oct 30 21:47:36.906869 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:36.906855    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nbcq9" event={"ID":"92ac4fd3-ce27-42bb-a582-0438cda86c28","Type":"ContainerDied","Data":"aa191d241e19887933ef62fe90b941fe46cc3e79ee76824453c3aabdb4d49310"}
Oct 30 21:47:36.907126 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:36.906930    5713 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa191d241e19887933ef62fe90b941fe46cc3e79ee76824453c3aabdb4d49310"
Oct 30 21:47:49.045264 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:49.045186    5713 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfe4e0bf-wxkhd"]
Oct 30 21:47:49.096316 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:49.096273    5713 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfe4e0bf-wxkhd"]
Oct 30 21:47:51.063233 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:51.063192    5713 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bf696ed-fdbc-4776-8527-7fc89ca2b35b" path="/var/lib/kubelet/pods/2bf696ed-fdbc-4776-8527-7fc89ca2b35b/volumes"
Oct 30 21:47:54.457936 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:47:54.457902    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:47:59.533540 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:59.533507    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-lkhvn_094a1958-a3ea-4c9e-a1bd-8761cfe8d783/revoke-keys/0.log"
Oct 30 21:47:59.758940 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:59.758911    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-nbcq9_92ac4fd3-ce27-42bb-a582-0438cda86c28/revoke-keys/0.log"
Oct 30 21:47:59.975843 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:47:59.975819    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-ftrj6_a1c2f92a-44cb-460d-9520-367f5447415e/revoke-keys/0.log"
Oct 30 21:48:03.557051 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:48:03.557002    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-4hh9b_d2d9bfed-999d-4d51-91a8-30b0e938ee1d/manager/0.log"
Oct 30 21:48:05.188482 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:48:05.188444    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7988bdb8dc-nzmcx_cfe209a2-82e3-4c1e-99d0-615d8294737a/kube-auth-proxy/0.log"
Oct 30 21:48:05.706829 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:48:05.706775    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-769df7846c-8nzgh_d91e8f40-c2d6-455c-8d73-9f83b615f3dd/istio-proxy/0.log"
Oct 30 21:48:06.152231 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:48:06.152134    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-769df7846c-fdp2g_b905ceb0-d3ed-426a-90fe-3717158cee72/istio-proxy/0.log"
Oct 30 21:48:06.611556 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:48:06.611519    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-769df7846c-pmsc4_77cf9324-4316-4631-9d6f-cf42ff54290f/istio-proxy/0.log"
Oct 30 21:48:07.913536 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:48:07.913506    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-64dd656c99-msv5c_2dd002cb-a7d3-4b19-bbf0-f3d2a04d9666/istio-proxy/0.log"
Oct 30 21:48:09.767758 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:48:09.767715    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-mb6fg_1ae4eaba-11ff-45e4-95d9-9bf25c06d42f/main/0.log"
Oct 30 21:48:09.993087 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:48:09.993059    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-dvl5x_4cc10a3b-ecd9-4dba-a799-7c4de06b7523/main/0.log"
Oct 30 21:48:10.448674 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:48:10.448646    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hmv85_fb3b89e5-8e1b-4881-9400-a8cd97ff6b19/main/0.log"
Oct 30 21:48:10.928432 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:48:10.928330    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-v2lfl_2cff5280-b60c-4285-8250-9482152de0d8/main/0.log"
Oct 30 21:48:11.150906 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:48:11.150869    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l_fc9510c5-2180-41d6-90bc-9acdff0f3616/main/0.log"
Oct 30 21:48:24.458127 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:48:24.458078    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:48:24.841259 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:48:24.841169    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-lkhvn_094a1958-a3ea-4c9e-a1bd-8761cfe8d783/revoke-keys/0.log"
Oct 30 21:48:25.064977 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:48:25.064931    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-nbcq9_92ac4fd3-ce27-42bb-a582-0438cda86c28/revoke-keys/0.log"
Oct 30 21:48:25.294492 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:48:25.294462    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-ftrj6_a1c2f92a-44cb-460d-9520-367f5447415e/revoke-keys/0.log"
Oct 30 21:48:28.650884 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:48:28.650835    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-4hh9b_d2d9bfed-999d-4d51-91a8-30b0e938ee1d/manager/0.log"
Oct 30 21:48:30.111705 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:48:30.111672    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7988bdb8dc-nzmcx_cfe209a2-82e3-4c1e-99d0-615d8294737a/kube-auth-proxy/0.log"
Oct 30 21:48:30.629869 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:48:30.629823    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-769df7846c-8nzgh_d91e8f40-c2d6-455c-8d73-9f83b615f3dd/istio-proxy/0.log"
Oct 30 21:48:31.087341 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:48:31.087300    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-769df7846c-fdp2g_b905ceb0-d3ed-426a-90fe-3717158cee72/istio-proxy/0.log"
Oct 30 21:48:31.558064 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:48:31.558014    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-769df7846c-pmsc4_77cf9324-4316-4631-9d6f-cf42ff54290f/istio-proxy/0.log"
Oct 30 21:48:32.835362 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:48:32.835333    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-64dd656c99-msv5c_2dd002cb-a7d3-4b19-bbf0-f3d2a04d9666/istio-proxy/0.log"
Oct 30 21:48:34.467440 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:48:34.467406    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-mb6fg_1ae4eaba-11ff-45e4-95d9-9bf25c06d42f/main/0.log"
Oct 30 21:48:34.702582 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:48:34.702554    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-dvl5x_4cc10a3b-ecd9-4dba-a799-7c4de06b7523/main/0.log"
Oct 30 21:48:35.147887 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:48:35.147854    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hmv85_fb3b89e5-8e1b-4881-9400-a8cd97ff6b19/main/0.log"
Oct 30 21:48:35.622806 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:48:35.622747    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-v2lfl_2cff5280-b60c-4285-8250-9482152de0d8/main/0.log"
Oct 30 21:48:35.850718 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:48:35.850677    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-4cb8l_fc9510c5-2180-41d6-90bc-9acdff0f3616/main/0.log"
Oct 30 21:48:37.934818 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:48:37.934769    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d4597073-8ff0-4a4e-933c-c64e934fe977/prometheus/0.log"
Oct 30 21:48:37.945487 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:48:37.945466    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d4597073-8ff0-4a4e-933c-c64e934fe977/prometheus/0.log"
Oct 30 21:48:37.949655 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:48:37.949634    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r7lpf_764461d5-e0b2-4856-a97e-0c5d3d34a497/ovn-acl-logging/0.log"
Oct 30 21:48:37.959599 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:48:37.959579    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r7lpf_764461d5-e0b2-4856-a97e-0c5d3d34a497/ovn-acl-logging/0.log"
Oct 30 21:48:40.748097 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:48:40.748052    5713 scope.go:122] "RemoveContainer" containerID="d06b8cca8eb4a4f538a552f2461253afe57a7bef880c7eb9f9abf8920e032724"
Oct 30 21:48:54.459025 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:48:54.458984    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:49:24.459071 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:49:24.458985    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:49:40.769956 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:49:40.769920    5713 scope.go:122] "RemoveContainer" containerID="a7962d533bd7042abc78f0fa61f59bd14def8b8401de6c0c4f7d37acd633fedd"
Oct 30 21:49:44.165043 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:49:44.165011    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8xd6d/must-gather-snzrw"]
Oct 30 21:49:44.165441 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:49:44.165425    5713 cpu_manager.go:414] "containerMap: removing container" podUID="92ac4fd3-ce27-42bb-a582-0438cda86c28" containerName="revoke-keys"
Oct 30 21:49:44.165441 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:49:44.165436    5713 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92ac4fd3-ce27-42bb-a582-0438cda86c28" containerName="revoke-keys"
Oct 30 21:49:44.165517 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:49:44.165501    5713 memory_manager.go:365] "RemoveStaleState removing state" podUID="92ac4fd3-ce27-42bb-a582-0438cda86c28" containerName="revoke-keys"
Oct 30 21:49:44.168393 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:49:44.168376    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8xd6d/must-gather-snzrw"
Oct 30 21:49:44.190639 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:49:44.190612    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1eacd058-6599-4f23-a441-c94ed3b7ed45-must-gather-output\") pod \"must-gather-snzrw\" (UID: \"1eacd058-6599-4f23-a441-c94ed3b7ed45\") " pod="openshift-must-gather-8xd6d/must-gather-snzrw"
Oct 30 21:49:44.190757 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:49:44.190669    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6rmt\" (UniqueName: \"kubernetes.io/projected/1eacd058-6599-4f23-a441-c94ed3b7ed45-kube-api-access-f6rmt\") pod \"must-gather-snzrw\" (UID: \"1eacd058-6599-4f23-a441-c94ed3b7ed45\") " pod="openshift-must-gather-8xd6d/must-gather-snzrw"
Oct 30 21:49:44.220397 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:49:44.220366    5713 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8xd6d\"/\"default-dockercfg-hwbdz\""
Oct 30 21:49:44.270105 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:49:44.270072    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8xd6d\"/\"kube-root-ca.crt\""
Oct 30 21:49:44.270302 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:49:44.270109    5713 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8xd6d\"/\"openshift-service-ca.crt\""
Oct 30 21:49:44.272270 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:49:44.272245    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8xd6d/must-gather-snzrw"]
Oct 30 21:49:44.291328 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:49:44.291301    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1eacd058-6599-4f23-a441-c94ed3b7ed45-must-gather-output\") pod \"must-gather-snzrw\" (UID: \"1eacd058-6599-4f23-a441-c94ed3b7ed45\") " pod="openshift-must-gather-8xd6d/must-gather-snzrw"
Oct 30 21:49:44.291465 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:49:44.291389    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6rmt\" (UniqueName: \"kubernetes.io/projected/1eacd058-6599-4f23-a441-c94ed3b7ed45-kube-api-access-f6rmt\") pod \"must-gather-snzrw\" (UID: \"1eacd058-6599-4f23-a441-c94ed3b7ed45\") " pod="openshift-must-gather-8xd6d/must-gather-snzrw"
Oct 30 21:49:44.291619 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:49:44.291601    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1eacd058-6599-4f23-a441-c94ed3b7ed45-must-gather-output\") pod \"must-gather-snzrw\" (UID: \"1eacd058-6599-4f23-a441-c94ed3b7ed45\") " pod="openshift-must-gather-8xd6d/must-gather-snzrw"
Oct 30 21:49:44.348701 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:49:44.348677    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6rmt\" (UniqueName: \"kubernetes.io/projected/1eacd058-6599-4f23-a441-c94ed3b7ed45-kube-api-access-f6rmt\") pod \"must-gather-snzrw\" (UID: \"1eacd058-6599-4f23-a441-c94ed3b7ed45\") " pod="openshift-must-gather-8xd6d/must-gather-snzrw"
Oct 30 21:49:44.481989 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:49:44.481954    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8xd6d/must-gather-snzrw"
Oct 30 21:49:44.854317 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:49:44.854287    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8xd6d/must-gather-snzrw"]
Oct 30 21:49:44.856258 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:49:44.856231    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1eacd058_6599_4f23_a441_c94ed3b7ed45.slice/crio-d10155ef0efd39371e7f8df1f44a509898fe288da38388f4c8fb1d0dd256d1ed WatchSource:0}: Error finding container d10155ef0efd39371e7f8df1f44a509898fe288da38388f4c8fb1d0dd256d1ed: Status 404 returned error can't find the container with id d10155ef0efd39371e7f8df1f44a509898fe288da38388f4c8fb1d0dd256d1ed
Oct 30 21:49:44.858069 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:49:44.858048    5713 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:49:45.624120 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:49:45.624066    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8xd6d/must-gather-snzrw" event={"ID":"1eacd058-6599-4f23-a441-c94ed3b7ed45","Type":"ContainerStarted","Data":"d10155ef0efd39371e7f8df1f44a509898fe288da38388f4c8fb1d0dd256d1ed"}
Oct 30 21:49:46.631922 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:49:46.631819    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8xd6d/must-gather-snzrw" event={"ID":"1eacd058-6599-4f23-a441-c94ed3b7ed45","Type":"ContainerStarted","Data":"5eac7289129f8f8634279a4bd166feda2cc0165515e395541d71adf619ce90d2"}
Oct 30 21:49:46.631922 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:49:46.631871    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8xd6d/must-gather-snzrw" event={"ID":"1eacd058-6599-4f23-a441-c94ed3b7ed45","Type":"ContainerStarted","Data":"c33db8d88def72d78c1843a4bff1d8a701f76ed641fd136ba3bf222259eaf9bb"}
Oct 30 21:49:46.748228 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:49:46.748155    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-8xd6d/must-gather-snzrw" podStartSLOduration=1.528703314 podStartE2EDuration="2.748138781s" totalImagesPullingTime="1.219435467s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:46.746131907 +0000 UTC m=+4269.810908298" watchObservedRunningTime="2026-10-30 21:49:46.748138781 +0000 UTC m=+4269.812915179"
Oct 30 21:49:54.023830 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:49:54.023761    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xj584_1a74da60-0284-410b-9d69-0d5192ce0928/global-pull-secret-syncer/0.log"
Oct 30 21:49:54.538235 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:49:54.538165    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"
Oct 30 21:49:56.341548 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:49:56.341486    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-lkxfb_c3704770-6fb6-4086-8a22-9b29468d5a70/konnectivity-agent/0.log"
Oct 30 21:49:57.303813 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:49:57.299339    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-165.us-west-1.compute.internal_40ad3a6a5fd37a90a3238925c9fca019/haproxy/0.log"
Oct 30 21:50:03.549353 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:50:03.549313    5713 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8xd6d/perf-node-gather-daemonset-mcld8"]
Oct 30 21:50:03.556910 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:50:03.556883    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8xd6d/perf-node-gather-daemonset-mcld8"
Oct 30 21:50:03.639398 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:50:03.639362    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8xd6d/perf-node-gather-daemonset-mcld8"]
Oct 30 21:50:03.688204 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:50:03.688168    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb1ed7f7-a42b-478e-9ac9-8bd6258e881a-sys\") pod \"perf-node-gather-daemonset-mcld8\" (UID: \"cb1ed7f7-a42b-478e-9ac9-8bd6258e881a\") " pod="openshift-must-gather-8xd6d/perf-node-gather-daemonset-mcld8"
Oct 30 21:50:03.688204 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:50:03.688209    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cb1ed7f7-a42b-478e-9ac9-8bd6258e881a-podres\") pod \"perf-node-gather-daemonset-mcld8\" (UID: \"cb1ed7f7-a42b-478e-9ac9-8bd6258e881a\") " pod="openshift-must-gather-8xd6d/perf-node-gather-daemonset-mcld8"
Oct 30 21:50:03.688479 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:50:03.688227    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cb1ed7f7-a42b-478e-9ac9-8bd6258e881a-lib-modules\") pod \"perf-node-gather-daemonset-mcld8\" (UID: \"cb1ed7f7-a42b-478e-9ac9-8bd6258e881a\") " pod="openshift-must-gather-8xd6d/perf-node-gather-daemonset-mcld8"
Oct 30 21:50:03.688479 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:50:03.688381    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cb1ed7f7-a42b-478e-9ac9-8bd6258e881a-proc\") pod \"perf-node-gather-daemonset-mcld8\" (UID: \"cb1ed7f7-a42b-478e-9ac9-8bd6258e881a\") " pod="openshift-must-gather-8xd6d/perf-node-gather-daemonset-mcld8"
Oct 30 21:50:03.688567 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:50:03.688518    5713 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkkpr\" (UniqueName: \"kubernetes.io/projected/cb1ed7f7-a42b-478e-9ac9-8bd6258e881a-kube-api-access-hkkpr\") pod \"perf-node-gather-daemonset-mcld8\" (UID: \"cb1ed7f7-a42b-478e-9ac9-8bd6258e881a\") " pod="openshift-must-gather-8xd6d/perf-node-gather-daemonset-mcld8"
Oct 30 21:50:03.789391 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:50:03.789350    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkkpr\" (UniqueName: \"kubernetes.io/projected/cb1ed7f7-a42b-478e-9ac9-8bd6258e881a-kube-api-access-hkkpr\") pod \"perf-node-gather-daemonset-mcld8\" (UID: \"cb1ed7f7-a42b-478e-9ac9-8bd6258e881a\") " pod="openshift-must-gather-8xd6d/perf-node-gather-daemonset-mcld8"
Oct 30 21:50:03.789391 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:50:03.789399    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb1ed7f7-a42b-478e-9ac9-8bd6258e881a-sys\") pod \"perf-node-gather-daemonset-mcld8\" (UID: \"cb1ed7f7-a42b-478e-9ac9-8bd6258e881a\") " pod="openshift-must-gather-8xd6d/perf-node-gather-daemonset-mcld8"
Oct 30 21:50:03.789652 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:50:03.789426    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cb1ed7f7-a42b-478e-9ac9-8bd6258e881a-podres\") pod \"perf-node-gather-daemonset-mcld8\" (UID: \"cb1ed7f7-a42b-478e-9ac9-8bd6258e881a\") " pod="openshift-must-gather-8xd6d/perf-node-gather-daemonset-mcld8"
Oct 30 21:50:03.789652 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:50:03.789447    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cb1ed7f7-a42b-478e-9ac9-8bd6258e881a-lib-modules\") pod \"perf-node-gather-daemonset-mcld8\" (UID: \"cb1ed7f7-a42b-478e-9ac9-8bd6258e881a\") " pod="openshift-must-gather-8xd6d/perf-node-gather-daemonset-mcld8"
Oct 30 21:50:03.789652 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:50:03.789494    5713 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cb1ed7f7-a42b-478e-9ac9-8bd6258e881a-proc\") pod \"perf-node-gather-daemonset-mcld8\" (UID: \"cb1ed7f7-a42b-478e-9ac9-8bd6258e881a\") " pod="openshift-must-gather-8xd6d/perf-node-gather-daemonset-mcld8"
Oct 30 21:50:03.789652 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:50:03.789541    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb1ed7f7-a42b-478e-9ac9-8bd6258e881a-sys\") pod \"perf-node-gather-daemonset-mcld8\" (UID: \"cb1ed7f7-a42b-478e-9ac9-8bd6258e881a\") " pod="openshift-must-gather-8xd6d/perf-node-gather-daemonset-mcld8"
Oct 30 21:50:03.789652 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:50:03.789560    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cb1ed7f7-a42b-478e-9ac9-8bd6258e881a-podres\") pod \"perf-node-gather-daemonset-mcld8\" (UID: \"cb1ed7f7-a42b-478e-9ac9-8bd6258e881a\") " pod="openshift-must-gather-8xd6d/perf-node-gather-daemonset-mcld8"
Oct 30 21:50:03.789652 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:50:03.789601    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cb1ed7f7-a42b-478e-9ac9-8bd6258e881a-proc\") pod \"perf-node-gather-daemonset-mcld8\" (UID: \"cb1ed7f7-a42b-478e-9ac9-8bd6258e881a\") " pod="openshift-must-gather-8xd6d/perf-node-gather-daemonset-mcld8"
Oct 30 21:50:03.789652 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:50:03.789594    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cb1ed7f7-a42b-478e-9ac9-8bd6258e881a-lib-modules\") pod \"perf-node-gather-daemonset-mcld8\" (UID: \"cb1ed7f7-a42b-478e-9ac9-8bd6258e881a\") " pod="openshift-must-gather-8xd6d/perf-node-gather-daemonset-mcld8"
Oct 30 21:50:03.846496 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:50:03.846428    5713 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkkpr\" (UniqueName: \"kubernetes.io/projected/cb1ed7f7-a42b-478e-9ac9-8bd6258e881a-kube-api-access-hkkpr\") pod \"perf-node-gather-daemonset-mcld8\" (UID: \"cb1ed7f7-a42b-478e-9ac9-8bd6258e881a\") " pod="openshift-must-gather-8xd6d/perf-node-gather-daemonset-mcld8"
Oct 30 21:50:03.871447 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:50:03.871407    5713 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8xd6d/perf-node-gather-daemonset-mcld8"
Oct 30 21:50:04.248082 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:50:04.248058    5713 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8xd6d/perf-node-gather-daemonset-mcld8"]
Oct 30 21:50:04.249887 ip-10-0-128-165 kubenswrapper[5713]: W1030 21:50:04.249850    5713 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podcb1ed7f7_a42b_478e_9ac9_8bd6258e881a.slice/crio-f691386e5214eec32c27afe62443d1dbe55a824a52c0a301610737899f0a5c55 WatchSource:0}: Error finding container f691386e5214eec32c27afe62443d1dbe55a824a52c0a301610737899f0a5c55: Status 404 returned error can't find the container with id f691386e5214eec32c27afe62443d1dbe55a824a52c0a301610737899f0a5c55
Oct 30 21:50:04.798862 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:50:04.798823    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8xd6d/perf-node-gather-daemonset-mcld8" event={"ID":"cb1ed7f7-a42b-478e-9ac9-8bd6258e881a","Type":"ContainerStarted","Data":"310bbc3a2dd7bd19aa517a86140eb67d48350baad99f366b3924f5c9475bc4a3"}
Oct 30 21:50:04.799397 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:50:04.798889    5713 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8xd6d/perf-node-gather-daemonset-mcld8" event={"ID":"cb1ed7f7-a42b-478e-9ac9-8bd6258e881a","Type":"ContainerStarted","Data":"f691386e5214eec32c27afe62443d1dbe55a824a52c0a301610737899f0a5c55"}
Oct 30 21:50:04.799397 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:50:04.799044    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-8xd6d/perf-node-gather-daemonset-mcld8"
Oct 30 21:50:04.914067 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:50:04.913994    5713 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-8xd6d/perf-node-gather-daemonset-mcld8" podStartSLOduration=1.9139618390000002 podStartE2EDuration="1.913961839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:50:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:04.91304075 +0000 UTC m=+4287.977817148" watchObservedRunningTime="2026-10-30 21:50:04.913961839 +0000 UTC m=+4287.978738236"
Oct 30 21:50:05.690119 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:50:05.690084    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-4hh9b_d2d9bfed-999d-4d51-91a8-30b0e938ee1d/manager/0.log"
Oct 30 21:50:08.322271 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:50:08.322235    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_80d8f079-dd81-4840-9fdc-3a3e056db0a4/alertmanager/0.log"
Oct 30 21:50:08.735975 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:50:08.735940    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_80d8f079-dd81-4840-9fdc-3a3e056db0a4/config-reloader/0.log"
Oct 30 21:50:09.154124 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:50:09.154046    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_80d8f079-dd81-4840-9fdc-3a3e056db0a4/kube-rbac-proxy-web/0.log"
Oct 30 21:50:09.567712 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:50:09.567679    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_80d8f079-dd81-4840-9fdc-3a3e056db0a4/kube-rbac-proxy/0.log"
Oct 30 21:50:09.981572 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:50:09.981536    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_80d8f079-dd81-4840-9fdc-3a3e056db0a4/kube-rbac-proxy-metric/0.log"
Oct 30 21:50:10.400370 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:50:10.400268    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_80d8f079-dd81-4840-9fdc-3a3e056db0a4/prom-label-proxy/0.log"
Oct 30 21:50:10.815774 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:50:10.815747    5713 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-8xd6d/perf-node-gather-daemonset-mcld8"
Oct 30 21:50:10.818513 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:50:10.818492    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_80d8f079-dd81-4840-9fdc-3a3e056db0a4/init-config-reloader/0.log"
Oct 30 21:50:11.757635 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:50:11.757606    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-cfcqd_5b900a91-15bc-4019-99f5-4619cea03518/kube-state-metrics/0.log"
Oct 30 21:50:12.166948 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:50:12.166870    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-cfcqd_5b900a91-15bc-4019-99f5-4619cea03518/kube-rbac-proxy-main/0.log"
Oct 30 21:50:12.579888 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:50:12.579850    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-cfcqd_5b900a91-15bc-4019-99f5-4619cea03518/kube-rbac-proxy-self/0.log"
Oct 30 21:50:13.004671 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:50:13.004641    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-bfd9f4b99-p95mj_1004eede-9d90-42b0-bd11-42a32056f2ae/metrics-server/0.log"
Oct 30 21:50:15.098359 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:50:15.098331    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lwhq8_53aaf696-7925-43b6-86fd-1b536b3095d3/node-exporter/0.log"
Oct 30 21:50:15.511824 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:50:15.511773    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lwhq8_53aaf696-7925-43b6-86fd-1b536b3095d3/kube-rbac-proxy/0.log"
Oct 30 21:50:15.927138 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:50:15.927046    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lwhq8_53aaf696-7925-43b6-86fd-1b536b3095d3/init-textfile/0.log"
Oct 30 21:50:17.675025 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:50:17.674995    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-c4xbh_7787041b-34dc-4567-8989-22bfd95456e0/kube-rbac-proxy-main/0.log"
Oct 30 21:50:18.133952 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:50:18.133921    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-c4xbh_7787041b-34dc-4567-8989-22bfd95456e0/kube-rbac-proxy-self/0.log"
Oct 30 21:50:18.547160 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:50:18.547128    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-c4xbh_7787041b-34dc-4567-8989-22bfd95456e0/openshift-state-metrics/0.log"
Oct 30 21:50:18.990663 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:50:18.990635    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d4597073-8ff0-4a4e-933c-c64e934fe977/prometheus/0.log"
Oct 30 21:50:18.992273 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:50:18.992233    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d4597073-8ff0-4a4e-933c-c64e934fe977/prometheus/1.log"
Oct 30 21:50:19.304060 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:50:19.303980    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d4597073-8ff0-4a4e-933c-c64e934fe977/config-reloader/0.log"
Oct 30 21:50:19.717915 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:50:19.717880    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d4597073-8ff0-4a4e-933c-c64e934fe977/thanos-sidecar/0.log"
Oct 30 21:50:20.131109 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:50:20.131036    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d4597073-8ff0-4a4e-933c-c64e934fe977/kube-rbac-proxy-web/0.log"
Oct 30 21:50:20.543993 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:50:20.543958    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d4597073-8ff0-4a4e-933c-c64e934fe977/kube-rbac-proxy/0.log"
Oct 30 21:50:20.957390 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:50:20.957358    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d4597073-8ff0-4a4e-933c-c64e934fe977/kube-rbac-proxy-thanos/0.log"
Oct 30 21:50:21.371457 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:50:21.371356    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d4597073-8ff0-4a4e-933c-c64e934fe977/init-config-reloader/0.log"
Oct 30 21:50:23.037779 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:50:23.037750    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-744b77f955-tzvzl_f94cda1e-ed8b-4b22-9620-a0c3cf1af271/telemeter-client/0.log"
Oct 30 21:50:23.451220 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:50:23.451186    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-744b77f955-tzvzl_f94cda1e-ed8b-4b22-9620-a0c3cf1af271/reload/0.log"
Oct 30 21:50:23.863865 ip-10-0-128-165 kubenswrapper[5713]: I1030 21:50:23.863781    5713 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-744b77f955-tzvzl_f94cda1e-ed8b-4b22-9620-a0c3cf1af271/kube-rbac-proxy/0.log"
Oct 30 21:50:24.458577 ip-10-0-128-165 kubenswrapper[5713]: E1030 21:50:24.458546    5713 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8khvs" containerName="extractor"